A Tougher AI Scoreboard Widens the Gap Between Open and Closed Models
Artificial Analysis rebuilt its Intelligence Index with mostly private tests, and the gap between the best open models and the leaders grew from 3 points to 12.
Artificial Analysis, a firm that scores AI models, released version 5 of its Intelligence Index, and the picture changed. Under the new scoring, the distance between the best open models and the top of the leaderboard is 12 points, up from 3 under the old index. Kimi K3.1 Pro, the open-weight model from Moonshot AI that launched last week, fell from fourth place to ninth, behind eight closed models. The Decoder also reports that Grok 4.7 lost 12 points, though its new rank was not given.
The main change is how the tests are built. For the first time, 60 percent of the index relies on private test sets, questions that labs cannot see in advance. The goal is to stop what the community calls “benchmaxxing”: training a model on the exact questions it will be graded on, like a student who memorised last year’s exam. Artificial Analysis also fixed ambiguous or flawed questions in AA-Omniscience, its test of factual knowledge. The Decoder’s piece does not list the full scores, so the primary source is the Artificial Analysis write-up it cites.
What’s behind it: a benchmark is a standardised exam for AI models, and every exam eventually gets gamed once the answers leak. Moving to private questions is the usual fix. It is also not the whole story. AI researcher Nathan Lambert argues that many gains come from reinforcement learning (training a model by rewarding good answers) in practice environments that resemble the tests, and that building those environments at scale takes infrastructure not every lab has. In other words, the closed labs may simply be better equipped, not just better at cheating or being honest.
There is a trade-off too. With private questions, outsiders cannot check the prompts for mistakes, and the fixes to AA-Omniscience show such mistakes really do occur. Transparent but gameable, or hidden but unverifiable: neither is perfect.
What this means for you: if you are new to this, the takeaway is simple. A single score on a leaderboard is a rough guide, not a verdict, and it can move a lot when the test changes. If you use open models for privacy or cost reasons, the new numbers are a reminder to test candidates on your own tasks rather than trusting a ranking. A 12 point gap on an index does not mean 12 points worse for your emails or spreadsheets. And if a launch announcement cites a rank, check which version of the index it used.
Sources
Source: https://the-decoder.com/new-artificial-analysis-index-widens-the-gap-between-open-and-closed-models/
Security Startup's AI Agents Got Into Anthropic's Business Systems, and Both Sides Tell It Differently
Researchers from Cathedral used AI agents to reach some of Anthropic's third-party business software. Anthropic confirms limited access and says Claude chats, code and model weights were untouched.
