A Model 28 Times Cheaper Found Three Quarters of the Bugs, and Missed Half the Security Ones

Entelligence published a comparison this week that answers a question a lot of teams are quietly wrestling with: how much do you actually lose by using the cheap model? They took 50 real pull requests, the proposed code changes developers submit for review, from projects including Sentry, Discourse, Keycloak, Cal.com and Grafana. Then they sent each one to GPT-5.6 Luna and to GPT-6 Astra with identical instructions, and counted the genuine defects each found.

Astra found 92 verified bugs, Luna found 69. Cost per review was 0.113 dollars for Astra and 0.0041 for Luna, a difference of about 28 times, and Luna was faster too, 23 seconds against 36. Taken as a headline that sounds like an easy win for the cheap model. The breakdown says otherwise. On ordinary data and logic mistakes the two were close. On concurrency, the class of bug where two things happen at once and step on each other, Luna slipped. On security defects it fell over: 9 of 24 found, against Astra’s 19. Precision, meaning how often a reported problem was real rather than noise, was 74 percent for Luna and 96 percent for Astra, so the cheap reviewer also wastes more of your attention. The team verified findings by having two other models judge independently and requiring them to agree, which is a reasonable method and also, to be fair, a circular one: models grading models.

What is behind this

The pattern fits what we know about how small and large models differ. Finding a null check that is missing is close to pattern matching, and small models are good at pattern matching. Finding a security flaw usually means holding several things in mind at once, tracing where data comes from, imagining an attacker, and reasoning about what happens two steps later. That is the capability that shrinks first when a model gets cheaper. So the right question is not “is the cheap model good enough” but “good enough at what”, and the answer here has a clear shape: yes for everyday correctness, no for anything where being wrong is expensive.

What this means for you: If you review code with AI, the sensible setup is both rather than either. Run the cheap model on everything as a first pass, since at four tenths of a cent a review there is no reason not to, and reserve the expensive one for changes that touch authentication, permissions, payments or anything handling data from strangers. If you do not write code, the transferable lesson is worth more than the benchmark: cheap models are not uniformly worse, they are worse in specific places, and those places tend to be the ones needing several steps of careful reasoning. That is a useful thing to keep in mind whenever you are choosing between the fast, free tier and the slow, paid one for your own work.

Sources

• Entelligence: GPT-5.6 Luna vs GPT-6 Astra, is a 1.20 dollar model good enough for code review
• HyperAI: Budget code review misses security bugs
• Hacker News discussion