That Advice About Which Language AI Codes Best In? It Falls Apart on Real Work
Dan Luu re-ran the popular claim that dynamic languages are cheaper for coding agents. On two large tasks the effect vanished, and the second-most-cited benchmark turned out to be broken.
There is a piece of advice going around that if you want an AI coding agent to be cheap and effective, you should pick a compact, dynamically typed language. Google’s own AI summary repeats it. Dan Luu, an engineer with a long track record of taking popular claims apart, ran his own tests. On anything resembling real work, the effect disappears.
The original claim came from a benchmark measuring how many tokens, the small chunks of text a model reads and writes and gets billed for, different languages need to solve problems from Rosetta Code. It found a 2.6 times gap between C, the worst, and Clojure, the best, and an even bigger win for the array language J at around 70 tokens per solution. Luu’s objection is blunt: a problem you can solve in 70 tokens is not a problem. Most of the work is printing the answer.
So he built two harder tests. In the first, agents were given the zstd compression specification and told to implement a working decoder, with no internet access and no sight of the tests. In the second, they were handed the Pandoc document converter benchmark and scored against tests they had never seen. Across both, no clear winner emerged between static and dynamic languages. At higher reasoning effort, several static languages came out on top. The obscure dense languages that had looked so impressive on toy problems did badly, which fits a simple explanation: AI labs spend their training effort on languages people actually use.
What is behind it
The most useful part of the write-up is the autopsy of the other benchmark people cite. One test in it ran an executable at a path that did not exist, so every Rust run failed. A later Go run “fixed” this by creating a symbolic link to its own binary, which meant every subsequent test in every language quietly ran the Go program. The benchmark’s author concluded that Rust struggles because it is a difficult language. Rescored properly, Rust gets a perfect mark. Two other tests had a copy-paste error that made them pass no matter what the code did.
Luu is careful about his own work too. He notes he fixed over a hundred setup bugs and expects more remain, and he refuses to make claims about individual languages from two tasks. The one pattern he will stand behind is weak but consistent: more popular languages tend to produce more correct and cheaper results.
What this means for you: Do not switch languages because a chart said an agent would be cheaper there. Stick with something mainstream and well supported. More broadly, this is a good lesson in how AI benchmarks go wrong: a striking result on tiny problems usually stops being striking the moment the problem gets big enough to matter. When you see one, ask how hard the tasks were.
Sources
Source: https://danluu.com/pl-tokens/
The Creator of Redis Made a Chinese Video Model Run on a Mac. Europeans Are Not Licensed to Use It
Salvatore Sanfilippo published h3.c, a C and Metal engine that runs MiniMax H3 natively on Apple Silicon. The model's own licence excludes the EU, UK, US and South Korea from local deployment.
