AI agents made research software 60 times faster, and were confidently wrong in ways nobody spotted for weeks
A field report from OpenAI and academic partners documents eight projects where coding agents modernised neglected scientific software. The speedups are real. So are the bugs that only turned up under proper statistical testing.
A great deal of science runs on software that was written once, for one paper, by people with no time to maintain it. A new field report from OpenAI and a set of academic partners documents eight attempts to hand that problem to AI coding agents, and the results cut both ways in a way worth reading carefully.
The speedups are not small. RustQC, which merged fifteen separate quality control tools into one program, cut a run from 15 hours and 34 minutes down to 14 minutes and 54 seconds, more than sixty times faster. HelixForge, a rewrite that moved genomic data generation onto graphics cards, finished a test pipeline 59.6 times faster than the tool it replaced. A rebuild of the genome aligner STAR, more than 20,000 lines of C and C++ that nobody maintains any more, was reimplemented in the language Rust and agreed with the original on 99.815 percent of test reads.
Then the other half. In a rewrite of a statistics package called bayesm, an agent inverted a control parameter so the program silently used the reciprocal of the intended values, and a second bug sat in the calculation itself. Neither showed up in ordinary output. Researchers only found them by running a calibration test against thousands of synthetic datasets with known answers. Philip Ewels, who led RustQC, described the agents as “eloquent, convincing, and confidently wrong in ways that are easy to miss,” and never let a model judge its own accuracy.
What is behind this. The division of labour that worked was consistent across all eight projects: humans defined the goal, the success criteria and the tests, agents wrote the code. That ordering is the whole finding. A coding agent optimises for producing something that runs and passes whatever check it is given, which is a different target from producing something correct. When the check is a test suite the humans designed independently, the two targets line up. When the model is asked to assess its own work, they come apart, and the failure is quiet because the output still looks plausible. This is not unique to science. A study by METR found that actual project maintainers would reject roughly half of the solutions a widely used coding benchmark scores as passing. The report’s authors are careful to say their eight cases are not a representative study, and they flag a second problem: cheap rewrites are easy to produce and expensive to maintain forever.
What this means for you: the pattern transfers directly to anything you delegate to a model, whether that is code, a spreadsheet or a research summary. Decide how you will check the result before you ask for it, and make the check independent of the thing being checked. Asking a model whether its own answer is right is close to worthless, because the same reasoning that produced the error will endorse it. For anyone who writes code, the concrete version is that time saved on writing tends to reappear as time spent on review, and the projects here that went well are the ones where somebody built the test harness first.
Sources
A real macOS flaw sat unreported because Apple's bug bounty inbox was buried in AI-written junk
Apple has capped how many bug reports each security researcher can file, because a flood of AI-generated reports with invented vulnerabilities is clogging the queue. One Italian startup could not report a genuine macOS flaw as a result.
