Meta's Answer to Wasted GPU Time: Ask a Model Which Experiment to Run
Researchers from Meta FAIR, Oxford and UCL introduced AI Research Preference Models, which rank unexecuted machine learning experiments against each other rather than predicting scores, to decide where to spend compute.
A paper from Meta’s FAIR lab with the University of Oxford and University College London tackles a problem that sounds mundane and costs a fortune: AI research agents can propose far more experiments than anyone can afford to run. Thinking up a candidate takes seconds. Actually training the model to find out whether it works can take hours or days of GPU time. So the bottleneck is not ideas, it is choosing.
Their proposal is called an AI Research Preference Model, or RPM, and the clever part is what it refuses to do. It does not predict how well an experiment will score, because the team found language models unreliable at forecasting absolute numbers or execution outcomes. Instead it treats the problem as ranking: given a set of candidate experiments that have not been run yet, put them in order and pick the top one. Comparing two options is a much easier question than estimating either one’s value, and it is a trick that shows up all over machine learning. The team built two versions from frozen pretrained language models, meaning models used as they are without further training. One reasons purely over the candidate plans, the code, and the results of experiments already run. The other additionally runs small-scale pilot experiments before ranking, spending a little compute to save a lot.
Why this matters more than it sounds. Almost every story about AI research automation is about agents that generate and execute. Very little is about agents that decline. As labs point more autonomous systems at their own research, the constraint stops being how many ideas the system can have and becomes how good its taste is, and taste is exactly what nobody has known how to measure. Framing it as preference ranking makes it something you can train and evaluate. It also connects directly to the numbers OpenAI published this week about agent runtime exceeding human labour three to one: runtime is only worth anything if the experiments were worth running, and a lab burning three agent-days per human day has an obvious interest in not wasting them.
What this means for you. For most readers this is background rather than something to act on, and it is fine to file it that way. But the underlying idea travels well beyond GPU clusters. If you have ever asked a model to score your options out of ten and got back a suspiciously flat set of eights, you have met the same failure the researchers did. Asking it to rank instead, or to compare two things at a time, tends to produce answers you can actually use. That is a free improvement to how you prompt, and it comes from the same observation as the paper.
Sources
Source: https://arxiv.org/abs/2608.13940
OpenAI Says Its Coding Agents Now Work Three Days for Every Human Day
OpenAI published internal figures showing its research organisation uses 3.1 agent-workdays of coding-agent runtime for every human workday, and says it has hit its goal of an automated research intern by September.
