A Robot That Only Tracks Objects Predicts the Wrong Move. Somebody Finally Measured By How Much
Adding beliefs, intentions and social norms to a world model lifted action prediction from 63.3 to 87.9 F1. GPT-4.1 with the framework beat GPT-5.6 Sol without it.
Here is a scene. Someone puts their cup in the cabinet while you are out of the room. You come back. Where do you look? At the table, of course, because that is where you left it. A system that only tracks objects gets this wrong every time, because physically the cup is in the cabinet and that is all it sees.
A paper published on 22 August turns that thought experiment into a measurement. The authors argue that today’s world models, meaning systems that predict how a scene changes when someone acts, only model the physical layer: objects, positions, motion, what is hidden behind what. Sora, Genie 3, JEPA and Marble all get grouped into the same family and criticised for the same omission. What the people in the scene believe, want or consider socially appropriate never enters the state at all.
Their framework, Mental World Modeling, adds those variables: beliefs, attention, goals, intentions, emotions, norms and relationships. Every action gets split into a physical carrier and a mental payload. Sliding a cup across a table is the same gesture whether it is an apology, a deception or an act of care, and only the mental layer tells them apart. The authors are careful to say they are not simulating consciousness. Mental states here are hypotheses inferred from behaviour, and systems should keep that uncertainty visible.
The numbers
To test it they built MENTIS, a pipeline that needs no extra training, and Menti-Bench, a set of 448 decision scenes made of 320 text descriptions, 100 picture stories and 28 sound-and-video clips. Each scene offers six response options plus a human reference answer.
Across eight models, answering directly scored 63.3 on F1, a standard accuracy measure. Asking the same question six times and taking the most common answer got to 77.9. The full pipeline reached 87.9. Humans hit 98.5 on the same protocol.
The most striking result is that you cannot buy this with extra compute. The weakest model running the framework, GPT-4.1 at 84.9, beat the strongest model answering directly with repeated sampling, GPT-5.6 Sol at 83.6. Structure beat raw capability. Remove the mental channel and models lose 12.1 points on average; remove the physical channel and they lose 16.5. Both are needed, and predicting them together rather than separately is worth another 6.4.
The gains land where the theory says they should: 26.4 points on scenes between people, only 14.0 on object-focused ones. About 80 percent of the remaining gap to humans comes from one step, simulating how the coupled physical and mental state changes next.
What this means for you: Nothing you can use today. But it explains a frustration you may already recognise, when an assistant does exactly what you literally asked and completely misses what you meant. That gap is not a politeness problem, it is a modelling problem, and this paper is one of the first to put a number on it. For anyone following where AI is heading after chatbots, world models are the big bet, and the field has not even agreed on a definition yet.
Sources
- MENTIS reference implementation and Menti-Bench, Mental World Modeling project
- World models that ignore human beliefs predict the wrong actions, The Decoder
Anthropic Hired the Man Who Built Google's AI Chips, and It Is Not Subtle About Why
Amir Salek shipped seven generations of Google's TPU. He now joins Anthropic's compute team, while Broadcom lines up more than 60 billion dollars of debt for chip capacity.
