SimpleQA
A benchmark that asks models short factual questions with one correct answer, and is much harder for them than it sounds.
SimpleQA is a test that does exactly what the name suggests: it asks a model short, unambiguous factual questions and checks whether the answer is right. No reasoning puzzles, no code, no essays. Just things like who did what in which year.
Models are startlingly bad at it. As of mid 2026 the leading scores sat around 53 percent, from models that can nearly ace olympiad-level mathematics competitions. The gap is the point. A language model stores facts as a side effect of learning to predict text, not as a lookup table, and squeezing more reasoning ability into a smaller model tends to squeeze factual recall out. That is the main argument for retrieval, where you hand the model the source document instead of trusting what it remembers.