Benchmark contamination
When the questions from a test have leaked into a model's training data, so a high score reflects memory rather than ability.
Benchmarks are supposed to measure what a model can work out. Contamination is what happens when the answers were sitting in the training data all along, so the model recognises the question instead of solving it. Because models are trained on enormous scrapes of the public web, and because benchmarks are published on the public web, this happens constantly and is often impossible to prove after the fact.
The practical effect is score inflation that looks like progress. A 2026 study of 60 benchmarks found nearly half showed signs of saturation, meaning scores had bunched near the top in a way that no longer separated strong models from weak ones. The fix is a contamination-resistant design: use material published after the training cutoff, strip identifying details that would let a model recognise the source, and freeze each test case. The Reconstruction benchmark, published in August 2026, is a strict example, and frontier models scored far lower on it than on comparable tests with richer inputs.