benchmark

A benchmark is a standardised test used to measure how well an AI model performs at some task — coding, reasoning, reading documents, and so on — so that different models can be compared on a level footing. Leaderboards built from benchmarks are a common way labs show off progress.

Benchmarks are useful but easy to over-trust. A model can be tuned to score well on a popular test without being better in real use, and the way a test is run can quietly distort the result. Researchers found, for example, that standard tests of AI agents understated them simply by cutting the agents off before they had enough time to finish.

The practical takeaway: treat benchmark numbers as a rough signal, not gospel, and weigh them against how a tool actually performs on your own work.