Vibe check
An informal, unscored test of an AI model where one person tries something unusual and judges the result by eye.
A vibe check is the opposite of a benchmark. A benchmark is a fixed set of tasks with known right answers, so scores can be compared between models and tracked over time. A vibe check is one person handing a model something strange and looking at what comes back. No score, no comparison, no repeat runs.
It has an obvious weakness: you only ever see the attempt that was worth posting, never the four that failed, and nobody can check the result independently. It also has a real strength. Public benchmarks leak into training data and get optimised for, so a high score slowly stops meaning what it did. A weird one-off task catches something scores miss, namely whether a model can hold a long, vaguely described job together without falling apart. Treat vibe checks as anecdotes, useful ones, but not as measurements.