DeepSWE
A benchmark that tests whether a model can carry a long software engineering task through to a working result.
DeepSWE tests something narrower and more useful than “can it write code”. It measures whether a model can take a real software task, one that needs many steps across several files, and finish it without losing the thread halfway through. Reading the existing code, working out what to change, making the change, running the tests, fixing what broke.
That endurance is the thing that separates a helpful autocomplete from something you can hand a task to and walk away from, which is why scores on it have been jumping so sharply between model generations. A caveat that applies to all coding benchmarks: they run on curated problems in tidy repositories, and your codebase is neither.