CWE-Bench
An external benchmark run by Collinear that measures how well AI models can write correct patches for known software vulnerabilities.
Finding a security flaw and fixing it are different skills, and CWE-Bench tests the second one. Models are given vulnerable code and scored on whether the patch they write actually closes the hole without breaking anything else. Results are usually quoted as pass@1, meaning the share fixed correctly on the first attempt.
It has become a common reference point in 2026 because labs started shipping models aimed specifically at defenders. Google reported 47.2% for Gemini 3.8 Flash Cyber, close to the leading frontier model’s 47.8% but at a much lower cost per attempt, which is the sort of trade-off the benchmark is good at exposing.