Exploiting AI Agent Benchmarks: The 2026 Crisis of Trust in Agent Evaluation
Introduction — The Year AI Agent Benchmarks Broke If you have been following AI agent benchmarks in 2026, you have likely seen headline numbers that look too good to be true. A tool called BenchJack scored 100% on SWE-bench Verified, SWE-bench Pro, and Terminal-Bench without solving a single task — most runs never even called a language model. OpenAI retired SWE-bench Verified after discovering that 59.4% of hard failed tasks had broken test cases rejecting functionally correct patches. And the same frontier model scored 64.7% on one wrapper and 57.5% on another — a 7.2-point gap from scaffolding alone. This is the state of exploiting AI agent benchmarks in 2026: a system where the incentives to publish high scores have outpaced the rigor of the evaluations themselves. ...