AI Evaluation

EdotEnv (YC S26) is a startup founded by former quants Rui and Michael that builds self-improving reinforcement learning environments from quantitative trading workflows, designed specifically to evaluate and train LLM agents. Unlike static benchmarks that saturate as models improve, EdotEnv uses live market dynamics where alpha decays 30-50% per year, creating a continuously evolving difficulty curve that keeps evaluation meaningful even as frontier models advance. What Is EdotEnv and Why Does It Matter for LLM Research? EdotEnv launched on Hacker News in August 2026, receiving 39 points and 34 comments from the AI and quant finance communities. The company’s tagline — “Environments for intelligence that adapts” — captures its core thesis: the most useful benchmarks for evaluating LLM agents are those that get harder as the models get better. Traditional NLP benchmarks like MMLU, GSM8K, and HumanEval have all experienced significant saturation, with frontier models now scoring above 90% on many of them. EdotEnv proposes a radical alternative: use real financial markets as the evaluation environment, where the difficulty level is set by the collective intelligence of all market participants and naturally increases over time. ...

Introduction — The Year AI Agent Benchmarks Broke If you have been following AI agent benchmarks in 2026, you have likely seen headline numbers that look too good to be true. A tool called BenchJack scored 100% on SWE-bench Verified, SWE-bench Pro, and Terminal-Bench without solving a single task — most runs never even called a language model. OpenAI retired SWE-bench Verified after discovering that 59.4% of hard failed tasks had broken test cases rejecting functionally correct patches. And the same frontier model scored 64.7% on one wrapper and 57.5% on another — a 7.2-point gap from scaffolding alone. This is the state of exploiting AI agent benchmarks in 2026: a system where the incentives to publish high scores have outpaced the rigor of the evaluations themselves. ...

You ran the same model on the same benchmark twice and got different scores. Then you changed one word in the prompt and got a different ranking. Then you realized the benchmark questions themselves have errors. Welcome to LLM benchmark variance — the problem that makes most published benchmark scores less reliable than they look. I’ve been evaluating LLMs for production deployment over the past year, and I’ve learned that benchmark scores are not the stable, objective measurements most people assume they are. A model that scores 87% on MMLU one week can score 82% the next week with a different instruction template. A 3-point lead on a micro-benchmark can flip entirely when you run the full evaluation. And 6.49% of MMLU questions — the most-cited benchmark in AI history — contain ground truth errors. This article breaks down every major source of benchmark variance I’ve encountered, with the numbers and research to back it up, and what to do about it. ...

Microsoft ASSERT is an open-source agent evaluation framework that turns written AI policies, product requirements, and safety rules into executable tests. For developers, the value is practical: instead of debating whether an agent “mostly follows policy,” ASSERT gives you repeatable scenarios, metrics, traces, and scorecards you can run before release. What Is the Microsoft ASSERT Agent Evaluation Framework? Microsoft ASSERT is a requirement-driven evaluation harness for AI agents and LLM applications that converts natural-language specifications into executable evaluations. ASSERT stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing, and Microsoft describes it as open source and framework-agnostic for the estimated 6 million to 13 million generative AI developers working across today’s agent ecosystem. The framework starts with written intent, such as a product requirement, policy document, system prompt, or launch checklist, then helps generate scenarios, datasets, metrics, and scorecards that can be run against hosted models, Python callables, or traced agent systems. The key idea is simple: agent behavior should be tested against your own requirements, not only against generic benchmarks. ASSERT is best understood as policy-as-evaluation for teams that need repeatable evidence before deploying autonomous workflows. ...

Claude Opus 4.6 resolves 80.8% of real GitHub issues on SWE-bench Verified while GPT-5.5 leads Terminal-Bench 2.0 at 82.7% — numbers that mean something precisely because they come from independent evaluation pipelines, not vendor press releases. Choosing an LLM in 2026 without understanding how these benchmarks work is like buying a server based solely on manufacturer marketing sheets. This guide covers the LM Council evaluation framework, the top independent leaderboards developers actually rely on, and how to read benchmark results without getting misled. ...

LLM benchmark scores flood every model release announcement — but as of 2026, most of those scores tell you almost nothing useful. This guide explains which benchmarks still matter for developers, which are saturated or compromised, and how to pick the right signal for your actual workload. Why LLM Benchmarks Matter for Developers (And Why Most Are Now Useless) LLM benchmarks are standardized test suites that measure model capabilities across defined tasks — coding, reasoning, math, or domain knowledge — so developers can compare models without running every candidate through their own production workload. Done right, they save weeks of internal evaluation. Done wrong, they create a false confidence loop where a model scores 92% on a benchmark and then fails on the first real customer ticket you throw at it. As of May 2026, the benchmark landscape has split sharply: a small set of hard, contamination-resistant evaluations still provide genuine signal, while the legacy suites — MMLU, HumanEval, GSM8K — have been effectively retired by the community because frontier models have saturated them. MMLU, once the canonical academic reasoning suite, now sees frontier models cluster at 85–90% with no meaningful spread between Claude, GPT, and Gemini variants. HumanEval similarly sees 93%+ scores across top-tier models as of April 2026. When every serious model aces the same test, the test stops being useful. The benchmarks worth tracking now are the ones that are still hard enough to differentiate — and that requires understanding why they’re hard. ...