Best of N Agent Verifier: How to Pick the Coding Agent Run That Actually Solved the Task

Best of N Agent Verifier: How to Pick the Coding Agent Run That Actually Solved the Task

Run N isolated candidate agents in separate git worktrees, capture each trajectory and patch, then rank them with a verifier that returns a continuous logprob score rather than a discrete grade. Terminal-Bench V2 went from 83.1% Pass@1 to 86.5% verifier-selected against a 92.1% oracle — the model had already solved it; selection was the bottleneck. That gap between what the model can do and what you can identify is the entire subject of this guide. The rest of it covers why a plain LLM judge silently fails, how to turn a 20-token score distribution into a calibrated reward, how to rank 8 candidates without paying for 28 pairwise comparisons, what worktrees do not isolate, how many samples are actually worth paying for, and where optimistic best-of-N claims collapse under measurement noise. ...

October 1, 2026 · 31 min · baeseokjae
Best-of-N Coding Agents With LLM-as-a-Verifier in Oh My Pi

Best-of-N Coding Agents With LLM-as-a-Verifier in Oh My Pi

Best-of-N coding agents run the same task N times in isolated workspaces, then use an LLM-as-a-verifier to score every full trajectory and select the winning patch. In Oh My Pi, the omp-best-of plugin does this with /best-of --n 5 --apply "Fix the failing authentication test", defaulting to 3 candidates and selection-only mode. That is the short answer. The rest of this guide is the part that decides whether the technique saves you money or burns it: what selection can and cannot fix, why the maintainers’ own benchmark numbers are much smaller than the headline claims circulating about them, and how to choose N without paying for a fourth opinion nobody needed. ...

October 1, 2026 · 17 min · baeseokjae