Best of N Agent Verifier: How to Pick the Coding Agent Run That Actually Solved the Task
Run N isolated candidate agents in separate git worktrees, capture each trajectory and patch, then rank them with a verifier that returns a continuous logprob score rather than a discrete grade. Terminal-Bench V2 went from 83.1% Pass@1 to 86.5% verifier-selected against a 92.1% oracle — the model had already solved it; selection was the bottleneck. That gap between what the model can do and what you can identify is the entire subject of this guide. The rest of it covers why a plain LLM judge silently fails, how to turn a 20-token score distribution into a calibrated reward, how to rank 8 candidates without paying for 28 pairwise comparisons, what worktrees do not isolate, how many samples are actually worth paying for, and where optimistic best-of-N claims collapse under measurement noise. ...