Same Seed, Different Bytes: Measuring Tool-Call Repeatability Before You Trust an Agent Eval (2026)

A team re-runs its agent eval after a one-line prompt change and the pass rate moves five points. They revert the prompt; the number does not come back. They blame the harness, then the model vendor, and eventually pin temperature=0 and seed=42 everywhere — and the number still moves. Nobody in that loop can answer the question that comes before all of it: at the sampling settings the harness already uses, is the emitted tool call even the same object twice? ...

November 15, 2026 · 11 min · baeseokjae