This article ships a small protocol that measures exactly that, and reports no model results of its own. It publishes a frozen four-task sandbox, a harness with a documented adapter contract, a scoring sheet and a fixture self-check whose raw output is recorded and hashed. The model-by-task matrix in the middle of this page is filled with not run, because no model was executed while writing it. That is the honest state of the evidence: the instrument is built and verified, the vendor numbers it compares against are third-party, and the first real row is generated by whoever points it at a served model.

The question a leaderboard cannot answer

Two published results are enough to show why the leaderboard number does not transfer. Moonshot’s own model card for Kimi K2 reports 65.8% pass@1 on SWE-bench Verified with bash/editor tools, single-attempt patches and no test-time compute, 47.3% on SWE-bench Multilingual under the same conditions, and 71.6% on the same benchmark when parallel test-time compute is used to sample several sequences and pick the best with an internal scoring model (Kimi K2 model card). The first and third numbers are the same model on the same benchmark: the difference is the harness around it.

The Aider polyglot leaderboard makes the same point with retries. Its raw data file lists, for the same 225 test cases and the diff edit format, first-attempt and two-attempt pass rates of 14.2% / 40.0% for Qwen3 32B, 13.8% / 41.8% for gpt-oss-120b (high), 20.4% / 59.1% for Kimi K2 and 38.7% / 70.2% for DeepSeek-V3.2-Exp (Chat) (Aider polyglot leaderboard data). A single number quoted from that table describes a retry policy as much as it describes a model.

Every cell above is a third-party or vendor figure, reproduced with its own harness and retry policy attached, and the Aider data file’s most recent rows are dated 2025-10-03. That file also records how its runs were served: each of the four rows reproduced above carries a hosted provider route in its command: field — aider --model openrouter/qwen/qwen3-32b, openrouter/moonshotai/kimi-k2, openrouter/openai/gpt-oss-120b --reasoning-effort high and deepseek/deepseek-chat — 11 rows in the file carry an openrouter/ command, and no row names a local endpoint, so the file contains no locally served comparison. Note what the table does not contain: a column for “finished the task on my machine, once, with no second attempt”. That column does not exist in public reporting, and it is the one that decides a local deployment.

Why the harness, not the weights, produces the number

The strongest published statement on this is also the oldest cited here. The SWE-agent paper attributes its state-of-the-art result — 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix at the time — to its agent-computer interface rather than to a different base model, and argues explicitly that interface design changes agent behaviour and performance (SWE-agent, arXiv:2405.15793). Read that as a measurement warning rather than a historical note: two harnesses wrapping the same weights are two different systems, and 2026 harnesses differ far more than the 2024 ones did.

Why completion has to be verified per task

Terminal-style agent benchmarks took the right structural decision. Terminal-Bench ships a task dataset and an execution harness, each task carries a test script that verifies whether the agent completed it and an oracle reference solution, and the project self-describes as beta with roughly 100 tasks (Terminal-Bench README). Machine-verified completion, per task, with an oracle to check the checker. The SWE-bench family does the same at larger scale: SWE-bench Verified is a 500-problem subset that real software engineers have confirmed are solvable, evaluated in a fully containerised Docker harness built for reproducible runs (SWE-bench README).

Both are the right shape and neither is the number you need. They are fixed, shared task sets scored on a common harness: useful for ranking the field, silent about whether the model that lands at position 20 in such a ranking can complete your three-file refactor on the GPU you own, at the quantisation that fits in your VRAM.

The gap between benchmark success and shipped work

The cautionary counterpoint is a randomised controlled trial. METR ran 16 experienced open-source developers through 246 real issues in repositories they maintain, and found that when AI tools were allowed the developers took 19% longer to finish, while the same developers had forecast a 24% speedup and still believed they had been sped up by 20% after the fact; METR argues that benchmarks may overstate capability because their tasks are self-contained and algorithmically scored (METR study). That trial used frontier hosted models and a small developer sample, so it says nothing specific about local open-weight models. It says something about measurement: self-reported productivity and benchmark position both diverge from observed end-state completion, which is the quantity this protocol records.

The protocol you can run

Everything below is published as files in the site repository, and it is small on purpose — four tasks, one sandbox, one harness, one scoring sheet:

• Frozen task set — tasks.json: edit a method, add a test module, fix a failing test, rename a method across every file. Each task declares its own scoped test suite, and the dataset carries a single 12-step budget that the harness applies per task.
• Frozen sandbox — sandbox/: a synthetic inventory package with a green tests/test_store.py and one deliberately failing tests/test_report.py, hashed as a tree in every run header.
• Harness — harness.py: the adapter contract, the run loop, the per-task verifiers and the scorer.
• Adapters — models.py: the deterministic doubles plus LocalOpenAICompatAdapter, which drives any OpenAI-compatible /v1/chat/completions endpoint (vLLM, llama.cpp server, Ollama /v1, LM Studio, TGI).
• Runner and score sheet — run_tasks.py writes transcript.json (the sanitized step log), matrix.csv (the scoring sheet) and result.json (run header with the task-set hash, the sandbox tree hash and the aggregates).
• Self-check — test_harness.py and fixture_report.py.

An adapter is one class with one method:

class TaskAdapter:
    name = "model-id + quantisation + endpoint"      # recorded in the run header
    def next_action(self, observation) -> dict: ...   # one action per call

It receives the task, the step index, the current repository as {relpath: text}, the last suite output and the steps left, and returns exactly one action: write, patch, run_tests, escalate or stop. run_tests returns feedback, which is what separates a single-shot number from a retried one — the scorer counts those runs so the retry budget is visible in the result instead of hidden inside it.

Scoring is per task and end-state, in three buckets: completed when every scored check passes, partially when at least one does (the method exists but the suite is red), failed when none does or the adapter escalated. The suite-success guards are recorded but deliberately excluded from the score, because the frozen tests/test_store.py is already green and “the suite passes” is not evidence that a task advanced.

Run it with three commands, standard library only:

python3 test_harness.py                                              # harness unit tests
python3 run_tasks.py --adapter reference --out "$TMPDIR/out"         # offline control
python3 run_tasks.py --adapter local --base-url http://127.0.0.1:8000 \
        --model <served-model-id> --out "$TMPDIR/out"                # a real model

Fixture self-check: proving the harness, not measuring a model

A scorer is only worth reading if a known defect moves its own column and nothing else, so the package ships five deterministic offline doubles and their recorded output. Running python3 fixture_report.py on 2026-10-13 (UTC) with Python 3.12.3 reported Ran 10 tests in <elapsed>s and OK — all ten harness tests passing, captured as fixture-tests.txt (sha256 55561cd37a0ba0f9a7b536b1342e5d849d08851866a675c8c46945cf53aa99aa), with the full score sheet, environment and code hashes in fixture-report.json and the run’s environment, commands and hashes in first-party-run.json.

The four doubles behaved as designed on that run. The reference double completed 4 of 4 tasks in a mean 0.143 s of harness time per task with 5 test runs and no escalations. The stall double (stops immediately) and the loop double (spins until the 12-step budget is exhausted, 48 test runs total) each scored 0 completed / 0 partially / 4 failed. The escalation double scored 0 completed and recorded exactly one escalation per task. The partial-rename double scored 3 completed and 1 partially, with the partially-scored cell being only the rename task, where the definition was renamed and the call sites were not.

That is the whole claim: a synthetic control scores 4 of 4, a single-defect double loses only its own task. These are fixture results produced by scripted in-process doubles to validate the instrument; no model was loaded, no GPU time was used, and not one of these numbers measures Qwen, Kimi, gpt-oss, DeepSeek or any other system.

Results: not run

The matrix this article exists to publish is empty, and it says so cell by cell.

No local model was executed for this article. Filling a cell requires a served model, a recorded endpoint and sampling parameters, a transcript and the sandbox tree hash — otherwise the number is an anecdote with a table around it. Until those exist, the row is not run.

Failure taxonomy: what stalling looks like

The instrument is useful mainly because the failure modes are distinguishable, and each one maps to a column rather than a vibe:

• No progress — the adapter stops immediately or edits files that no check reads. Shows up as steps low and every scored check false.
• Looping — the same action repeats until the step budget is exhausted. Shows up as steps at the budget with a high test_runs count and red checks.
• Red suite after an edit — the method exists but the suite fails: partially, and the failing test name is in the transcript.
• Partial refactor — the rename lands in the definition and not in the call sites. This is the single most common way a multi-file task “succeeds” in a transcript and fails in a repository, and it is why the rename task checks call sites separately from the definition.
• Escalation — the model asks for help. Counted, not hidden: a high escalation count with a good partial rate is a different product decision from a low completion rate with none.

Reproducing this on your own hardware

1. Pin the sandbox: copy the tree, record harness.tree_sha256(sandbox) before the first run, and never edit it in place.
2. Serve the model and record the endpoint, model id, quantisation, context length and sampling parameters in the adapter’s name. Those strings are the run header; a result without them is not reproducible.
3. Implement next_action against the endpoint’s chat API, and keep temperature at zero with a fixed seed for the first run so the variance you see is the harness, not the sampler.
4. Add tasks rather than changing them. Each new task needs a scored check that a script can evaluate and a suite scope that no unrelated frozen failure can mask.
5. Publish the transcript and the matrix together. One headline completion rate hides which of the five failure modes you are paying for; the transcript is what lets a second person disagree with the score.

If your VRAM cannot serve the model you want to test, change the quantisation and record it — the point of the protocol is that the number belongs to a configuration, and an honest smaller configuration beats an unattainable bigger one.

When a managed agent is still the right call

Two quantities decide it, and this protocol produces both. Escalations per task is the first: a local model that completes most tasks but escalates on the remainder still needs a human in the loop, and that human’s time is the real cost. Test runs per task is the second: the gap between the first-attempt and second-attempt columns in the published tables above is the price of feedback, and on a local rig that price is paid in wall-clock time rather than in tokens. Where a managed agent remains cheaper per completed task under those two measures, that is a measurement result, not a preference — and the harness is what makes the comparison fair, because it scores both sides on the same frozen tasks with the same verifier.

Limitations and what this protocol cannot tell you

• No first-party model result exists here. The matrix is not run; every external figure is attributed to its own harness, date and retry policy.
• The sandbox is synthetic and sequential. Tasks run in one working copy in order, so an early failure is inherited by later tasks; the sandbox is a controlled fixture, not a snapshot of a real repository.
• A four-task set is a probe, not a score. It detects “does this model finish this kind of work”, not a percentage to quote.
• Borrowed numbers are never ours. The Kimi K2, Aider polyglot, Terminal-Bench, SWE-bench and METR figures are the vendors’ and authors’ results on their own harnesses, reproduced here with their scope attached.
• Demand evidence is directional only. The topic’s measured demand is a small adjacent query row in the operator’s own search analytics plus adjacent serving-capacity queries; no search volume, difficulty or conversion number is asserted.
• Unknown cost stays unknown. No monetary cost is claimed anywhere in this article; no paid provider was used and no price estimate is inferred from a zero balance.

Primary sources

Fetched and re-checked live on 2026-10-13 (UTC) while writing. The response bodies were saved and hashed as a private local author verification during this article’s own research; they are third-party downloads and are deliberately not republished. The files a reader can obtain are the original protocol artifacts linked above, together with the recorded self-check output.

• Kimi K2 model card (65.8% SWE-bench Verified with bash/editor tools; 47.3% Multilingual; 71.6% with parallel test-time compute) — https://huggingface.co/moonshotai/Kimi-K2-Instruct/raw/main/README.md
• SWE-agent paper (agent-computer interface design; pass@1 12.5% SWE-bench, 87.7% HumanEvalFix) — https://arxiv.org/abs/2405.15793
• Aider polyglot leaderboard data (first-attempt vs two-attempt pass rates, diff format, 225 cases) — https://raw.githubusercontent.com/Aider-AI/aider/main/aider/website/_data/polyglot_leaderboard.yml
• Terminal-Bench README (task dataset plus execution harness, per-task test script, oracle solution, beta ~100 tasks) — https://raw.githubusercontent.com/laude-institute/terminal-bench/main/README.md
• SWE-bench README (500-problem Verified subset, containerised Docker harness) — https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md
• METR randomised controlled trial (16 developers, 246 issues, 19% slowdown) — https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/