What a coding agent harness actually is (and what it is not)

A coding agent is a model plus a harness. That framing is now standard, but the term itself is overloaded. Martin Fowler’s harness engineering article (Boeckeler, 2 April 2026) narrows it to two bounded contexts: the built-in harness that ships inside a coding agent (its context assembly, tool exposure, and execution loop) and the outer harness a team builds around that agent (instructions, linters, tests, review gates, CI rules).

This article is about the first one. Everything below concerns the runtime machinery that wraps a large language model during a coding run:

• Planning — whether the agent produces an explicit plan before acting on the repository.
• Action space — whether the agent gets a predefined tool set (file read, file write, search, patch) or just bash.
• Context management — how the conversation is compacted, elided, summarized, and recovered as a long run approaches its window limit.

What the harness is not: it is not the model weights, it is not the verifier, and it is not the benchmark. Those are the other two-thirds of the number you read on a leaderboard.

Vendor anatomy confirms the split. The GitHub Copilot in VS Code write-up (15 May 2026) describes the coding harness as four concrete responsibilities — context assembly, tool exposure, the agent loop, and the round-versus-turn distinction. A source-code study of eleven production systems (arXiv:2609.00006, 15 July 2026) analyzed roughly four million lines across Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw. It found no agent runtime that imports a general-purpose agentic framework and none that retrieves code with vector embeddings, then distilled 13 observations, 29 recurring design patterns, 18 design recommendations, and a 90-line minimum-viable-harness scaffold. Harnesses are built, not imported.

The 2026 evidence: how big is the harness effect, really?

Four independent lines of evidence converged on “as large as the model term,” and they measured it in different ways.

The Claw-SWE-Bench paper is the cleanest of these because harness is a controlled experimental variable: 350 tasks across 8 languages and 43 repositories. Five claws running on GLM 5.1 span 60.9% to 73.4% — a 12.5-point spread with the model held fixed. On Qwen 3.6-flash the spread widens to 38.6%–66.0%. Weaker models are more sensitive to harness quality, not less.

The position paper “Stop Comparing LLM Agents Without Disclosing the Harness” (arXiv:2605.23950) supplies the ranking-instability evidence that makes this a methodological problem rather than a curiosity. Claude Opus 4.5 reaches 45.9% on SWE-bench Pro under the standardized SEAL scaffold, but 55.4% under Claude Code. HAL reports same-model cross-scaffold swings of nearly 48pp on SWE-bench Verified Mini, against a mere 4.9pp spread spanning six frontier models under a standardized scaffold. That inversion — where the harness term swamps the model term — is the empirical core of what that paper calls the Binding Constraint Thesis.

Third-party tracking agrees in the same direction. LangChain moved Terminal-Bench from 52.8% to 66.5% (+13.7pp) by changing only the system prompt, tool choice, and execution flow, with the model fixed. Anthropic reports that resource configuration alone swings scores about 6pp, and that differences below 3pp are noise. Practical consequence: any leaderboard delta under three points is not a finding.

There is also a synthesis worth citing directly. A controlled factorial by Zhang et al., compiled in the Deep Feed’s June 2026 analysis, found harness-induced variance exceeding model-induced variance by 7.8× on a SWE-bench subset, with 6 of 9 model-pair rankings reversing under a different scaffold. Model-paper advances in the same period moved scores 2–4 points; harness swaps moved them 6–54.

How was the harness effect measured? Inside 176 matched settings, 4 models, 2 benchmarks

The anchor study for component-level design decisions is “An Empirical Study of Harness Design for Coding Agents” (arXiv:2609.20804v1, 17 September 2026). Where Claw-SWE-Bench treats the harness as one variable, this study decomposes it.

It holds the execution loop fixed and varies three components across 176 matched experimental settings — 22 settings per model-benchmark pair, spanning 20 context-management settings (five tiers from T0 to T4 at 32k, 64k, 96k, and 128k windows) plus one planning ablation and one bash-only ablation (Section 3.1).

Temperature zero and a fixed loop make the comparisons matched rather than anecdotal. Note the study’s own stated limits up front, because they constrain every number below: single runs per setting, and both the planning and action-space ablations were run only at the T4 tier with a 128k baseline.

The five context tiers are the backbone of the design:

Finding 1: Is context management a reasoning upgrade or budget insurance?

This is the most actionable result in the paper, and it is routinely misread. The value of context management is almost entirely a function of how likely your runs are to run out of window.

The managed-minus-T0 success gap on SWE-Bench falls from 35.7pp at 32k to 15.9pp, 5.5pp, and 2.7pp at 64k, 96k, and 128k respectively. On Terminal-Bench the same gap goes 9.5pp, 7.5pp, 4.8pp, 2.8pp (Section 3.2, Tables 3 and 4).

The mechanism is overflow, and the numbers are unambiguous. The unmanaged T0 window-overflow rate falls from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench as the budget grows. Every managed tier — T1 through T4 — overflows on exactly zero tasks at every budget (Section 3.2, Figure 3).

That is the whole story. Context management is not making the model smarter; it is preventing a catastrophic failure mode where the run blows its window.

The extreme cases show how much of a tight budget is really a harness problem. Nemotron-3 30B scores 0% on SWE-Bench at 32k with no context management (T0) versus 20.6% with elision only (T1); at 128k the same model goes 24.8% (T0) versus 25.0% (T1), and the harness difference nearly vanishes. Nemotron-3 550B on SWE-Bench at 32k goes from 0% at T0 to 58.4% at T3 — a 58.4-point swing from one harness component at a tight budget (Table 3).

The design decision this produces: buy window capacity or buy compaction machinery, but paying for both is the common mistake. If you are running at 128k, the marginal value of a sophisticated compaction stack measured against your own baseline is under three points. If you are at 32k, it is the difference between a working agent and a broken one.

Finding 2: Should you elide, summarize, or build lossless recall?

Two sub-results here, and the second one is a warning about shipping features you never measure.

First, ordering matters. Staged elision-then-summarization (T4) matches the accuracy of T1 through T3 while posting the lowest cost in 7 of 8 model-benchmark panels and the lowest mean cost per task at every window budget (Section 3.2, Figures 4 and 5). The reason is arithmetic: early cheap elision removes material before it can be sent to a costly LLM summarization call. You pay for summarization only on what survived truncation.

Second, the lossless recall feature did not pay off. T2 (elision plus recall_event) beats T1 (elision alone) in 15 settings, loses in 14, and ties in 3, across 32 model-benchmark-window comparisons — an equal-weight mean difference of −0.36pp. More damning than the null result is the invocation data: 36 of 64 T2/T4 settings (56.3%) never called recall_event at all, the median invocation rate is zero, and mean calls per task fall from 0.540 at 32k to 0.069, 0.011, and 0.007 at 64k, 96k, and 128k (Section 3.2 and Table 13).

The feature was built, documented, and almost never used. If you only looked at outcome metrics, you would never learn that. Instrument invocation rates alongside outcomes.

There is a competing design worth knowing about, because it argues the opposite of “summarize more cleverly.” CliffCompaction (arXiv:2609.26779, 22 September 2026) cuts cost by up to 50% under a bounded context while maintaining or improving Terminal-Bench performance, and adds over 10pp on Terminal-Bench for less than the cost of two full-context runs. Its rule is strict: only truncate or drop content, never rephrase, and never compact a compaction — prior compacted output is discarded rather than re-summarized.

A third position is to avoid compaction entirely. NVIDIA’s NOOA reports 82.2% on SWE-bench Verified with GPT-5.5 using roughly half the tokens and LLM calls of comparison harnesses, and needs no context compaction because tool results pass by reference instead of being serialized into the window (NVIDIA Developer Blog, “Six Agent Harness Capabilities for Higher Model Performance”). Recall this interacts directly with Finding 1: pass-by-reference and a large window are substitutes for the same failure mode.

Finding 3: Do you actually need a planning step?

Planning is not universally good or universally bad. Its sign depends on model capability, and the mechanism explains why.

For Nemotron-3 30B, planning adds 11.6pp on SWE-Bench and 4.5pp on Terminal-Bench, at higher cost. For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning cuts SWE-Bench cost by roughly 30% and 32% while success drops 2.0pp and 0.4pp (Section 3.2, Figure 6).

The mechanism is in where runs stop. With planning off, 68.6% of Nemotron-3 30B SWE-Bench runs ended without an edit and 58.4% stalled in the Localize phase. With planning on, those collapse to 27.8% and 10.4%. For the strongest models, the turns planning removes are mostly post-edit verification — useful-looking work that was not changing the outcome (Table 6).

One more number reframes the planning debate. Fewer than 3% of runs ended without editing code regardless of planning. Planning is not rescuing otherwise-doomed attempts across the board; it is trimming repetition once the fix already exists, and preventing early abandonment for weak models that cannot find the problem without a scaffold.

The practical rule: turn planning on when your model abandons tasks before editing; turn it off when your model succeeds anyway and you want the ~30% cost back.

Finding 4: Should you ship predefined tools or bash-only?

The action space is the component with the sharpest crossover, and the crossover point moves with both model capability and task type.

The same Mistral model needs opposite action spaces on two benchmarks. That is the whole point: there is no fixed answer, because the correct action space is a function of the model’s shell competence and of whether the task is shell-centric.

The failure mechanism for bash-only on weak models is instructive. 66% of Nemotron-3 30B bash-only Terminal-Bench trajectories terminate after out-of-interface tool emissions — the model emits learned tool-call patterns that the bash registry cannot resolve, and the run dies. Average trajectory length collapses from 71 turns to 15. On models that can drive a shell, the same restriction produces the opposite effect: bash-only trajectories issue 32% fewer calls on SWE-Bench and 24% fewer on Terminal-Bench, consistent with denser composite shell commands (Section 3.2).

The minimalism baseline makes the ceiling visible from the other direction. mini-SWE-agent is roughly 100 lines of Python with bash as its only action space. It posted about 65% on SWE-bench Verified in mid-2025 and its own docs report 74%+ with newer frontier models. In the Lita comparison it trails OpenHands by 3.2pp on Claude Sonnet 4 (64.8 vs 68.0) but by only 0.2pp on Claude Opus 4 (67.6 vs 67.8). Scaffolding’s marginal value shrinks as model capability rises — which is exactly the crossover the ablation study measures.

Why does each component work? The trajectory-level mechanism

Scores tell you that a component helps. Trajectory shape tells you why, which is what you need when debugging your own harness. The anchor study’s Section 4 gives each component a distinct signature:

Context management does not make the agent reason differently. It lets the same reasoning run longer before the window kills it — consistent with Finding 1’s overflow numbers. Planning does not change what the agent does; it changes whether the agent gets to the point of doing it. Action space does not change whether the agent acts; it changes how many turns each action costs.

This is also why “the harness effect” is better understood as sabotage capacity. A weak harness raises per-step difficulty repeatedly across a long loop, so the ceiling is set by how badly the harness obstructs a capable model. As that easy sabotage gets engineered out, expect the harness term to shrink — so build measurement capability rather than a permanent “best harness” claim.

How do you match harness design to model, task type, and budget?

Everything above collapses into a decision procedure. Use it as a starting configuration, then validate on your own workloads.

Two configuration principles cut across the table. First, cost belongs on the same chart as accuracy — bash-only’s 30–53% cost reduction for capable models only exists in the cost column, and an accuracy-only reading of the same data produces a different and worse decision. Second, every component has an optimal budget, and paying for the same insurance twice — a 128k window and a full compaction stack — is the single most common design error the 2026 evidence exposes.

A supporting layer sits outside these three components but changes the same numbers: the outer harness of linters, tests, and structural checks. Fowler’s split into guides (feedforward controls) and sensors (feedback controls) is the right vocabulary, and the strongest sensors emit LLM-optimized signals — linters that include self-correction instructions, not just error codes. The concept of harnessability is the corollary: strongly typed languages, clear module boundaries, and opinionated frameworks raise an agent’s chance of success before any runtime change is made. OpenAI’s harness-engineering report describes layered architecture enforced by custom linters and structural tests, plus recurring “garbage collection” passes to remove drift.

How do you measure your own harness without fooling yourself?

The measurement problems in this literature are as instructive as the results, because they are the mistakes teams repeat internally.

Disclose model, harness, context budget, and step budget together. A published number is model × harness × verifier, and two of those three are usually unreported. The position paper’s core complaint (arXiv:2605.23950) is that leaderboards report each model under its own best-tuned scaffold — which is neither a locked-harness comparison nor a factorial decomposition. Every ranking it examines is unstable under that regime.

Know which regime you are in.

Zhang et al.’s 7.8× harness-over-model variance result is a factorial finding and is not transferable from a locked-harness leaderboard.

Remember the verifier is the third term. UTBoost found SWE-bench tests weak enough that wrong patches passed, with corrections affecting 40.9% of SWE-bench Lite entries and 24.4% of Verified entries (Yu et al., ACL 2025). Roughly two in five of the gradings you have been comparing may have been wrong. If your harness work is graded by a suite you have never audited, you are optimizing a noisy objective.

Respect the noise floor. Anthropic’s report that differences below about 3pp are noise is a useful default. The anchor study’s own limits point the same way: single runs per setting, and planning and action-space ablations only at T4 with 128k. Run repeats before you ship a component based on a two-point delta.

Instrument invocation, not just outcomes. The recall_event result is the canonical case: an entire feature shipping to production with a median invocation rate of zero.

Use trajectory shape as a diagnostic. Length for context problems, stop-location for planning problems, call density and command size for action-space problems. A score tells you something changed; the trajectory tells you which component changed it.

What does the evidence not settle (limits, contradictions, and open questions)?

The honest position is that the 2026 literature is strong on magnitudes and weak on universals.

• No universal best harness. The anchor study explicitly limits its claims to the components and implementations it tested, with single runs per setting and planning/action-space ablations only at the T4/128k baseline. Do not generalize its tier results to tiers you did not test.
• The effect size is probably inflated by immaturity. Harness-induced variance exceeds model-induced variance by 7.8× in one factorial, but that is a statement about current harnesses. Harness effects are large precisely because easy sabotage is still shipping.
• Compaction design is genuinely unresolved. LLM summarization (the anchor study’s T3/T4), truncate-only CliffCompaction, and pass-by-reference NOOA represent three incompatible philosophies, each with supporting numbers. The disagreement is about whether losing information is worse than paying to compress it.
• Benchmark coverage is narrow relative to practice. SWE-Bench Verified and Terminal-Bench 2.1 are repository-patching and shell tasks. Frontend, data, and multi-repo agent work are barely represented.
• The cost axis is under-tabulated across studies. The anchor study is unusually good here; most harness comparisons still report only success rates, which is why bash-only’s 30–53% savings stays invisible in casual reading.
• Vendor and third-party numbers come from different regimes. The 19.1% → 73.4% adapter swap, the 27.4pp vs 29.4pp split, LangChain’s +13.7pp, and Kimi K2 Thinking’s +15pp are not a single controlled ladder. Treat them as convergent signals about magnitude, not as a comparable scale.

FAQ

Is the harness more important than the model?

It depends on the comparison regime, and both answers are correct in their own regime. Under a locked harness, the model term dominates. In a factorial decomposition over immature harnesses, the harness term dominates — one controlled factorial reports harness-induced variance exceeding model-induced variance by 7.8× on a SWE-bench subset, with 6 of 9 model-pair rankings reversing under a different scaffold. Under fixed models, harness choice moved Pass@1 by 27.4pp and model choice by 29.4pp (arXiv:2606.12344). Public leaderboards do not report either regime cleanly, because they run each model under its own best-tuned scaffold.

Should I keep my context window large or invest in compaction?

The measured margin shrinks to 2.7–2.8pp by 128k on the reported benchmarks, so a large window genuinely substitutes for part of the machinery. But the two are not equivalent in failure terms: managed tiers had zero overflow failures at every budget while unmanaged 128k still lost 8.7–12.1% of tasks to window overflow. If your workload already fits comfortably, buy window and skip the stack. If you are anywhere near the limit, manage the context.

Do I need a plan step?

Only if the model abandons tasks too early. Planning added 11.6pp on SWE-Bench for Nemotron-3 30B by cutting runs that ended without an edit from 68.6% to 27.8% and Localize-phase stalls from 58.4% to 10.4%. For stronger models it mostly trims post-edit verification, cutting cost by roughly 30% (Nemotron-3 550B) and 32% (Mistral-Medium-3.5-128B) while success drops 2.0pp and 0.4pp. Test both; the sign depends on your model.

Should I give the agent predefined tools or just bash?

Predefined tools for models with weak shell control — up to 15.0pp on the smallest model, where 66% of bash-only Terminal-Bench trajectories died after out-of-interface tool emissions. Bash-only for bash-capable models, especially on command-line-centric tasks: 3.6–6.7pp better with 30–53% lower cost. Mistral-Medium-3.5-128B needs the full tool set on SWE-Bench (+23.2pp) and bash-only on Terminal-Bench (+6.7pp), so validate per task type rather than per model.

Is lossless context recall worth building?

In the controlled study, no. Elision plus recall_event beat elision alone in 15 settings, lost in 14, tied in 3, for an equal-weight mean difference of −0.36pp. 56.3% of eligible settings never invoked it, the median invocation rate was zero, and mean calls per task fell from 0.540 at 32k to 0.007 at 128k. That is a null result plus a usage collapse — but the deeper lesson is to instrument invocation rates before shipping features like this at all.

Where should you go next?

The component-level view is what makes harness work tractable. Once you know that context management is overflow insurance, planning is an abandonment scaffold, and action space is a capability-dependent cost lever, you can change one thing at a time and measure it. Start with the guide to AI harness engineering for the broader practice, then browse the curated harness collection and the 2026 coding benchmark guide to see how these numbers are produced. If your immediate problem is long-run degradation, Claude Code context management and agentic workflow context management cover the operational side. For harnesses that adapt themselves, see Proteus self-evolving harness, TrueForge, OpenHarness, and the DeepSeek harness handbook.