<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Agent Eval on RockB</title><link>https://baeseokjae.github.io/tags/agent-eval/</link><description>Recent content in Agent Eval on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Thu, 17 Sep 2026 07:01:48 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/agent-eval/index.xml" rel="self" type="application/rss+xml"/><item><title>Better-Harness: An Open-Source Harness Engineering Platform for Better Harness Coding Agents</title><link>https://baeseokjae.github.io/posts/better-harness-open-source-harness-engineering-platform/</link><pubDate>Thu, 17 Sep 2026 07:01:48 +0000</pubDate><guid>https://baeseokjae.github.io/posts/better-harness-open-source-harness-engineering-platform/</guid><description>Better-Harness gives better harness coding agents an open-source, host-agnostic loop: feedforward guides, feedback sensors, and evidence-bounded reports across five work dimensions.</description><content:encoded><![CDATA[<p>If you want better harness coding agents, the fastest lever is rarely a newer model — it is the harness that surrounds the model. Better-Harness is an MIT-licensed, open-source harness engineering platform from QoderAI that runs inside your existing coding agents (Claude Code, Codex, Cursor, and more) rather than replacing them. It combines feedforward guides with feedback sensors, then evaluates your agent across five work-dimension scores that keep missing evidence explicit instead of fabricating plausible numbers.</p>
<h2 id="what-is-the-agent-harness-anyway">What Is the Agent Harness, Anyway?</h2>
<p>The phrase &ldquo;Agent = Model + Harness&rdquo; is not marketing spin; it is a structural claim about where reliability actually comes from. Martin Fowler defines the harness as everything around the model — the outer user harness that shapes how an agent behaves before, during, and after a coding task. That outer harness serves two goals: raise the odds of first-try success, and build a self-correction loop that works before a human ever reviews the result.</p>
<p>Feedforward mechanisms push guidance in ahead of action: AGENTS.md files, task specs, acceptance criteria, and reusable skills. Feedback sensors pull signals back after action: linters, tests, CI gateways, review agents, and evaluation agents. The engineering skill is not adding more of either — it is pairing the right guide with the right sensor so the loop closes quickly.</p>
<p>The same principle appears across the industry. OpenAI documented shipping a product built with roughly 1 million lines of code across about 1,500 pull requests with zero manually written code, running at about 3.5 PRs per engineer per day. Their sharpest lesson was not &ldquo;try harder&rdquo;; it was that early progress stayed slow because the environment was underspecified, and the fix was better legibility — observability and CDP-based introspection — not a stronger prompt.</p>
<p>A widely-cited teardown of Claude Code went further, finding that roughly 98 percent of the tool is the harness, not the model. That number is a useful heuristic even if you discount it: when your agent misbehaves, the surrounding system of guides, sensors, and gates is where most of the fix lives.</p>
<h2 id="what-better-harness-actually-is">What Better-Harness Actually Is</h2>
<p>Better-Harness is an open-source harness engineering platform written in JavaScript, released under the MIT license, and added to GitHub in late July 2026. It sits at roughly 2,261 stars and about 180 forks, with three npm versions and ten dependents. It does not wrap your coding agent in a new client; instead, it runs through whichever agent you already use.</p>
<p>Supported hosts include Claude Code, Codex Desktop and CLI, Qoder, Cursor, qwen-code, GitHub Copilot CLI, Kimi Code, Pi, WorkBuddy, and Grok. That host-agnostic design matters because it means your existing provider, workflow, and prompts survive; Better-Harness layers the evaluation loop on top of them.</p>
<p>Conceptually, the platform formalizes the same feedforward-and-feedback architecture that Fowler and Augment both describe. Guides steer the agent before it acts; sensors — linters, tests, hooks, and evaluation agents — trigger self-correction when output drifts. The whole system is meant to be iterated on: when a failure recurs, you adjust the harness, not the model.</p>
<h2 id="the-five-dimensions-of-the-agent-work-loop">The Five Dimensions of the Agent Work Loop</h2>
<p>Better-Harness evaluates your agent on five Agent Work Loop dimensions rather than a single headline number. That split matters because a coding agent can be excellent at one stage and weak at another, and a total score hides the failure.</p>
<ul>
<li>Task understanding — did the agent correctly interpret the request and the acceptance criteria before acting?</li>
<li>Controlled execution — did it act within the guardrails and scope you set, or did it wander?</li>
<li>Change validation — did linters, tests, and hooks actually run, and did the change pass them?</li>
<li>Reliable delivery — did the agent deliver a mergeable, working result on the first real attempt?</li>
<li>Learning capture — did the loop record what went wrong so the next run avoids the same pitfall?</li>
</ul>
<p>These five dimensions line up with wider measurement thinking. Augment Code recommends tracking task resolution rate, code churn, verification tax, defect escape rate, and pass@1. Better-Harness&rsquo;s dimensions are effectively a practical mapping of those ideas onto a single runnable loop, evaluated in your own repository rather than on a benchmark you never ship.</p>
<h2 id="why-evidence-bounded-findings-matter">Why Evidence-Bounded Findings Matter</h2>
<p>One of the more refreshing properties of Better-Harness is what it refuses to do: it will not fabricate a score. When evidence for a dimension is missing, the report says so explicitly instead of assigning a plausible-looking number. That &ldquo;missing evidence stays explicit&rdquo; stance is genuinely different from opaque evaluation suites that return confident grades with no trail to reproduce them.</p>
<p>The contrast matters because the industry is full of inflated-looking numbers. Top coding agents post 65% to 76.8% resolution rates on SWE-bench-verified Python tasks, yet METR has warned that many benchmark-passing PRs would never actually merge in a real codebase. A DORA study found that 30% of developers reported little to no trust in AI-generated code. When every vendor quotes a benchmark, the only trustworthy signal is one you can reproduce against your own repository — and that is exactly what a harness with explicit &ldquo;evidence missing&rdquo; markers forces you to look at.</p>
<p>Reproducibility is the point: if your report cannot tell you which dimension was never measured, you cannot know whether your harness improved or your benchmark got easier.</p>
<h2 id="where-better-harness-sits-vs-prompt-and-context-engineering">Where Better-Harness Sits vs. Prompt and Context Engineering</h2>
<p>Prompt engineering, context engineering, and harness engineering are often conflated, but they operate at different temporal scopes. Augment Code offers a clean way to slice them: prompt engineering shapes a single turn, context engineering shapes a single context, and harness engineering shapes the full task lifetime. A prompt is a snapshot of intent; a harness is a system that runs before, during, and after the model acts.</p>
<p>In that framing, Better-Harness is unambiguously a harness-engineering tool. It does not improve your one-shot prompt, and it does not curate your working context. It builds the loop around the lifetime of the task: guides in ahead, sensors out after, and a steering loop for you to tighten each time a failure recurs. Monolithic instruction files tend to fail, as OpenAI observed — skills-as-map beats encyclopedia context. Better-Harness operationalizes the map rather than the encyclopedia.</p>
<h2 id="the-open-source-advantage-harness-as-code">The Open-Source Advantage: Harness as Code</h2>
<p>Because Better-Harness is MIT-licensed and written in JavaScript with npm packaging, your harness configuration becomes version-controlled infrastructure rather than a private configuration blob. Teams can treat guides, sensors, and quality gates as code: reviewed, diffed, and reverted like any other change. That is the natural fit for the &ldquo;harness as code&rdquo; idea — your reliability system belongs in the same repo discipline as your application.</p>
<p>The open license also means you are not locked into a closed evaluation platform. If the built-in gates do not cover your compliance checks, your linters, or your own review agents, you extend the platform instead of filing a feature request. The ~180 forks and 10 npm dependents are early evidence that the community is already bending it to fit real workflows rather than adopting a one-size-fits-all score.</p>
<p>This matters for teams that treat coding agents as an ongoing system to improve, not an event to benchmark once. Longitudinal validation — running the loop repeatedly to show the loop itself got better — is only practical when the harness is cheap, local, and yours to modify.</p>
<h2 id="caveats-and-honest-limits">Caveats and Honest Limits</h2>
<p>Before you adopt Better-Harness, keep three limits in mind.</p>
<p>First, evidence is not causal proof. A high score across the five dimensions shows your agent and harness behave well under your conditions; it does not prove the loop caused the improvement. You still need controlled before-and-after runs to attribute gains.</p>
<p>Second, harnessability is not uniform. Martin Fowler&rsquo;s point holds here: not every codebase is equally amenable to harness controls. A greenfield service with fast tests and clean interfaces is far more harnessable than a legacy monolith with flaky suites and tangled dependencies. If your feedback sensors are unreliable, the loop will tell you that loudly — but it cannot repair poor test infrastructure for you.</p>
<p>Third, Better-Harness optimizes what you can measure. It is designed to keep missing evidence explicit, but it cannot force you to define the right dimensions in the first place. Teams that skip the guide-and-sensor design will get an honest report of an underbuilt harness, which is useful information but not a solution.</p>
<h2 id="verdict-who-should-adopt-better-harness">Verdict: Who Should Adopt Better-Harness</h2>
<p>Adopt Better-Harness if you already run one or more of the supported coding agents and you want a reproducible, host-agnostic way to measure and tighten the loop around them. It is especially well-suited to teams that value explicit, evidence-bounded reports over benchmark-friendly but unverifiable scores, and to engineering organizations that want their quality gates version-controlled as open, extendable code.</p>
<p>Skip it if you are looking for a benchmark to quote, or if your codebase lacks the feedback infrastructure (tests, linters, hooks) that any harness loop depends on — the platform will expose that gap, not close it for you.</p>
<p>For most teams running Claude Code, Codex, or Cursor in earnest, the practical verdict is positive: at zero license cost, with a mental model that matches how reliability actually improves — via the harness, not the model — Better-Harness is one of the most honest tools available for better harness coding agents.</p>
<h2 id="how-to-get-started-with-better-harness">How to Get Started with Better-Harness</h2>
<p>Starting takes minutes and follows the same shape on every supported host.</p>
<p>First, install the platform through your existing agent workflow (npm-based, per the repository). Because it is host-agnostic, the installation pattern for Claude Code differs only trivially from Codex CLI or Cursor.</p>
<p>Second, run a baseline pass on a representative task without changing anything about how you normally work. Capture the initial score across the five dimensions — this baseline is your point of comparison, and it will already tell you which dimensions were never actually measured.</p>
<p>Third, read the report for &ldquo;evidence missing&rdquo; markers. Those gaps are your highest-value work because they point at the sensors you lack, not the guides you should write more of.</p>
<p>Fourth, tighten one loop: add a guide where the agent misread intent (task understanding), add a pre-push linter or hook where it shipped sloppy changes (change validation), or add an eval agent where it delivered the wrong shape repeatedly (reliable delivery). Re-run the baseline.</p>
<p>Fifth, repeat and compare longitudinally. The goal is not a one-time score but a rising floor: each run should show the loop measurably improving, with every score traceable to evidence in your repository.</p>
<h2 id="faq">FAQ</h2>
<p><strong>What is Better-Harness?</strong>
Better-Harness is an MIT-licensed, open-source harness engineering platform from QoderAI that runs inside existing coding agents — Claude Code, Codex, Cursor, and others — combining feedforward guides with feedback sensors and reporting evidence-bounded scores across five work-dimensions.</p>
<p><strong>What does &ldquo;Agent = Model + Harness&rdquo; mean?</strong>
It means the reliability of a coding agent depends on the system around the model — guides in ahead of action and sensors after it — more than on the model itself. A popular teardown attributed roughly 98% of Claude Code to the harness, not the model.</p>
<p><strong>Is Better-Harness free to use?</strong>
Yes. It is released under the MIT license with npm packaging, so there is no license cost, and teams can extend its quality gates for their own compliance checks instead of buying a closed evaluation platform.</p>
<p><strong>Which coding agents does Better-Harness support?</strong>
It is host-agnostic and runs through Claude Code, Codex Desktop and CLI, Qoder, Cursor, qwen-code, GitHub Copilot CLI, Kimi Code, Pi, WorkBuddy, and Grok — it layers the evaluation loop on top without replacing your client or workflow.</p>
<p><strong>How does Better-Harness keep scores honest?</strong>
It refuses to fabricate numbers: when evidence for a dimension is missing, the report says so explicitly instead of assigning a plausible score. That &ldquo;missing evidence stays explicit&rdquo; stance keeps every claim reproducible in your own repository.</p>
]]></content:encoded></item></channel></rss>