<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>JSON Key Order on RockB</title><link>https://baeseokjae.github.io/tags/json-key-order/</link><description>Recent content in JSON Key Order on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 15 Nov 2026 14:00:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/json-key-order/index.xml" rel="self" type="application/rss+xml"/><item><title>Same Seed, Different Bytes: Measuring Tool-Call Repeatability Before You Trust an Agent Eval (2026)</title><link>https://baeseokjae.github.io/posts/agent-tool-call-repeatability-2026/</link><pubDate>Sun, 15 Nov 2026 13:38:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/agent-tool-call-repeatability-2026/</guid><description>13 replays of one fixed tool-calling request on Ollama 0.20.3 with qwen3:0.6b at seed 42: all 13 calls were semantically identical, but only 11 were byte-identical, and both flips happened at temperature 0.0. Includes the probe, the raw per-call JSONL and a normalise-then-hash helper.</description><content:encoded><![CDATA[<p>A team re-runs its agent eval after a one-line prompt change and the pass rate moves five points. They revert the prompt; the number does not come back. They blame the harness, then the model vendor, and eventually pin <code>temperature=0</code> and <code>seed=42</code> everywhere — and the number still moves. Nobody in that loop can answer the question that comes before all of it: at the sampling settings the harness already uses, is the emitted tool call even the same object twice?</p>
<p>Three different properties get conflated in that question. <strong>Semantic agreement</strong> is the same tool name with the same argument values, which is all an eval should assert on. <strong>Byte identity</strong> is the same argument text, including key order and numeric formatting, which is what a golden-file or snapshot test actually asserts on. <strong>Prompt-shape dependence</strong> is the fact that the same question asked as turn one and as turn four arrives with different history and different prefill, so a flip measured in one shape says nothing about the other.</p>
<p>This page answers four questions with a measurement rather than an argument: how often a pinned call repeats semantically and how often byte-for-byte; whether those two rates differ and in which direction; whether adding one history turn or padding the prompt changes either; and what a harness should therefore assert. The sample is small and is stated up front so nothing below has to be discounted later: <strong>13 calls across four prompt shapes</strong>, one model, one host, seed 42, zero errors. All 13 were semantically identical. Eleven of 13 were byte-identical, and both of the byte flips happened at <code>temperature=0.0</code>. The N=20-per-condition matrix in the original plan was not run — the attempt budget does not hold 20–40 minutes of wall time on the host — so the numbers below are <strong>counts with denominators, not rates</strong>, and the difference matters.</p>
<h2 id="what-the-vendors-actually-promise">What the vendors actually promise</h2>
<p>Ollama&rsquo;s API documentation offers the seed as the route to repeatability: &ldquo;For reproducible outputs, set <code>seed</code> to a number&rdquo; (<a href="https://github.com/ollama/ollama/blob/main/docs/api.md">github.com/ollama/ollama/blob/main/docs/api.md</a>). Its Modelfile reference states the promise more directly — setting <code>seed</code> to a specific number &ldquo;will make the model generate the same text for the same prompt&rdquo; (<a href="https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx">docs/modelfile.mdx</a>). vLLM frames the same property at the sampling layer, and goes further about temperature: &ldquo;In V1, the seed parameter defaults to 0 which sets the random state for each worker, so the results will remain consistent for each vLLM run even if temperature &gt; 0&rdquo; — while also opening its page with &ldquo;vLLM does not guarantee the reproducibility of the results by default, for the sake of performance&rdquo; (<a href="https://docs.vllm.ai/en/latest/usage/reproducibility.html">docs.vllm.ai reproducibility</a>).</p>
<p>Both promises are about the same kind of object: generated text, or the random state that produces it. The thing an agent harness asserts on is a different kind of object. <code>tool_calls[0].function.arguments</code> is a structured value that the server serialises back to JSON, and a JSON object is, per RFC 8259, &ldquo;an unordered collection of zero or more name/value pairs&rdquo; (<a href="https://www.rfc-editor.org/rfc/rfc8259.txt">rfc-editor.org/rfc/rfc8259.txt</a>). Two serialisations of the same arguments can therefore differ in member order while denoting the same value. None of the three vendor pages above says anything about member order in a structured argument payload, and none of them measures it. A harness that treats the serialised arguments as a string inherits a property that its vendor documentation never claimed to control.</p>
<h2 id="what-was-measured-and-what-it-cannot-tell-you">What was measured, and what it cannot tell you</h2>
<p>The fixture is one fixed coding-agent request: a single tool <code>stat_path</code> with <code>path</code> (string), <code>mode</code> (enum <code>stat|read|hash</code>) and <code>max_bytes</code> (integer), and the instruction &ldquo;Check whether docs/config.md exists and get its hash. Use the tool.&rdquo; Every response was recorded verbatim; no expected output was written by hand. The probe posted to <code>http://127.0.0.1:11434/api/chat</code> with <code>stream=false</code>, <code>think=false</code>, <code>num_predict=80</code>, <code>num_ctx=4096</code> and <code>seed=42</code>, and recorded per call the raw arguments value, the deserialised object, its key order, and two digests: <code>sha256(json.dumps(args))</code> for the byte form and <code>sha256(json.dumps(args, sort_keys=True))</code> for the semantic form.</p>
<p>The server was Ollama 0.20.3 on a CPU-only Linux host (2 vCPU, 11 GB RAM, no GPU). The model was <code>qwen3:0.6b</code>, digest <code>7df6b6e09427a769808717c0a93cadc4ae99ed4eb8bf5ca557c90846becea435</code>, Q4_K_M, 751.63M parameters as reported by the server. <code>qwen2.5:0.5b</code> was installed and identified but not measured, so nothing here is a two-model claim. The four conditions and their sample sizes are in the table below; the exact environment, request body and the deviations from the original plan are in <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/METHOD.md">METHOD.md</a>.</p>
<table>
	<thead>
			<tr>
					<th>Condition</th>
					<th>Shape</th>
					<th>Temp</th>
					<th>N</th>
					<th>Semantic distinct</th>
					<th>Byte distinct</th>
					<th>Byte identical</th>
					<th>Key orders observed</th>
					<th>prompt_eval_count</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>S1</td>
					<td>single turn</td>
					<td>0.0</td>
					<td>4</td>
					<td>1</td>
					<td>1</td>
					<td>4/4</td>
					<td>(path,mode)×4</td>
					<td>197</td>
			</tr>
			<tr>
					<td>S2</td>
					<td>one prior turn (user, assistant tool_call, tool result)</td>
					<td>0.0</td>
					<td>3</td>
					<td>1</td>
					<td>2</td>
					<td>2/3</td>
					<td>(mode,path)×2, (path,mode)×1</td>
					<td>268</td>
			</tr>
			<tr>
					<td>S3</td>
					<td>~1,200-token repository listing prefix (1,420 total prompt tokens)</td>
					<td>0.0</td>
					<td>3</td>
					<td>1</td>
					<td>2</td>
					<td>2/3</td>
					<td>(path,mode)×2, (mode,path)×1</td>
					<td>1,420</td>
			</tr>
			<tr>
					<td>S4</td>
					<td>single turn</td>
					<td>0.7</td>
					<td>3</td>
					<td>1</td>
					<td>1</td>
					<td>3/3</td>
					<td>(path,mode)×3</td>
					<td>197</td>
			</tr>
	</tbody>
</table>
<p>Three caveats belong next to that table rather than after it. The host was <strong>shared with a concurrent stage workload for the entire window</strong>, so wall time was recorded but is not interpretable and no latency or throughput claim is made. The design was <strong>round-robin rather than condition-by-condition</strong>, because a sequential loop would have spent the whole budget on S1 when each long-prompt call took 87–116 seconds. And the digests score the arguments object <strong>as the <code>/api/chat</code> response serialises it</strong> — this run cannot separate a model-side token-order difference from a server-side parse-and-reserialise step, and does not assert which it was.</p>
<h2 id="result-1313-semantic-1113-byte">Result: 13/13 semantic, 11/13 byte</h2>
<p>Every one of the 13 calls emitted exactly one tool call, named <code>stat_path</code>, with <code>path=docs/config.md</code>, <code>mode=hash</code> and no <code>max_bytes</code>. Argument values never drifted, no required argument went missing, and no extra field appeared. Under sorted-key comparison the 13 calls collapse to a single digest; replayed through the normaliser in the last section, all 13 records produce one distinct semantic digest.</p>
<p>Byte identity is the part that did not hold. In S2 and S3, one call out of three carried the same two members in the opposite order from its two siblings. The flip is the whole finding, so it is worth seeing in full:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{<span style="color:#f92672">&#34;path&#34;</span>: <span style="color:#e6db74">&#34;docs/config.md&#34;</span>, <span style="color:#f92672">&#34;mode&#34;</span>: <span style="color:#e6db74">&#34;hash&#34;</span>}
</span></span><span style="display:flex;"><span>{<span style="color:#f92672">&#34;mode&#34;</span>: <span style="color:#e6db74">&#34;hash&#34;</span>, <span style="color:#f92672">&#34;path&#34;</span>: <span style="color:#e6db74">&#34;docs/config.md&#34;</span>}
</span></span></code></pre></div><p>That is a difference a consumer of the call cannot observe — <code>json.loads</code> produces an equal dict either way — and a difference a text diff observes immediately. In this sample it appeared in the two conditions whose prompt was not the minimal one, and it did not appear at <code>temperature=0.7</code>.</p>
<p>Which is exactly where the result has to be handled carefully. The most tempting sentence to write here is &ldquo;temperature does not matter&rdquo;, and it is not supported. With N=3–4 there is no power to order conditions by flip probability; temperature and prompt shape are confounded with <strong>which condition happened to catch a flip</strong>; and 3/3 stability at <code>temperature=0.7</code> is not evidence that 0.7 is safer. The defensible claim is narrower and still useful: in this sample, pinning <code>temperature=0</code> was <strong>not sufficient</strong> for byte identity, so a harness that attributes a moved pass rate to its prompt has at least one other candidate cause it has not eliminated.</p>
<h2 id="why-a-golden-file-test-is-the-wrong-assertion">Why a golden-file test is the wrong assertion</h2>
<p>A snapshot test that stores <code>json.dumps(arguments)</code> and compares strings is asserting on member order. JSON defines that order as carrying no meaning, so the assertion is stricter than the contract the data is published under. The failure modes that follow are ordinary and expensive:</p>
<ul>
<li><strong>Snapshot churn for no reason.</strong> A stored call expires because the same value came back in a different order. Someone re-records the snapshot; the test goes green; nobody learns anything.</li>
<li><strong>Retry accounting that hides the real difference.</strong> If a harness retries on snapshot mismatch, a flip costs an extra call and the recorded pass rate now depends on retry policy — a separate and much better-documented hazard (<a href="https://baeseokjae.github.io/posts/agent-tool-retry-duplicate-side-effects-2026/">agent-tool-retry-duplicate-side-effects-2026</a>).</li>
<li><strong>A pass rate that moves for reasons unrelated to the change under test.</strong> This is the opening scene. The flip is not proof that it explains a five-point move — 13 calls cannot support that — but it is a mechanism that produces exactly that symptom while every input is held fixed.</li>
</ul>
<p>Two adjacent failures are worth separating out, because both look like this and neither is. An agent that repeats a failing action because its loop has no state guard is a control-flow bug, not sampling variance (<a href="https://baeseokjae.github.io/posts/deterministic-agent-loop-failures-2026/">deterministic-agent-loop-failures-2026</a>). A tool call that comes back with its arguments retyped or its history corrupted is a transcript-handling problem, and the first thing to rule out is ordinary run-to-run variance, which is what this page measures (<a href="https://baeseokjae.github.io/posts/ollama-tool-call-history-corruption-2026/">ollama-tool-call-history-corruption-2026</a>).</p>
<h2 id="what-to-assert-instead-normalise-then-compare-then-hash">What to assert instead: normalise, then compare, then hash</h2>
<p>The recipe is short. Extract the tool name and the arguments, coerce a JSON-string argument into an object, compare with sorted keys, and hash the sorted form when a test needs a stable identifier. The helper below is a real file in the evidence directory, standard library only, and it is what produced the &ldquo;one distinct digest&rdquo; statement above when replayed over the 13 raw records:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">normalise_tool_call</span>(message, required<span style="color:#f92672">=</span><span style="color:#66d9ef">None</span>):
</span></span><span style="display:flex;"><span>    calls <span style="color:#f92672">=</span> (message <span style="color:#f92672">or</span> {})<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#34;tool_calls&#34;</span>) <span style="color:#f92672">or</span> []
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> <span style="color:#f92672">not</span> calls:
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">None</span>, {}, <span style="color:#66d9ef">None</span>, [<span style="color:#e6db74">&#34;no_tool_call&#34;</span>]
</span></span><span style="display:flex;"><span>    fn <span style="color:#f92672">=</span> calls[<span style="color:#ae81ff">0</span>]<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#34;function&#34;</span>) <span style="color:#f92672">or</span> {}
</span></span><span style="display:flex;"><span>    name, args, problems <span style="color:#f92672">=</span> fn<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#34;name&#34;</span>), fn<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#34;arguments&#34;</span>), []
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> isinstance(args, str):                 <span style="color:#75715e"># args emitted as a JSON string</span>
</span></span><span style="display:flex;"><span>        args <span style="color:#f92672">=</span> json<span style="color:#f92672">.</span>loads(args)
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> <span style="color:#f92672">not</span> isinstance(args, dict):
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">return</span> name, {}, <span style="color:#66d9ef">None</span>, [<span style="color:#e6db74">&#34;arguments_not_object&#34;</span>]
</span></span><span style="display:flex;"><span>    problems <span style="color:#f92672">+=</span> [<span style="color:#e6db74">&#34;missing:&#34;</span> <span style="color:#f92672">+</span> k <span style="color:#66d9ef">for</span> k <span style="color:#f92672">in</span> (required <span style="color:#f92672">or</span> []) <span style="color:#66d9ef">if</span> k <span style="color:#f92672">not</span> <span style="color:#f92672">in</span> args]
</span></span><span style="display:flex;"><span>    canon <span style="color:#f92672">=</span> json<span style="color:#f92672">.</span>dumps({<span style="color:#e6db74">&#34;name&#34;</span>: name, <span style="color:#e6db74">&#34;arguments&#34;</span>: args},
</span></span><span style="display:flex;"><span>                       sort_keys<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>, separators<span style="color:#f92672">=</span>(<span style="color:#e6db74">&#34;,&#34;</span>, <span style="color:#e6db74">&#34;:&#34;</span>))
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> name, args, hashlib<span style="color:#f92672">.</span>sha256(canon<span style="color:#f92672">.</span>encode())<span style="color:#f92672">.</span>hexdigest(), problems
</span></span></code></pre></div><p>The four cases a normaliser has to survive are the ones that actually appear in transcripts: arguments already an object; arguments emitted as a JSON string; a required argument missing; and no tool call at all. The version in <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/normalise_tool_call.py">normalise_tool_call.py</a> carries those four plus the missing-argument variant as runnable self-tests, and <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/normalise_selftest.txt">normalise_selftest.txt</a> is the captured output of the run, including the replay over all 13 records: one digest, five cases passing.</p>
<p>Normalising fixes the assertion. It does not fix the ambiguity about whether the model&rsquo;s <em>values</em> are stable, which is why the second half of the recipe is a stability check at the settings the harness actually uses.</p>
<h2 id="how-to-size-your-own-check">How to size your own check</h2>
<p>The probe is in the evidence directory as it was run — interleaved round-robin, an internal wall-clock budget, append-only JSONL flushed before the next call starts, tolerant of str-or-dict arguments (<a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/repeat_probe.py">repeat_probe.py</a>). Point it at your own server and model, and run it in the prompt shapes your harness actually uses, not just the minimal one: in this table the flips appeared only in the two non-minimal shapes, and a stability check that only covers single-turn prompts would have reported 4/4 and 3/3 and missed them.</p>
<p>The decision rule is what to do with the output, and it has to respect the sample. If every call in your sample is identical, that is a statement about your sample, your model and your prompt shapes — not about determinism in general. If some calls flip, you have found the mechanism and still cannot size it from a handful of replays; report the counts with their denominators and say so. A 2/3 result is compatible with anything from a few percent to most calls flipping, and a table that prints it as &ldquo;67% flaky&rdquo; is inventing a rate the sample cannot support.</p>
<p>Every file behind the table is downloadable and was inspected for credentials and host-identifying data before publication: the raw per-call records are in <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/run.jsonl">run.jsonl</a>, the one-line-per-call log including the two <code>BUDGET_EXHAUSTED</code> markers is <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/probe.log">probe.log</a>, and the table above is also stored as <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-tool-call-repeatability-2026/flip_table.csv">flip_table.csv</a>. Each record carries the condition label, the emitted tool call with its key order and both digests, and the server counters; the prompt text itself (the single-turn instruction, the history turn and the repository listing) lives in <code>repeat_probe.py</code>.</p>
<h2 id="what-this-does-not-show">What this does not show</h2>
<p>One host, one model, one fixture, 13 calls, a shared machine. Nothing here estimates a flip rate, and nothing here attributes the key-order difference to the model or to the server — the measurement scores the serialised response and stops there. Only <code>qwen3:0.6b</code> was run, so there is no quantisation comparison and no two-model comparison. The fixture instruction has one obviously correct call, which means argument-<em>value</em> drift under genuine ambiguity is untested and is a different question from the key-order drift observed here. The vendor pages establish what is promised, not what a specific server build implements, and none of them was checked for version drift against Ollama 0.20.3.</p>
<p>The follow-up that would change the picture is ordinary and unglamorous: the same fixture across two models, with N large enough on an idle host to separate a rate from a coincidence. Until someone runs it, the safe harness is the one that normalises before comparing — safe regardless of how these counts replicate — and the safe benchmark claim is the one that names its sample size instead of implying determinism it never measured.</p>
<p>Related reading: the principle that eval scores move between runs, at benchmark scale rather than at the level of one emitted call, is covered in <a href="https://baeseokjae.github.io/posts/llm-benchmark-variance-2026/">llm-benchmark-variance-2026</a>; the request shape and tool registry used here are in <a href="https://baeseokjae.github.io/posts/ollama-api-local-llm-integration-developer-guide-2026/">ollama-api-local-llm-integration-developer-guide-2026</a>; what the same registry costs in prompt tokens is in <a href="https://baeseokjae.github.io/posts/ollama-tool-definition-prompt-token-cost-2026/">ollama-tool-definition-prompt-token-cost-2026</a>; and how to declare the assertion this table justifies is in <a href="https://baeseokjae.github.io/posts/assert-agent-evaluation-harness-2026/">assert-agent-evaluation-harness-2026</a>.</p>
]]></content:encoded></item></channel></rss>