<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Sentence Chunking vs Document Chunking on RockB</title><link>https://baeseokjae.github.io/tags/sentence-chunking-vs-document-chunking/</link><description>Recent content in Sentence Chunking vs Document Chunking on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 15 Nov 2026 08:55:36 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/sentence-chunking-vs-document-chunking/index.xml" rel="self" type="application/rss+xml"/><item><title>One Fact per Vector: What Chunk Granularity Does to Agent-Memory Recall, Measured on Three Local Embedders (2026)</title><link>https://baeseokjae.github.io/posts/agent-memory-chunk-granularity-recall-2026/</link><pubDate>Sun, 15 Nov 2026 08:55:36 +0000</pubDate><guid>https://baeseokjae.github.io/posts/agent-memory-chunk-granularity-recall-2026/</guid><description>An 8-document, 13-question fixture on three local Ollama embedders: two-fact hit@1 falls from 13/15 with whole-document vectors to 0/15 at 3-sentence and single-sentence chunks, while mean top-3 context drops from 1,444 to 210 characters. Includes the probe script and fixture.</description><content:encoded><![CDATA[<p>An agent memory store is usually built by a chunker that nobody re-examines after the first week. One vector per conversation turn, per note, or per fixed block of text. The store looks healthy, because the questions used to check it are single-fact questions about a short note, and those pass at almost any granularity. Then a user asks a two-part question — &ldquo;how do access-token and signing-key lifetimes compare?&rdquo;, &ldquo;who owns the deploy key and how often does it rotate?&rdquo; — and the retriever returns one half of the answer with no error at all. The other half is in a different chunk and did not fit in the top-k.</p>
<p>The opposite mistake is just as common. Keeping whole documents to preserve context means every retrieval drags 1,400 characters of mostly irrelevant prose into the prompt instead of 210.</p>
<p>This article measures the tradeoff instead of asserting it. On one host, with three locally served embedding models, an 8-document fixture and 13 questions, the short answer is:</p>
<ul>
<li>Single-fact recall did not degrade as chunks got smaller (21/24 pooled whole-document, 23/24 at 3-sentence windows, 23/24 at single sentences).</li>
<li>Two-fact recall at k=1 collapsed from 13/15 to 0/15 the moment document-level vectors were replaced by 3-sentence windows, and stayed at 0/15 at sentence level.</li>
<li>The two facts usually survived somewhere in a deeper ranking: both required sentences appeared in the top 8 for 15/15, 13/15 and 12/15 respectively.</li>
<li>Mean characters in the top-3 retrieved chunks fell from 1,444 to 475 to 210 — about 6.9x less prompt context for the same k.</li>
</ul>
<p>So the decision is not &ldquo;smaller chunks retrieve better&rdquo; or &ldquo;bigger chunks preserve meaning&rdquo;. It is: what query shape do you serve, and can your retriever merge two chunks before the prompt budget closes?</p>
<h2 id="the-two-questions-your-chunker-decides-for-you">The two questions your chunker decides for you</h2>
<p>A chunk boundary is a retrieval contract. Whatever else you configure, two properties of your system are fixed the moment the chunker runs.</p>
<p>The first is <strong>composability</strong>: can any single retrieved unit contain all the facts that one realistic question needs? A sentence chunk can answer &ldquo;what is the retry interval&rdquo; and can never answer &ldquo;how does the retry interval compare with the escalation delay&rdquo;, because a comparison needs two sentences and one chunk holds one. Nothing errors. The ranker returns its best single sentence, which is a correct answer to half of the question.</p>
<p>The second is <strong>context cost per retrieval</strong>: how much text does k=3 hand to the model? With whole-document vectors, three hits are three documents. That is fine when documents are 480 characters and expensive when they are 8,000.</p>
<p>Both properties are measurable before you pick a chunk size, and neither is answered by vendor documentation. Vendor pages tell you the context window and the API shape, not what your boundaries do to your recall.</p>
<h2 id="what-the-fixture-measures-and-what-it-does-not">What the fixture measures, and what it does not</h2>
<p>Eight invented project documents (deploy, billing, auth, search, mobile, pipeline, support, infra), seven sentences each, 456–500 characters per document, 52–99 characters per sentence. Each document carries one needle fact, plus facts that only mean something when combined with an adjacent sentence. Eight single-fact questions (one per document) and five two-fact questions whose two required sentences sit on opposite sides of a 3-sentence window boundary.</p>
<p>Three granularities of the same corpus:</p>
<ul>
<li>whole document: 8 vectors per model</li>
<li>non-overlapping 3-sentence windows: 24 vectors (sentences 1–3, 4–6, 7)</li>
<li>single sentences: 56 vectors</li>
</ul>
<p>Three embedders served by local Ollama 0.20.3 through <code>/api/embed</code>: all-minilm:latest (384-d), snowflake-arctic-embed:33m (384-d) and nomic-embed-text:latest (768-d). Cosine on L2-normalised vectors, a deterministic full sort, no generation, no re-ranker, no hybrid search, no metadata filters. Each row reports single-fact hit@1 (the needle sentence is inside the top-1 chunk), two-fact hit@1 (both required sentences are inside the top-1 chunk), and the mean character count of the top-3 chunks. The full sweep took 54 seconds.</p>
<p>What this does <strong>not</strong> measure: answer quality, latency, throughput, token counts, production recall, or user satisfaction. hit@1 here means &ldquo;the right text was in the returned chunk&rdquo;. Nothing else.</p>
<h2 id="result-1-fine-chunks-cost-nothing-on-single-fact-questions">Result 1: fine chunks cost nothing on single-fact questions</h2>
<p>Pooled over the three embedders and the eight single-fact questions — 24 model-question pairs per granularity:</p>
<table>
	<thead>
			<tr>
					<th>Granularity</th>
					<th>Vectors per model</th>
					<th>Single-fact hit@1</th>
					<th>Two-fact hit@1</th>
					<th>Both facts in top 8</th>
					<th>Mean top-3 chars</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Whole document</td>
					<td>8</td>
					<td>21/24</td>
					<td>13/15</td>
					<td>15/15</td>
					<td>1,444</td>
			</tr>
			<tr>
					<td>3-sentence windows</td>
					<td>24</td>
					<td>23/24</td>
					<td>0/15</td>
					<td>13/15</td>
					<td>475</td>
			</tr>
			<tr>
					<td>Single sentence</td>
					<td>56</td>
					<td>23/24</td>
					<td>0/15</td>
					<td>12/15</td>
					<td>210</td>
			</tr>
	</tbody>
</table>
<p>Per model, single-fact hit@1 moved 8/8 → 8/8 → 8/8 (all-minilm), 6/8 → 7/8 → 8/8 (arctic-33m) and 7/8 → 8/8 → 7/8 (nomic). Those are one-query swings on an eight-question set. The honest reading is &ldquo;no measurable damage from finer chunks&rdquo;, not &ldquo;fine chunks beat documents&rdquo;. Anyone who has read a single-fact recall table will recognise the shape: at 480-character documents, the whole-document vector is not diluted enough to hurt, and the small deficits are noise.</p>
<p>This is the result that makes chunking decisions go wrong. A team shrinks chunks, reruns its single-fact regression suite, sees 8/8 unchanged, and concludes the change was free. It was not free. The cost is simply invisible to that test suite.</p>
<h2 id="result-2-fine-chunks-lose-the-two-fact-answer-completely-at-k1">Result 2: fine chunks lose the two-fact answer completely at k=1</h2>
<p>Same fixture, five two-fact questions, top-1 only:</p>
<table>
	<thead>
			<tr>
					<th>Granularity</th>
					<th>Two-fact hit@1 (pooled)</th>
					<th>all-minilm</th>
					<th>arctic-33m</th>
					<th>nomic</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Whole document</td>
					<td>13/15</td>
					<td>4/5</td>
					<td>4/5</td>
					<td>5/5</td>
			</tr>
			<tr>
					<td>3-sentence windows</td>
					<td>0/15</td>
					<td>0/5</td>
					<td>0/5</td>
					<td>0/5</td>
			</tr>
			<tr>
					<td>Single sentence</td>
					<td>0/15</td>
					<td>0/5</td>
					<td>0/5</td>
					<td>0/5</td>
			</tr>
	</tbody>
</table>
<p>Two mechanisms produce that zero, and they are not equally general.</p>
<p>At <strong>sentence granularity</strong> the zero is structural: one sentence cannot contain two sentences, so no ranking can ever satisfy the condition. Any corpus, any embedder, any question shape. If your questions ever require two facts, single-sentence chunks at k=1 are guaranteed to fail, and the failure is silent because the top hit is still a sensible sentence.</p>
<p>At <strong>3-sentence windows</strong> the zero is constructed. The five pairs were deliberately placed to straddle a window boundary, so each question&rsquo;s two sentences necessarily land in different windows. That is a worst case, and a corpus whose two-fact answers happen to fall inside one window would score differently. It is also not an artificial worry: any fixed-width window will straddle some fraction of real co-reference pairs, and the fraction grows as the window shrinks.</p>
<p>At <strong>top-3</strong> the loss softens but does not disappear: both required sentences appear among the three returned chunks for 15/15 whole-document, 10/15 at 3-sentence windows and 8/15 at single sentences. A retriever that merges its top-3 into one context window recovers some of the damage — but only if it merges, and only if the ranker put both halves inside k.</p>
<h2 id="result-3-the-facts-are-usually-still-there-just-not-together">Result 3: the facts are usually still there, just not together</h2>
<p>The most useful number in the run is the top-8 check. Both required sentences appeared somewhere in ranks 1–8 for 15/15, 13/15 and 12/15 across the three granularities. At sentence level, twelve of fifteen two-fact questions had both halves one deeper ranking away.</p>
<p>That reframes the fix. If your retriever can see the fragments, the problem is assembly, not indexing, and assembly is cheaper than re-chunking a live store:</p>
<ul>
<li><strong>Merge before the prompt.</strong> Retrieve 8–10 chunks, then concatenate neighbours from the same source document (or group by document id) before the model sees them. This converts a ranking problem into a grouping problem.</li>
<li><strong>Two-stage retrieval.</strong> Retrieve wide at fine granularity, then re-rank or expand the winning chunks to their parent window. The classic parent-document pattern.</li>
<li><strong>Keep a dual index.</strong> Fine chunks for precision, whole documents for context, and choose per query shape.</li>
</ul>
<p>All three need the store to record what the chunk <em>is</em>: source document id, sentence offsets, the encoder&rsquo;s identity and digest, and the vector&rsquo;s dimension. Without that metadata you cannot expand a hit to its neighbours, and you cannot tell whether a stored vector is comparable to a query vector. <a href="https://baeseokjae.github.io/posts/agent-memory-write-path-reliability-2026/">The write-path article covers the record each chunk needs</a>.</p>
<h2 id="result-4-what-coarse-chunks-cost-in-prompt-characters">Result 4: what coarse chunks cost in prompt characters</h2>
<p>Mean characters of the top-3 retrieved chunks, averaged over the eight single-fact queries per row: 1,444 for whole documents, 475 for 3-sentence windows, 210 for single sentences. That is 6.9x less text for the same k.</p>
<p>Two cautions, both of which matter more than the ratio itself.</p>
<p>It is characters, not tokens. No tokenizer was run, so this is not a cost figure and should not be converted into one. If you need tokens, measure them; the Ollama embed response carries <code>prompt_eval_count</code> specifically so you do not have to guess (<a href="https://docs.ollama.com/api/embed">embed API reference</a>).</p>
<p>The ratio is a property of <em>your</em> document lengths, not a constant. These documents are 482 characters on average, so a whole-document chunk is about six sentences&rsquo; worth of prose — the coarse setting here is only mildly coarse. At 8,000-character documents the same table would show a far larger ratio, and at 200-character notes it would nearly vanish. The transferable finding is the direction and the mechanism, not 6.9.</p>
<h2 id="does-the-embedder-change-the-answer">Does the embedder change the answer?</h2>
<p>No — the shape held on all three models. Every embedder went 4/5-or-5/5 → 0/5 → 0/5 on two-fact hit@1 while staying flat on single-fact recall. That consistency is why the effect reads as a property of the chunk boundary rather than of one model&rsquo;s geometry.</p>
<p>The nomic row needs a specific caveat. nomic-embed-text-v1.5&rsquo;s <a href="https://huggingface.co/nomic-ai/nomic-embed-text-v1.5">model card</a> states that the text must carry a task instruction prefix: <code>search_document: </code> for indexed text and <code>search_query: </code> for queries. This harness sent no prefixes to any model, and Ollama&rsquo;s own <a href="https://ollama.com/library/nomic-embed-text">nomic-embed-text library page</a> shows an example without them. So nomic&rsquo;s absolute numbers here are not a fair model comparison and must not be read as a ranking. The prefix omission applies in code to all three rows equally, but only nomic documents the requirement — meaning nomic is the row most likely to be depressed by the choice.</p>
<p>The practical consequence: choose the chunk size before you choose the embedder. The boundary failure shows up on every model tested; the encoder choice changes absolute recall, not whether a two-sentence question can be answered by a one-sentence chunk.</p>
<p>One sizing check belongs here, because it explains why truncation is not a confound in this run. all-minilm and snowflake-arctic-embed:33m both publish a 512-token context window on their Ollama library pages (<a href="https://ollama.com/library/all-minilm">all-minilm</a>, <a href="https://ollama.com/library/snowflake-arctic-embed">snowflake-arctic-embed</a>), and <code>/api/embed</code> defaults to <code>truncate: true</code>, silently clipping longer inputs. At roughly 482 characters per document — on the order of 120 tokens at a rule-of-thumb 4 characters per token — no indexed chunk here came close to that ceiling. That is a sizing argument, not a measurement: <code>prompt_eval_count</code> was not recorded in this run. The truncation ceiling itself is a different failure and is <a href="https://baeseokjae.github.io/posts/local-embedding-context-ceiling-2026/">measured separately</a>.</p>
<p>This also separates the result from the encoder-side failure measured in <a href="https://baeseokjae.github.io/posts/agent-memory-embedding-swap-recall-2026/">the embedding-swap harness</a>, where replacing an encoder with a same-width model turned ranking into noise. Here the encoder is fixed within each row and only the boundary moves, which is the only way to attribute the change to chunking.</p>
<h2 id="a-decision-procedure-for-your-own-corpus">A decision procedure for your own corpus</h2>
<p>The fixture is small on purpose: it makes the mechanism visible in under a minute, and it is meant to be replaced by your own documents. The harness is published — point it at your corpus, not at mine.</p>
<p>To reproduce this exact run you need Ollama serving the three models locally. Then:</p>
<pre tabindex="0"><code>python3 chunk_probe2.py fixture.json results2.json
</code></pre><p><code>chunk_probe2.py</code> records each needle&rsquo;s rank to depth 8 and reports <code>both_in_top8</code> alongside hit@1 and hit@3; <code>chunk_probe.py</code> is the earlier hits-only sweep. Both read the fixture JSON, call <code>http://127.0.0.1:11434/api/embed</code> only, and print one row per (model, granularity).</p>
<p>The published files, all original and inspectable before you run them:</p>
<ul>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-memory-chunk-granularity-recall-2026/chunk_probe2.py">chunk_probe2.py</a> — the probe, parameterised by fixture and output path</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-memory-chunk-granularity-recall-2026/fixture.json">fixture.json</a> — 8 documents, 8 single-fact and 5 two-fact questions</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-memory-chunk-granularity-recall-2026/results2.json">results2.json</a> — the raw per-row output behind every number above</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-memory-chunk-granularity-recall-2026/environment.json">environment.json</a> — model names, digests, host and scoring settings</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/agent-memory-chunk-granularity-recall-2026/METHOD.md">METHOD.md</a> — method, commands and known deviations</li>
</ul>
<p>Note that the fixture text ships with the probe and is entirely invented; nothing in it comes from a customer, a vendor, or a public benchmark.</p>
<p>The procedure to apply to your own store:</p>
<ol>
<li><strong>Write your real question shapes down first</strong>, separated into single-fact and multi-fact. If you have no multi-fact questions in that list, this entire article is not your problem yet — but your users will find some.</li>
<li><strong>Build a labelled set of 30–100 questions</strong> over your own documents, each with the sentence or span that answers it. Thirty is enough to see a collapse from 13/15 to 0/15; it is not enough to resolve a two-query difference.</li>
<li><strong>Run candidate chunk sizes</strong> with the embedder you actually deploy, at your real k. Report hit@1 <em>per question shape</em>, never pooled across shapes — pooling is exactly how a two-fact regression hides inside a flat single-fact score.</li>
<li><strong>Read the two numbers together.</strong> If multi-fact hit@1 collapses but deep-rank coverage holds, fix assembly: raise k and merge by parent document. If coverage also collapses at your k, the chunker is too fine for that query shape and the boundary has to move.</li>
<li><strong>Report the context cost of whatever you choose.</strong> Doubling k to recover assembly costs prompt characters on every turn; measure it in characters or tokens, not in vibes.</li>
</ol>
<h2 id="limitations">Limitations</h2>
<ul>
<li>Eight documents, thirteen questions, three embedders: a fixture sized to expose a mechanism in under a minute, not a benchmark. No production recall, answer-quality, latency or satisfaction claim follows from it.</li>
<li>The five two-fact pairs were placed to straddle a window boundary by construction, so 0/15 at fine granularity is a designed worst case.</li>
<li>Retrieval only: no generation, no re-ranker, no hybrid BM25, no metadata filters.</li>
<li>Character counts are not tokens, and the 6.9x ratio must not be converted to token cost without measuring a tokenizer.</li>
<li>nomic-embed-text ran without its required task prefixes, so its absolute numbers are not a fair model comparison.</li>
<li>Single-fact differences between granularities are one-query swings on an eight-question set and are not statistically meaningful.</li>
<li>One host (2 vCPU, 11 GiB, Ollama 0.20.3), localhost only; elapsed times are load-in dominated and are not throughput figures.</li>
<li>Embeddings are reproducible only for an identical model digest. Any rerun with a different digest is a different experiment, and results2.json records the digests it ran against.</li>
</ul>
<p>The result should be falsifiable in public: rerun the fixture, and if 3-sentence windows do not beat whole-document chunks on single-fact hit@1 for at least two of three embedders — or if two-fact hit@1 does not fall when moving from whole documents to sentence chunks — then this fixture is not reproducing and you should trust your own numbers over this table.</p>
<p>If you are still choosing where memory lives at all, <a href="https://baeseokjae.github.io/posts/agent-memory-architecture-guide-2026/">the framework comparison</a> covers the vendor side. This article assumes you already have a store and asks the smaller question that decides whether it works: how big is a vector&rsquo;s worth of truth.</p>
]]></content:encoded></item></channel></rss>