How I measure whether my agent's memory works

29 July 2026 · 8 min read

I run my coding agents on top of a personal memory system I built: durable facts live in a markdown file, an indexer embeds them with a local model (bge-m3), and a recall command does semantic search over the result. Sessions query it constantly. It looked healthy for months.

Then I asked it something I knew was in there: whether my system supports Windows. The fact existed, word for word, in the durable file. recall returned zero hits from it. Not ranked low - absent.

This post is the story of finding out why, and more usefully, the methods that made every step provable instead of plausible. If you are building agent memory, the specific bug will probably not be yours. The measurement traps will be.

The bug: the text was never embedded

My indexer chunked the memory file by markdown section. One section had grown to 904 tokens. The embedding library I use (fastembed) silently truncates input at a default of 512 tokens unless you override it, and nothing in my code did. Nothing upstream bounded chunk size either.

So the encoder saw the first 1687 characters of a 3090-character section and threw away the rest - 45.4% of the text, containing three entire facts. No error, no warning, no log line. The stored vector simply did not contain their meaning. Retrieval was not failing to find them. There was nothing to find.

The nasty property: this bug is monotone with growth. Every new fact appended to the section pushed more content past the cut. The memory system degraded because it was being used.

Proving it causally, not correlationally

"The section is long and recall is bad" is a correlation. Before touching any code I wanted the mechanism pinned, because my first diagnosis was wrong (more on that below). The probe that settled it:

The full section and its truncated prefix embed to the same vector - exact zero distance. The tail contributes nothing at all. That is not what dilution looks like; that is a step function. Correlating each fact's character offset with its self-retrieval distance gave a Spearman rho of +0.952: a cliff at the cut position, not a gradient.

This mattered because my original hypothesis was centroid dilution - one embedding averaging over many facts, dominated by the numerous ones. Dilution suggests fixes like re-ranking or splitting long sections more finely. Truncation means data is missing from the index entirely, and no amount of ranking cleverness can retrieve a vector that was never computed. Wrong mechanism, wrong fix. The ten-minute probe was cheaper than shipping the wrong repair.

The fix that looks right and the one that is

The one-line fix exists: the model genuinely supports an 8192-token window, and a single .with_max_length(8192) restores it. I rejected it for three reasons:

1. It trades a proven defect for an unproven one - the multi-fact centroid I could now actually measure. Even on a five-fact section that fits the window completely, querying any single fact's verbatim text returned its section at distance 0.21-0.34, against a 0.0000 floor for whole-chunk self-queries. Bundling costs retrieval sharpness even without truncation.
2. Quadratic attention on a 16x sequence, paid on every indexed chunk, for the benefit of exactly one oversized section.
3. It leaves the granularity wrong. Citations, quality reports, and injection payloads all wanted to address individual facts and could not.

So the fix was to make the fact the unit of embedding: the indexer now parses the memory file's per-fact structure and emits one chunk per fact. Every fact is 58-335 tokens, comfortably inside any window.

And separately - this is the part that prevents recurrence - the indexer now counts every chunk's tokens against the encoder's actual window and warns on offenders, and my health command reports the count. The window guard, not the window raise, is what stops this bug class from coming back. Four wiki documents are over the limit today; the detector names them, and that debt is visible instead of silent.

Measuring the fix instead of arguing about it

The design had one open risk I could not reason away: per-fact chunks might lose on broad queries ("what side projects am I working on"), where a big section chunk plausibly wins by covering more topics. I had accepted this as a trade-off in the design review.

Instead of accepting it, I measured it. Two indexes over the identical corpus: arm A chunked by section (exactly reproducing the live store), arm B chunked by fact. Twenty queries against both - memory-directed ones, broad ones, and eight wiki-directed queries as a counter-direction check, because an improvement that comes from crowding out the rest of the corpus is not an improvement.

The accepted trade-off inverted:

• Rank-1 distance was never worse: 12 of 12 memory-directed queries.
• Memory facts present in a default top-5 rose from 9 to 27 across queries.
• All six broad queries improved or tied. The flagship broad query went from one memory hit at d=0.4763 to three hits led by d=0.3537.
• Four narrow probes went from zero memory chunks in the top 5 to rank 1.
• The eight wiki queries kept rank 1 unchanged, all eight.

The section chunk's assumed advantage on broad queries never existed. It was an artifact of there being only two memory chunks in the index at all - of course one of them "won broad queries"; there was nothing else to return. This is my strongest argument for measuring over arguing: the risk I had formally accepted in a design document was not real.

End to end, the six-probe acceptance set went from 0/6 correct to 6/6, and every one of the twelve facts now self-retrieves at exact d=0.0000 - which doubles as a standing proof that no fact is truncated anymore.

The two ways my "before" measurement almost lied

An honest before/after table requires the "before" arm to actually measure the old behavior. Mine silently stopped doing that twice, and both failures produced the same poisonous output: a table showing no change at all.

Trap one: the old store was auto-upgraded by the new binary. My index store wipes and rebuilds itself when its schema version is outdated. The fix bumped the schema. So when I pointed the new binary at the preserved old store to get "before" numbers, the first query quietly rebuilt the entire store with per-fact chunking - converting my before-arm into a second after-arm. The before-arm has to be driven by the old binary, not just the old data.

Trap two: the old binary overwrote the new one. I built that old binary from a git worktree, but with the build cache directory shared with the main checkout - which overwrote the release binary I was using for the "after" column. Both columns were now measuring pre-fix code.

I caught it because of two smells: the after column exactly equaled the before column, and a known-good result I had measured minutes earlier had silently regressed. The rule I keep now: pin each arm to its own binary path, and assert the two binaries differ before trusting any comparison between them. "Before equals after, everywhere, exactly" is not a null result. It is an instrument failure.

A third, related trap lives in the test suite. My mock embedder hashes the full input text with no truncation, so every truncation test passes against it - including on the broken code. The causal tests require the real encoder (a 558MB model download I keep out of CI), so CI instead pins the structural properties the fix guarantees: one chunk per fact, identity fields populated. The proof runs by hand at the gate; the regression guard runs on every commit; and I wrote down which is which, so a green CI run cannot be mistaken for the causal proof.

What transfers

None of this needed special tooling - a CLI, a scratch directory for each index, and discipline about what counts as evidence. The parts I would carry to any agent-memory system:

• Prove mechanism before choosing a fix. The cheap causal probe (full vs. prefix vs. tail) killed a wrong diagnosis that would have led to a plausible, useless repair.
• Make missing data observable. The worst property of my bug was silence. The token-count guard turns the next occurrence into a warning with a filename in it. And "not measured" must render as not measured - my health check deliberately refuses to show a green tick when the data to judge is absent, because a green that certifies nothing is worse than no check.
• Measure accepted risks; they may not exist. The broad-query trade-off survived a design review as a reasonable-sounding caveat. It did not survive twenty queries.
• Distrust symmetric results. Identical before/after numbers mean your instrument broke, until proven otherwise.
• Claim what you measured, nothing more. Everything above is one corpus of 129 chunks, one embedding model, twenty queries, on my machine. It generalizes as a method, not as numbers.

The whole investigation - diagnosis, fix, and both measurement arms - is recorded as an architecture decision record and a work plan in the project's docs, which is also why I can write this post months of context later without reconstructing anything from memory. But that discipline is a story for another post.