case study · September 2026
MemVet
Memory tools record what happened in your coding sessions and hand it back later. None of them know whether what they handed back is still true. MemVet stores a decision with the evidence that made it true, and re-checks that evidence against the code as it exists now.
The failure it targets
An agent retrieves a decision from three months ago, treats it as current, and writes code against a constraint that stopped existing in April. Nothing in the loop checks. The memory reads exactly as confident as it did the day it was true, which is the property that makes it dangerous.
So a memory here is not a sentence. It is a claim plus the commit it was recorded at, the files it depends on, the symbols inside those files, and hashes of those symbol bodies. Retrieval re-checks all of it against the current commit and returns a status with a reason: active, needs revalidation, stale, superseded.
Symbols, not files
A tool that flags every touched file flags a typo fix in an unrelated function, then the next one, and gets muted inside a week. MemVet keys on qualified symbols and hashes their normalized bodies, so an unrelated function landing in the same file leaves the decision alone, while a rewrite of the function it actually rested on flags it.
Nothing in the checking path calls a model. The verdict comes from git and a parser, which makes it reproducible, offline, and fast enough to gate a pull request.
The first real run
It had 40 passing tests before it was ever pointed at a repository that was not built for it. I recorded four genuine decisions from this site’s own history at a commit from August, then jumped 29 commits forward. It flagged exactly the decision that had drifted, a job title that had since been rewritten, and left the three that still held alone. That is the result I wanted.
It also broke in three ways worth describing.
A regex ate the rest of the file. The brace scanner met a quote character inside a regular expression character class, concluded a string literal had opened, and lost count of the braces. Symbol bodies silently ran to the end of the file, so every later edit anywhere in that file reported the tracked symbol as changed. Real web code is full of regexes containing quotes. The test fixtures were not.
Improving the parser invalidated every memory. Changing how a body is sliced changes its hash. Ledgers written by the previous version now mismatched, and the tool reported that as “symbol body changed” for code nobody had touched. A tool whose purpose is to stop agents trusting false claims was making one. Records now carry a hash version, and a record from an older version has its baseline recomputed from the introduction commit.
Drifted memories were served as silence. Agent context exported only fresh decisions, so a constraint that had drifted simply vanished from the output. An agent could not distinguish a rule that had moved from a rule that never existed. Drifted decisions are now exported with an explicit warning and the reason, and the narrow export sits behind a flag.
What that taught me
A green test suite measures the inputs you thought of. Each of those three bugs needed one run against code written for other reasons, and the second only appears when you change something you already believed was correct. The landing page now runs entirely on that run: the commits, verdicts and reasons on it are the real output, because a page arguing for evidence should not illustrate itself with an invented example.