2026-08-23 · engineering notes
My own tool told me a lie
I built MemVet in a day, because agent memory has no expiry date. The memory layers record what happened in your coding sessions and hand it back later, and none of them know whether what they handed back is still true. An agent retrieves a decision from three months ago, treats it as current, and writes code against a constraint that stopped existing in April.
So in MemVet a memory is not a sentence. It is a claim plus the evidence that made it true: the commit it was recorded at, the files it depends on, the symbols inside those files and hashes of those symbol bodies. Retrieval re-checks that evidence against the code as it exists now and returns a status with a reason.
It keys on symbols rather than files on purpose. A tool that flags every touched file flags a typo fix in an unrelated function, then the next one, and gets muted inside a week. Nothing in the checking path calls a model, so a pass is deterministic, offline and fast enough to gate a pull request.
It had 40 passing tests when I stopped. That number is the subject of the rest of this post.
Postscript, 17 September: the first real run
Three weeks later I finally pointed it at a repository that was not built for it, this site, and it turned out 40 passing tests mostly proved I could imagine my own inputs.
The thesis held. I recorded four genuine decisions from this site’s history at a commit from August: the domain allowlist in the answer sanitizer, the rate limiter’s behavior when Redis is absent, the assistant being model first with scripted answers as a fallback, and a claim about my job title on the site. Then I jumped 29 commits forward to HEAD.
It flagged exactly one: the job title, which had been rewritten from Full Stack Engineer to Software Engineer in the meantime. The other three stayed marked active. That is the whole product thesis, working on code I did not write for it.
Then it broke in three ways.
Bug one: a regular expression ate the rest of the file
Then I appended a throwaway helper to the end of a file, nowhere near the tracked symbol, and MemVet reported that the tracked symbol had changed. That is precisely the failure it exists to avoid, the one that gets a tool muted inside a week.
The scanner walks a file counting braces to find where a function ends. The function it was tracking contained this:
.replace(/(?:\/[^\s)"'<>,]*)?/gi, (match) => { ... })There is a double quote inside that character class. The scanner saw it, concluded a string literal had opened, and stopped counting braces while it waited for a closing quote that was never coming. The function body silently ran to the end of the file, so every later edit anywhere in that file looked like a change to the tracked symbol.
The fixtures in my suite were small hand-written functions, none containing a regex, because when you write your own fixtures you write the code you are already thinking about. Real web code is full of regexes with quotes in them.
Bug two: fixing bug one made the tool lie
This is the one I keep thinking about. I fixed the scanner, reran the check, and a different memory now reported symbol body changed. Nothing about that function had changed. Nothing in that file had changed since the previous run.
Changing how a body is sliced changes its hash. Every hash in every existing ledger had been computed by the old scanner, so after the upgrade they all mismatched, and the tool dutifully reported that as code having changed.
Read that back slowly. A tool whose entire purpose is to stop agents acting on claims that are no longer true had started making claims that were not true, with a confident reason string attached. An upgrade quietly converted every stored memory into a false positive, and the output gave the reader no way to tell.
The fix is not clever: records carry a hash version. When the version on a record does not match the version of the adapter checking it, the baseline is recomputed from the introduction commit rather than compared across incompatible formats, and the output says that is what happened. Any tool that hashes code and persists those hashes needs this on day one. Mine did not have it.
Bug three: the silence
The last one was a design mistake rather than a parser mistake. The command that exports memories to an agent returned only fresh ones, so a decision that had drifted was simply absent from the output.
Which means the agent cannot tell a constraint that moved from a constraint that never existed. Withholding felt cautious when I wrote it. In practice it throws away the most useful thing the tool knows, which is that a decision was made here once and may no longer hold. Drifted memories are now exported with an explicit warning and the reason, and the narrow export sits behind a flag for anyone who really wants the silence.
What I took from it
A green suite measures the inputs you thought of. Everything above needed one run against code written for other reasons, and the second bug only appeared because I changed something I already believed was correct.
The landing page at memvet.vercel.app used to illustrate the idea with an invented repository and an invented function. It now runs on this run instead: the real commits, the real verdicts, the real reason strings, and a section describing all three bugs. A page arguing that claims need evidence should not make its own case with fiction.