2026-08-18 · engineering notes
A verifier that can't hallucinate
Last week I built and shipped doccrew, a small question-answering system over a markdown corpus, live at doccrew.vercel.app. It exists to explore one idea: what happens when you treat an ungrounded claim as a hard failure instead of a style problem.
The problem with “please cite your sources”
Most RAG systems handle grounding with a prompt: every sentence must cite a chunk. The model usually complies with the citation format. Whether the cited chunk actually supports the sentence is checked by nobody. The citations make answers look verified, which is arguably worse than no citations at all.
Making the gate deterministic
Doccrew runs four stages: a planner splits the question into subqueries, a lexical retriever (BM25 fused with token overlap) pulls numbered chunks, and a writer drafts an answer where every sentence must end in a citation. Then the interesting part: the verifier is not a model. It is plain Python, regex sentence splitting plus a token-overlap check against each sentence's cited chunk. A sentence survives only if it shares at least 35% of its content tokens with a chunk it cites. No citation, or no support: struck. If every sentence is struck, the product refuses to answer, and the UI shows exactly what was removed and why.
Because the gate is deterministic, it cannot hallucinate, costs nothing to run, and behaves identically regardless of which model wrote the draft. I swapped the writer from one provider to another mid-project and the grounding guarantee didn't move.
What the heuristic honestly can and can't do
Token overlap is a proxy for entailment, and I want to be precise about the trade: a sentence that paraphrases its chunk with different vocabulary can be struck unfairly, and a sentence that contradicts its chunk while reusing its words can survive. The 35% threshold is a tunable dial between those failure modes. What the heuristic buys in exchange is auditability: every verdict can be explained by pointing at tokens, which is something no LLM judge can offer. The obvious next step is an entailment model behind the same interface, and the interesting question is how much determinism that has to give up.
Refusal as a feature
The design decision I'd defend hardest: when nothing survives, doccrew says so instead of guessing. A system that visibly refuses when its evidence is thin earns more trust on the answers it does give. Watching the live trace, subqueries, retrieved chunks, the draft, then verdicts landing sentence by sentence, makes the refusal feel like rigor rather than failure.
The code is MIT-licensed on GitHub; the verifier is ~100 lines and the whole pipeline is under a thousand.