A Citation Is a Promise: Making AI Summaries You Can Open

Last Friday an AI summary in my system cited line 68 of a journal file. The file has 56 lines, and the account the sentence was about isn't mentioned anywhere in it.

A citation is the one part of an AI answer that looks like proof. file:68 reads as "I opened it, and line 68 says so". That makes an unfollowable citation the most convincing thing an AI system can get wrong.

My eval caught it. Too late.

My system writes a summary of every client and project, and every load-bearing claim has to cite the file behind it. A daily evaluator checks those citations, and it caught this one: after the summary was already in my files, in front of me, reading like a fact.

The citation parser lived inside the evaluator, so the check ran after the write. One of my citation rules had been in the prompt for almost two weeks with nothing checking it at all. A rule with no check behind it is a wish.

Check before the write

So the parser moved into one shared module with two callers. The evaluator still reports every citation in every summary, with its own weights and wording. The generator now refuses to write one it can't follow.

• Hard fail: a file that doesn't exist. A model can produce a perfectly plausible path for a file that was never there.
• Hard fail: a line past the end of the file. This incident, exactly. A line number that feels right costs a model nothing to write.
• Hard fail: a line into the summary's own file. The summary is saved into that file, which shifts its lines, so any line number there is wrong by construction. The prompt already forbade it; now code does.
• Soft: a bare name that resolves nowhere. "The pipeline row" is prose, not a citation.

The re-ask is specific: it names exactly the citations that couldn't be opened, so the model fixes those instead of rewriting everything. If the second draft still cites something it can't open, nothing is written. The old summary stays, and its "last verified against the sources" timestamp correctly doesn't move.

The re-ask fired on the very first real run. The model's first draft was still unfollowable; the repair fixed it, and the new summary cites lines that exist.

The fix that was worse than the bug

My first cut didn't refuse. It deleted the bad line number and kept the sentence.

Strip :68 and what's left resolves: the file exists, no line is named, every check is green. But the claim it supports was never in that file. The invented line number was the only evidence the harness had, and I had deleted it.

An adversarial review of my own design caught it before it shipped, and showed me that my own unit test had asserted the laundering as correct. The rule now is simple: one miss rejects the whole draft.

A check whose failure path produces text the check then passes is a laundering machine. And it raises the score for doing it.

From my system notes

What the same review found

• A prompt branch with no citation rules. The rules had been pasted into two of three branches. Eight summaries of people from the third carried citations nobody could follow. The rules are now one constant, shared.
• A range's low end was never checked. :0-5 passed.
• A thrown-away re-ask was logged as a success. The ledger said ok while the output went in the bin, which would have made the repair look free and effective at the same time.
• A web link ending in .md looked like a missing file, which under the new rule would have cost a paid re-ask.
• The score. With all of it fixed, my evals went from 98 to 100: zero critical findings, zero failures.

Followable is a floor

Here's what this check doesn't do. It proves a citation can be opened. It doesn't prove the line says what the sentence claims.

That's a floor, not a guarantee, so every message in my system says "cannot be followed", never "is false". A check that overclaims is the same bug as a model that overclaims.

Whether a line supports a claim is a judgment, and judgments get a different tool. In my evals a model judge can flag it, but only if it quotes the line, otherwise its finding is downgraded. That's a second check with its own failure modes, not a replacement for the first.

Make every citation a promise someone can check, or don't make it.