For at least five days, my app reported a failing test that had never failed on my machine. The test was right, and so was the app. They were running different versions of Node.

Since June, an independent evaluator has run over every AI output in my system once a day. It checks what code can check: links that point at nothing, citations to files that don't exist, stale reads, broken JSON contracts. It started with 17 checks across eight AI features.

It's isolated on purpose: it never writes outside its own report, never takes a slot from the AI it judges, and if it crashes, the app carries on without it. It proposes fixes and never applies one. That was the easy part; the four lessons below came from the tests themselves going wrong.

1. Prove every check can fail

In September I had five agents write fixtures for every check. Each fixture plants one realistic defect in a sandboxed copy of my real data, the check must go red, and a clean control run proves the red came from the defect. Three checks could never fail.

  • A journal-link check ended in || true. Any date-shaped string passed, including 30 February.
  • A coverage check compared a number with itself.
  • A memory check was hard-coded to pass.
A two-sided fixture
  1. 1Copy the real data
  2. 2Plant one defect
  3. 3The check must go red
  4. 4A clean run stays green

0+

of 67 check paths proven to go red on a planted defect, after that one pass

up from 10 before it

A check that has never been red is a guess. So a new guard isn't done until a mutation turns it red. This week's example: the watcher I built to run jobs on my laptops has a smoke test with 46 checks against the real code, and all 16 deliberate mutants go red.

2. Run evals where the feature runs

Back to the test that only failed inside the app. The app runs its daily self-test under its own Electron, which ships Node 20. My terminal runs Node 22.

A script wrote a large JSON blob to stdout and exited at once. In a pipe, that keeps only the first 64 KiB: I measured 65,536 of 200,000 bytes. Under the app's Electron the parent read even less, enough to cut one test's own blob, so it failed on every app run and never in my terminal.

// Before: exiting at once can cut stdout off in a pipe
process.stdout.write(json);
process.exit(0);

// After: exit only once stdout has taken every byte
process.stdout.write(json, () => process.exit(0));

Worse, the app's record of a crash was the last line of output, which was a cut JSON line instead of the error. Now the self-test records why a test failed, and the guarding test pushes 300 invented chats through the pipe, well past 64 KiB, and must parse whole under both runtimes.

3. Don't trust a gate that reads live data

My app is usually running while I work, and its background AI rewrites files under me. Yesterday a single rewrite of one goal file turned three gates red at once, including a critical: a summary cited line 63 of a file that had just shrunk to 55 lines.

None of it was my change. Before I own any red now, I stash my work and re-run at HEAD. Three for three, the failures were already there.

Before

A gate on live data

  • Asserts what the files hold today
  • Flips when the app rewrites a file
  • A red tells you nothing about your change

After

A gate on planted data

  • Asserts what the code does
  • Brings its own invented inputs
  • A red means something broke

The deeper fix is fixtures that bring their own inputs. In one pass this month, fifteen fixtures got invented inputs of their own and six stale ones started planting what they need. Live data belongs in monitoring, not in a pass or fail.

4. Test the failure path

In early September a model setting changed under my background AI, and every generation started failing. Nothing turned red. All eight generators reported success on failure, because a sync result was spread over the error object and its ok: true won.

A generator that returns success with a reason attached is lying. An eval should check the last real success, not the last tick.

What I'd tell a team adding AI

  1. 1Write the eval before the feature shipsDeterministic checks first: links, citations, contracts, freshness. A model judge comes later, and it must quote its evidence or downgrade its finding.
  2. 2Mutate every checkPlant the defect, watch it go red, keep a clean control. A check that has never failed hasn't been tested.
  3. 3Run it where users run itSame runtime, same binary, same shape of data. A green terminal proves your terminal works.
  4. 4Plant your inputsGates assert what the code does. Dashboards watch what the data holds. Don't mix them.
  5. 5Keep it out of the blast radiusRead-only, its own storage, and a crash the product never notices. An evaluator that can break the app will eventually be switched off.

Your AI feature already has a test suite. The question is whether any of it can fail.

Free · one emailThe Mobile App Launch ChecklistGet the checklist