Before You Ship Your First AI Feature: A Checklist

Your first AI feature will work in the demo. That's the trap.

A demo runs once, on data you picked, while you watch. A feature runs every day, on data nobody picked, while nobody watches. Almost everything that goes wrong lives in that gap.

Since June I've run my business on a system I built for myself: a markdown vault, a desktop app and a phone app, with AI reading all of it. It writes my morning brief, keeps a summary of every client and drafts my weekly review. Most of what follows I learned by getting it wrong first.

The shape of a safe AI feature

The third step is the one teams skip. A model that writes straight into your data will, sooner or later, write a confident mistake into it. A proposal costs one click to accept and nothing to reject.

Every item below protects one of those four steps. Go through the list before the feature touches real data, not after the first incident.

The checklist

1. 1Ground itThe model reads before it answers. Hand it the files, the rows, the records, plus one rule: no invented facts, dates, numbers or states. If something is missing, it says so and asks. My system has had that rule since the first commit. It's the cheapest guardrail there is.
2. 2Make it citeEvery claim that matters names the file it came from. A citation turns "trust me" into "check me". Then check the citations in code: my evaluator flags any citation to a file that doesn't exist.
3. 3Cap the costPut a hard limit on what a background job may spend. My background AI makes at most two calls per pass, one at a time, and the sweep runs every four hours. It used to run hourly. I slowed it down to cut cost.
4. 4Design the failureBackground AI fails quietly. At first, a failed run in my system returned its error into the void, so a link that never appeared looked exactly like "nothing to link". Now every background failure lands in a debug log, and only failures go there.
5. 5Decide what it can reachLeast privilege, in code, not in the prompt. By default my agents can read and search the vault and nothing else: no shell, no web, no writes. Even a run that may write can't leave the vault. A write outside it kills the run. And remember that whatever the model reads goes to the model provider: decide what that includes before launch.
6. 6Run evals from week oneAn independent checker runs daily over every AI output: links that point at nothing, citations that don't resolve, stale reads, broken JSON. It reads, it never writes, and if it crashes, the app carries on. It proposes fixes. It never applies one.
7. 7Keep a human in the loopThe AI suggests; I accept. Suggested milestones show up as ghosts with accept and dismiss. Email triage drafts tasks, it doesn't file them. The goals at the top of my system are mine to write.
8. 8Build the kill switchOne flag stops every background AI job and empties the queue. Build it on day one and flip it once, on purpose, so you know it works before you need it.

The timeout that ate an answer

One failure mode hides better than the rest. In July my in-app assistant had a flat three-minute timeout, and a long run doing real work hit it in the middle of writing. The process was killed, and the screen showed a bland note.

A half-finished answer looked exactly like a finished one. A full analysis I had asked for never reached my files, and nothing told me.

The fix was to stop timing the run and start listening to it.

Caps before dashboards

A dashboard tells you what you spent. A cap decides it. If you only have time for one before launch, ship the cap.

Then add the dashboard: log every call with what it cost and what it did, so you can tell a busy system from a stuck one. The cap keeps a mistake small; the log tells you the mistake exists.

The order I'd do it in

1. 1Before the first prompt: decide what the model may read, what it may never touch, and where the kill switch lives.
2. 2Before the first user: citations, a cap, and a failure path that leaves a trace.
3. 3Before the second week: an eval that runs without you, and a human accept step for anything the AI wants to change.

None of this is exotic. It's the same discipline you'd apply to any dependency you don't control. The difference is that this one sounds confident when it's wrong.

An AI feature is a feature with an unreliable dependency at its centre. Ship it like one.

What I tell teams adding AI to a product