What Never Goes Into the Prompt

Last weekend the AI chat in my command center's phone app went live. Getting it to answer was one problem. Deciding what it's allowed to know is the one that deserves the time.

If you're adding AI to a mobile app, this is the decision that matters most, and it's usually made by accident. Someone adds "helpful context" to the system prompt, and from then on every message a user sends ships that context to the model.

Every turn is a disclosure

Context that's prepended to every turn isn't a convenience. It's a disclosure surface. Whatever sits in it goes out with every message, including the ones about something else entirely.

Ask a banking assistant about a card limit, and the always-on context may already have sent your address, your recent transactions and anything else someone thought might help. Nothing leaked, technically. It was just never needed for that question.

And it doesn't stay in the chat. The prompt goes to a model provider, gets logged on your server, maybe ends up in a crash report or an analytics event. Every field you add to the always-on context is a field you now have to account for in all of those places.

Three rules

1. 1An allow-list for ambient contextDecide field by field what every turn gets, and write it down as a list in code. New data doesn't join that list by default. Someone adds it, on purpose, in a change someone else reviews.
2. 2Sensitive data on demand, for one featureHealth, location, messages, money. The one feature that needs them reads them when the question needs them, scoped to that turn. Not every surface, not every turn, not every agent.
3. 3A surface without tools says "not here"If a surface can't open the data, tell the model to say so plainly and point to where the answer lives. Never instruct it to answer anyway.

The first rule does most of the work. An allow-list fails closed: forget a field and the model doesn't see it, someone notices, and it gets added on purpose. A deny-list fails open: forget a field and it ships on every turn until someone notices, if anyone ever does.

Same bytes, different disclosure

The data can be identical. The disclosure isn't. A model that reads a file because you asked about it is a different privacy story from a model that was handed the same file before you said hello.

The surface that can't look it up

My phone chat has no tools: it posts to a small server function and gets an answer back. It can't open a file, so it only knows the context it's handed with the question, while on the desktop the same assistant opens the files it needs. Same assistant, very different capabilities, and the instructions have to know the difference.

That creates a trap. If the instructions say "answer from the user's data" and the data isn't in the conversation, the model has exactly one way left to comply: produce something that sounds right.

What this looks like in a mobile app

• The API key lives on the server. Never in the binary: an APK or an IPA can be unpacked. My phone chat talks to a server function that holds the key, and the app never sees it.
• Rows are scoped in the database, not in the client's query. My cloud data is locked to one account at the database level, so a bug in the app can't widen what it reads.
• The context builder is code with tests. One function decides what goes into the prompt. Test it like any other function: given this user, assert this field is absent.
• Shared instructions name no secrets. If the same system prompt runs on the phone and the desktop, write it for the weakest surface. Assume it can be read.

I started writing rules like these in March. The instruction file my agent loads every session has had a privacy section since its first commit: client names anonymised in anything published, no financial figures in drafts, the journal private unless I ask for it.

It's much easier to design the doors before there's something valuable behind them. Retrofitting privacy into an AI feature that already ships means finding every place the context leaked to, and you won't find them all.

The safest data in an AI feature is the data the model never received.

The rule I start every AI feature with

Decide what never goes into the prompt before you decide what does. The second list grows by itself. The first one only exists if you write it.