My AI system is three weeks old. It already runs jobs on its own every hour. And nothing in it records what a single call costs.

It isn't unmonitored. Since mid-June an independent checker reads what my AI jobs produce and flags anything that isn't grounded in my files, so I know whether the answers are right. I don't know what they cost, how often each one really runs, or whether running them changed anything at all.

That gap is easy to miss, because from the inside everything looks fine. Which is exactly why I'd close it first in any new AI feature, mine included.

"It works" isn't a metric

An AI job that works can still be broken. It can run ten times when once was enough, or rewrite the same summary with the same words every hour. It can fail quietly and still report success, because a later step in the same function succeeded.

None of that shows up in a demo, a code review or a happy-path test. It shows up on an invoice weeks later, as one number with no story attached, and by then nobody remembers which feature to ask.

One row per call

The fix is cheap if you do it on day one: every model call writes one row to a ledger. Not a sample, not a daily aggregate, but every call, at the call site. An append-only file is enough to start; you can move it into a database the day you need queries.

0

ledger row for every model call

no sampling, no exceptions

at        when the call started
feature   which job or screen made it
owner     which agent or feature the result belongs to
trigger   timer, tap, conversation, retry
model     the model that actually answered
tokens    in, out, cached
cost      worked out from the tokens
ms        how long it took
outcome   ok, or failed with a failure class
changed   did the output differ from what was already there?

Most of these are obvious. Two of them carry most of the value, and they're the two people skip: trigger and changed.

Trigger: who caused this call?

A call caused by a user's tap and a call caused by a timer are different animals. When the cost moves, the first question is always "who's calling?", and without a trigger on every row nobody can answer it without reading code. It's also how you catch a job calling itself: if the trigger says "timer" on a feature you thought only ran on a tap, you've found your bug.

Changed: did it matter?

After the call, compare the output with what was already there, and if it's the same text, the same decision, nothing new, mark it. That one boolean turns a cost line into a quality signal.

It doesn't need to be clever, either: hash the new output and compare it with the hash of what's stored. Crude, cheap, and good enough to find the waste.

The health metric: calls that changed nothing

Of everything a ledger tells you, watch one number: the share of calls that changed nothing. It's what a regeneration loop looks like from the outside, long before anyone feels it in the budget.

Before

Near zero

  • Jobs run because something they read changed
  • Each call earns its cost
  • Nothing to do

After

Climbing

  • Something runs on a clock, not on a cause
  • Same input, same output, paid for again
  • A loop, or a cache that never hits

A job that regenerates on a timer can look perfectly healthy from every angle but this one. Every call succeeds, every check is green, the output is correct. And most of it is waste you'll never see unless something counts it.

What to do with the ledger

  1. 1Write the row before you ship the featureThe call site is the only place that knows the trigger and the owner. Retrofitting that later means guessing, and the rows you guessed are the ones you'll need most.
  2. 2Put the numbers on a screen you actually openCalls per day, cost per job, failures by class, the share that changed nothing. Not in a log file nobody reads.
  3. 3Cap itPer job and per day. A runaway job should hit a ceiling and stop loudly, not drain a budget quietly.
  4. 4Review the expensive ones every monthIs this job worth what it costs? Is a smaller model enough? Should it run when its inputs change instead of when the clock says so?

My hourly heartbeat already has one of these: a cap on how many jobs a single tick may run. A cap stops the worst case, but it can't tell you whether the jobs under the cap were worth running. Only the ledger can.

Metering won't make an AI feature good. It makes it honest. You can't fix a cost you can't see, and you can't defend one either.

If your AI feature shipped this quarter, add the ledger this week. If it hasn't shipped yet, you're at day one, and that's the cheapest moment you'll ever get.

Weekly-ishEngineering OS, in your inboxJoin Engineering OS