Observability & Monitoring

In production you can’t fix what you can’t see — log generations, track cost and quality, trace agent runs, and get alerted before the invoice does it for you.

Building with AI · Lesson 33 · 11 min read

Your AI app is live. A user reports, “it gave a weird answer yesterday.” You have nothing: no record of the prompt sent, the model’s reply, how long it took, or what it cost. You’re flying blind. Meanwhile your bill quietly tripled and you only found out from the invoice. In production, what you can’t see, you can’t fix — and observability is how you see.

The three pillars: logs, metrics, traces

Observability has three parts. Logs record what happened — the prompt sent, the response, any errors. Metrics are numbers over time — latency, error rate, token cost, eval scores. Traces follow a single request through the system, which matters most for an agent’s plan→act→observe steps (L19). Together they answer the three questions you’ll always have in production: what happened, how often, and where did it go wrong?

What’s special about watching an AI app

AI breaks the usual monitoring assumptions. Output is non-deterministic and quality is fuzzy, so you log the actual generations (inputs + outputs) to inspect later, and you run evals continuously to catch quality drift. Token cost is a first-class metric — it’s literally your bill, and it can spike overnight. And agent runs need tracing, because their failures (L20) hide in the middle of the loop. You’re not just asking “is it up?” — you’re asking “is it still good and still affordable?”

Monitoring closes the loop with alerts

Metrics are useless if no one’s watching them. Monitoring sets automatic alerts: latency over a threshold, error rate climbing, daily cost over a budget, an eval score dropping. Alerting turns “found out from an angry user / from the invoice” into “caught it in minutes.” One honest caution: log responsibly — generations can contain user data, so handle PII and retention with care (the security lesson goes deeper). Visibility shouldn’t become a privacy leak.

Diagnosing the “weird answer” with observability:
1. Pull the logged request: you see the exact prompt, the retrieved RAG context, the model output, latency 8s, 4k tokens, $0.03.
2. The trace shows retrieval returned an irrelevant chunk — the model was grounded in the wrong text (garbage in, garbage out, from M4).
3. Metrics show the eval score dropped right after a prompt change last week — the regression is located.
4. Had an alert on “eval score < threshold” been set, you’d have caught it the day it shipped, not from a complaint.
5. Fix the retrieval/prompt, watch the metric recover. You debugged with evidence, not guesses.