Testing & CI for AI

A prompt change is a code change — so it needs a test suite that runs automatically, or you’re shipping blind.

Building with AI · Lesson 52 · 10 min read

You’ve built a real AI system across twelve modules. This final module is about keeping it working in production — the unglamorous discipline (LLMOps) that separates a demo from a product. It starts with a scary realization: in AI, a tiny change you’d never test in normal software — tweaking a word in a prompt, or a model version updating underneath you — can silently break outputs across your whole app. In regular code you’d catch a regression with tests run automatically on every change. AI needs the same, but it’s harder, and most beginners skip it entirely. Hold the question: if changing one word in a prompt can break things, how do you catch that before your users do?

Your evals are your test suite

Recall from Module 3 that an eval is a graded test of your AI’s output (lessons 9–11). The LLMOps insight is to treat your collection of evals as a regression test suite — the same role unit tests play in normal software. You assemble a set of representative inputs with known-good expectations (a “gold set”), and any change to your system — a new prompt, a new model, a retrieval tweak — gets run against that whole suite to check nothing regressed. Without this, you’re editing a system whose behavior you can’t see, changing one thing and hoping the rest still works. The eval suite turns “I think it’s better” into “it scores better on the tests, and didn’t break the others.”

CI: run the tests automatically on every change

Continuous integration (CI) is the practice of automatically running your test suite on every change before it ships — a gate that blocks changes that break things. Applied to AI: every time you edit a prompt, bump a model version, or change retrieval, CI re-runs your evals and refuses the change if scores drop below a threshold. This matters more for AI than normal software, because AI has extra ways to silently break that code doesn’t: a model provider updates their model underneath you (your untouched prompt now behaves differently), or a prompt tweak has non-obvious ripple effects (lesson 43). A human eyeballing a few outputs will miss these; an automated eval gate catches them. The mindset shift: a prompt change deserves the same rigor as a code change — because it is one.

A prompt tweak caught by CI:
• You edit the system prompt to make answers more concise.
• CI runs your 200-item eval suite: conciseness score ↑, but accuracy on a category of questions ↓ 15% (the shorter answers dropped key caveats).
• The gate blocks the change and shows exactly which category regressed → you fix it before users ever see it.
• Without CI, you’d have shipped a “concise” improvement that quietly got worse.

Testing non-determinism: thresholds, not exact matches

Here’s what makes AI testing genuinely different (and why you can’t just copy normal testing). Normal tests assert exact outputs (“this function returns 42”). But AI is non-deterministic (lesson 2 — it can give different valid answers each run) and open-ended, so exact-match assertions don’t work. Instead you test with thresholds and rates: “≥90% of these questions get an acceptable answer,” “the graded quality score stays above X,” “no answer contains a banned failure.” Grading itself often uses the deterministic checks and AI-as-judge from Module 3. So an AI test suite reports a pass rate, not pass/fail per item, and a regression is a drop in the rate. Accepting this — testing a fuzzy system with statistical thresholds instead of exact matches — is the core skill of AI testing, and it’s what lets you change a probabilistic system with confidence. (This is exactly the discipline behind the validator and gold-set gates in serious AI projects.)

Setting up AI testing for a support bot:
1. Build a gold set: 150 representative questions with known-good answer criteria.
2. Grade with deterministic checks (does it cite the policy?) + AI-as-judge (is it accurate and helpful?).
3. Wire it into CI: every prompt/model/retrieval change auto-runs the suite; block if the pass rate drops below, say, 92%.
4. Now a model-provider update or a prompt tweak that quietly regresses one category is caught by the gate — not by an angry user. Test with rates and thresholds, run automatically, ship with confidence.