Prompt & Version Management
Your prompt is production code and your model is a live dependency — so version them, roll them out carefully, and be able to roll back in seconds.
Last lesson: a prompt change is a code change, so it needs tests. This lesson takes that seriously and asks the obvious follow-up: if prompts are code, why do so many teams manage them like sticky notes — edited live, with no history, no rollout plan, and no way to undo? And there’s a second, sneakier dependency you don’t even control: the model itself, which a provider can change under you. Managing both like the production dependencies they are is a core LLMOps skill most people learn the hard way. Hold the question: if your prompt controls your whole app’s behavior, how should you treat changing it — and what about the model you don’t own?
Prompts are code — version them
Your prompt often controls more of your app’s behavior than any other single thing — so it deserves the discipline you’d give critical code. That means version control: every prompt change is tracked, with a history of what changed, when, and why, and each version tied to its eval scores (lesson 52). Editing prompts live in a text box with no history is like editing production code directly on the server with no backups — fine until the day it isn’t. Versioning gives you three superpowers: you can see what changed when behavior shifts, compare versions’ eval scores, and — critically — roll back instantly to a known-good version if a change goes wrong. A prompt without a version history is a liability waiting to bite.
The model is a dependency you don’t control — pin it
Here’s the subtle one. Your app depends on a model you don’t own, and providers update models — sometimes silently. That means your app’s behavior can change without you touching anything (the silent-breakage problem from lesson 52). The defense is version pinning: wherever possible, specify the exact model version you tested against (e.g. a dated snapshot), rather than a floating “latest,” so an upstream update can’t change your behavior until you choose to adopt it — at which point you re-run your evals first. When you do upgrade models, treat it exactly like a big code change: test it against your suite, roll it out carefully, watch for regressions. Un-pinned, you’re building on ground that can shift beneath you; pinned, you decide when the ground moves.
Two teams when a provider updates the model: • Team A pins nothing (“latest”). Overnight the model updates; their untouched app starts formatting answers differently and failing a downstream parser. Mystery outage, no code changed. • Team B pinned the exact model snapshot. The provider’s update doesn’t touch them; later, they choose to test the new version against their evals, confirm it’s good, and adopt it deliberately. • Same event; only Team B stayed in control.
Roll out changes carefully — and keep the escape hatch
Because prompt and model changes can have wide, hard-to-predict effects, you don’t just flip them on for everyone. Borrow the deployment playbook: staged rollout (release the change to a small percentage of traffic first, watch the metrics, then expand) and A/B testing (run old vs new side by side on real traffic and compare quality — the honest way to answer “did this prompt change actually help?”, building on measuring changes from lesson 43). And always keep the escape hatch: instant rollback. When a change misbehaves in production, your first move isn’t to debug live under pressure — it’s to revert to the last known-good version in seconds (which requires the versioning from section 1), then investigate calmly. The whole mindset: treat prompts and models as versioned production dependencies — tracked, tested, rolled out gradually, and instantly reversible. That’s the difference between an AI app you operate and one that operates you.
Imagine a restaurant that changes its signature recipe. A reckless owner scribbles over the only copy of the recipe card in pen, serves the new version to every table at once, and has no idea what the old recipe was if it flops. A professional keeps every version of the recipe on file (version control), tries the new one on a few tables first and compares reactions (staged rollout / A/B), and can instantly go back to last week’s beloved version if the new one bombs (rollback). And they don’t let a supplier swap their flour brand without noticing (model pinning). Your prompts and model are that recipe and those ingredients — treat changing them with a pro’s care, not a pen and a shrug.
Shipping a prompt improvement the professional way: 1. Version it: commit the new prompt with a note on what/why, keeping the old version on record. 2. Gate it: CI runs the eval suite (lesson 52); it must pass the thresholds. 3. Roll out staged: send 5% of traffic to the new prompt, watch quality metrics, then expand if healthy — or A/B it against the old. 4. Keep the escape hatch: if something regresses in production, roll back to the last known-good version in seconds, then investigate. (And pin your model version so a provider update can’t blindside you.)
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →