Cost Optimization

Three levers — caching, model routing, token discipline — cut the bill 10× while holding quality.

Building with AI · Lesson 32 · 12 min read

Your AI feature works and users love it. Then the monthly bill arrives and it’s 20× what you modeled — most of it from calls that didn’t need the biggest model or didn’t need to happen at all. A working feature can still be a failing business. Cost optimization is the unglamorous skill that decides whether you can afford to keep it running.

Three levers

• Caching — if the same (or semantically similar) request recurs, serve a stored answer instead of paying again. Identical inputs are free money left on the table.
• Model routing — send easy requests to a small/cheap model and reserve the frontier model for the hard ones (the Module 1 quality/latency/cost idea, applied per request).
• Token discipline — every token in the prompt and output costs money; trim bloated system prompts, retrieve fewer/better chunks, and ask for concise output.

Together these routinely cut spend by an order of magnitude with no visible quality loss.

A support bot: 40% of questions are FAQs. Cache those → 40% of calls cost $0. Route the rest: simple ones to a cheap model, only genuinely tricky tickets to the frontier model. Trim a 1,500-token system prompt to 400. Net: a fraction of the original bill, same answers.

Measure before you cut

You can’t optimize what you don’t measure. Track cost per request (and per feature), then attack the biggest line item first. And tie every cut back to your eval (Module 3): a cheaper model or shorter prompt is only a win if the pass rate holds. Cost work without an eval is how you quietly trade money for quality and never notice.

A cost pass, measured:
1. Instrument: cost per request shows the summarizer is 70% of the bill.
2. Cache repeated documents → big drop, eval pass rate unchanged.
3. Route short docs to a cheaper model → run the eval: pass rate holds at 96%.
4. Trim the prompt → re-run eval, still 96%. Bill down ~8×, quality intact — because every cut was eval-checked.