Capstone: Ship a Real AI App

Put it all together — pick a real problem and design the whole app, from the user’s first click to a deployed, safe, evaluated product.

Building with AI · Lesson 31 · 15 min read

You’ve learned the pieces — prompting, evaluation, RAG, agents and tools, guardrails, and the shipping stack (frontend, backend, database, GitHub, deploy). Now comes the real test: can you assemble them into one coherent app? Pick a genuine problem and design the whole thing — from the user’s first click to a deployed, safe, evaluated product. That synthesis IS the skill, and this capstone makes you practice it.

The anatomy of a real AI app

Trace one request through a full app: a frontend (streaming, shows sources, holds no keys) calls a backend API — the trusted chokepoint that does auth, limits, and the model call. Behind it sits the AI work: maybe RAG (embeddings + vector search to ground answers in your data), maybe an agent loop with scoped tools and guardrails. A database (with Row-Level Security) persists users and progress. It’s all version-controlled on GitHub and deployed with secrets in env vars. Every module you finished is one organ of this single body.

Build it right: evaluate and guard

A real app isn’t done when it works once. You define evals (golden examples + clear pass/fail criteria, from Module 3) to measure quality and catch regressions as you change things. You add guardrails (least privilege, treat external input as untrusted, spend caps — Module 6). And you ship through git so every change is reviewed, safe, and reversible. Quality and safety aren’t afterthoughts bolted on at the end — they’re part of what “built” means.

Start small, then iterate

Don’t build everything at once. Ship the thinnest version that delivers real value — say, RAG Q&A over a single document source — then measure it with an eval and add capability only where the problem demands it. The whole track is a toolkit, and a good builder reaches for the right piece at the right time and resists over-engineering (you saw this with vector indexes, frameworks, and multi-agent: add complexity only when it earns its place). Match the technique to the problem, not to the hype.

A worked design for a “company docs assistant”:
1. Goal: let employees ask questions and get answers grounded in internal docs.
2. Data / RAG: chunk + embed the docs into pgvector (M4); retrieve top-k and ground with a “use only the context” prompt.
3. Backend: a /api/ask endpoint — auth, a per-user rate cap, the model call, key server-side.
4. Frontend: streaming chat that shows sources, with edit/retry and graceful errors.
5. Safety: least privilege (read-only over docs), treat doc text as data, a spend cap.
6. Eval: 20 golden Q&A pairs with criteria like “contains the right fact, or says ‘I don’t know.’” Run on every change.
7. Ship: GitHub repo + Vercel deploy, ANTHROPIC_API_KEY in env vars, PR previews.
8. Iterate: launch with one doc source; add hybrid search / rerank (M4) only if retrieval misses.