Reliability at Scale

Your dependencies will be slow, rate-limited, and occasionally down — design so users get slightly degraded service, not a dead app.

Building with AI · Lesson 34 · 12 min read

Your app is flawless in the one-user demo. Launch day: 500 people hit it at once, the model API starts returning rate-limit errors and the occasional timeout, and one of those spikes lands right as the provider has a brief outage. Your whole app goes down with it. How do you build something that stays useful even when the things it depends on fail?

Plan for partial failure — it’s normal, not exceptional

At scale, dependencies fail routinely: rate limits, timeouts, transient errors, latency spikes, the odd outage. A reliable system assumes this and degrades gracefully instead of crashing. The core tools: timeouts (never hang forever), retries with exponential backoff (ride out a transient blip without hammering), and fallbacks (a smaller or alternate model, a cached answer, or an honest “try again shortly”). The mindset shift is treating failure as an expected input, not a surprise.

Don’t make a bad moment worse

Naive handling can amplify an outage. If every client retries instantly and simultaneously, you get a thundering herd that hammers an already-struggling service. So add jitter (randomize retry timing), cap the number of retries, and respect rate limits. Smooth bursts with a queue so 500 requests don’t all hit at once. And make operations idempotent — safe to repeat — so a retry can’t double-charge or double-send. Reliability is as much about not amplifying failures as about handling them.

Graceful degradation + redundancy

When the model is unavailable, the app should still do something useful — a cached or self-check answer, a partial result, or a clear “we’re degraded, try again” — rather than a blank error page. (This app’s proxy does exactly this: every AI call degrades to a self-check when the proxy is down or over cap.) Redundancy — a backup model or provider — removes single points of failure. Honest note: 100% uptime is impossible; the realistic goal is graceful, bounded failure that users barely notice.

Surviving launch day:
1. Set a request timeout (say 30s) so a hung call fails fast instead of freezing users.
2. On a transient or rate-limit error, retry with exponential backoff + jitter, capped at 2–3 tries.
3. Still failing? Fall back: serve a cached answer, drop to a smaller model, or return “We’re busy — try again in a moment.”
4. Put a queue in front so 500 simultaneous requests are smoothed instead of all hitting the API at once.
5. Make the refund/email action idempotent so a retried request can’t fire twice.
6. Result: during the provider’s brief outage, users see slightly degraded service, not a dead app — and nothing double-fires.