The connection pool drained in about eleven minutes. Twenty connections, all of them stuck open, waiting. The app was down, and the alert that finally fired said "database connection exhaustion" — which pointed nowhere near the actual cause, a slow vendor two hops away. The client found out from a customer tweet before they found out from their own systems. Recovery took ninety minutes, most of it spent looking in the wrong place.
What actually happened: the app had a feature that enriched user records through a third-party LLM call. It ran great for months. Then the vendor rolled out a change that pushed p99 latency from about 900ms to roughly 6 seconds under load — not an outage, just slow. The client's code had no per-call timeout, so requests piled up and held database connections while they waited. The pool drained, and the whole app went down over a feature that, on paper, still "worked."
That gap — between a thing that shipped and a thing that keeps working — is what this post is about. The feature launched. Someone demoed it, the flow ran end to end, and the hard part felt like it was behind them. It wasn't. A demo runs once, on a warm cache, with one user, on inputs someone hand-picked. Production runs forever, cold, with everyone at once, on inputs nobody imagined. Those are different jobs. We do the first one all the time — backend, ML, data pipelines, the whole build. The reason we push clients to add reliability engineering to their pod isn't that we like selling hours. It's that we've watched too many good systems quietly rot after launch because nobody owned the boring part.
Why the incident was preventable, not unlucky
None of the reliability practices that would have caught this are exotic. They're just invisible until you skip them.
A latency SLO on that endpoint would have burned error budget and paged before the pool drained. A distributed trace would have pointed at the vendor call in thirty seconds instead of an hour. A per-call timeout plus a circuit breaker would have shed the slow requests and kept the connection pool alive — the feature degrades, the app stays up. Each of those is a specific, learnable discipline, and the incident touched every one of them. The rest of this post is what they are and why they matter.
An SLO turns "it feels slow" into a number you can act on
Most teams we walk into have no agreed definition of "up." Someone says the app is slow, someone else says it seems fine, and there's no way to settle it because there's no target. So the first thing we do is write down a Service Level Objective — a real one, tied to what users actually feel.
Not "99.9% uptime" in the abstract. Something like: 95% of search requests return in under 400ms, and 99.5% of them succeed, measured over a rolling 28 days. Now "it feels slow" is a measurable claim. And the inverse of an SLO — the error budget — is the most useful management tool we've ever handed a founder. If your target is 99.5% success, you've got 0.5% to spend. When you're burning budget fast, you stop shipping features and fix reliability. When you've got budget to spare, you ship aggressively. The argument about "should we slow down and harden things" stops being a vibe and starts being arithmetic.
You can't fix what you can't see — instrument before you need to
You cannot debug production from stack traces alone. You need the three signals: metrics (aggregate health — latency percentiles, error rates, throughput), logs (what happened on one specific request), and traces (how a single request flowed across every service and vendor call it touched). Miss any one and you're guessing — which is exactly why the enrichment incident took ninety minutes instead of thirty seconds.
The trap is that observability feels optional until the exact moment you desperately need it — and by then it's too late to add. You can't instrument the past. So we wire it in during the build, not after the fire. When a request is slow, a trace tells you in seconds that 380ms of a 500ms response was spent waiting on an LLM provider, not in your own code. Without it, you'd spend a day profiling the wrong service.
Alerts should wake a human only when a human is needed
The fastest way to make on-call useless is to alert on everything. We've inherited systems with 200 alerts a day, and the team had — rationally — muted all of them. A muted pager is worse than no pager, because it comes with the illusion of coverage.
Good alerting is ruthless. Page a human only for symptoms users feel and that need action now — SLO burn rate spiking, error rate crossing a threshold, the checkout path failing. Everything else is a dashboard or a ticket, not a 3am phone call. The test we use: if the alert fires and the on-call engineer can't do anything about it right now, it shouldn't have paged. Fewer, sharper alerts mean people actually respond when the pager goes off, because it means something.
Assume your dependencies will fail, because they will
This one bites AI products especially hard — and it's exactly the trap the client fell into. If your product calls an LLM API, you've bolted your uptime to a vendor who will, on some random Tuesday, return 529s, get slow, or change behavior mid-quarter. Treat every external call as something that can and will fail.
That means timeouts on everything, retries with backoff and jitter, circuit breakers so one slow dependency doesn't cascade into total collapse, and — most importantly — a graceful degradation path. When the primary model provider is throttling, do you fail the whole request, or fall back to a secondary provider, a smaller cached model, or a "we're having trouble, try again" that doesn't lose the user's work? A product that degrades gracefully under vendor failure feels far more reliable than one that's binary up-or-down, even with identical raw uptime.
The postmortem is where you buy the fix
After the incident we ran a blameless postmortem — the point is never who typed the bad line, it's what in the system let one slow vendor take down everything. The action items were concrete: timeouts on every external call, a circuit breaker on that path, a fallback provider, and a latency SLO with alerting. An incident you don't learn from you just get to have again.
Follow-the-sun means someone competent is already awake
Reliability is also a scheduling problem. A single-timezone team means every off-hours incident either wakes someone exhausted or waits until morning while the error budget bleeds. Our pod runs Irvine plus a Vietnam bench, which lands close to round-the-clock coverage across the two. When something breaks at 3am in California, it's mid-afternoon in Vietnam — an engineer who knows the system is at a desk, coffee in hand, not fumbling for a laptop in the dark. Incidents get shorter when the responder is actually awake.
Shipping is the easy part. Keeping a thing up, fast, and honest is a separate job with its own disciplines: SLOs, tracing, sharp alerts, defensive dependency handling, real postmortems, and coverage that spans the clock. The eleven-minute pool drain wasn't bad luck. It was the boring 80% with no owner.