We ship a lot of launch demos. Somebody stands up in a conference room, the flow works end to end, everyone claps, and there's a real sense that the hard part is behind us. It almost never is. The demo runs once, on a warm cache, with one user, on inputs someone hand-picked. Production runs forever, cold, with everyone at once, on inputs nobody imagined. Those are different jobs, and pretending they're the same is how you end up on a Slack call at 2:14am reading logs you never set up.
This is an essay about the second job. We do the first one all the time — backend, ML, data pipelines, the whole build. But the reason we push clients to add reliability engineering to their pod isn't that we like selling more hours. It's that we've watched too many good systems quietly rot after launch because nobody owned the boring part.
The demo proves it can work; reliability proves it keeps working
A launch is an existence proof. It shows the happy path is possible. It says nothing about what happens on the ten-thousandth request, when a downstream vendor gets slow, when a deploy at 4pm on a Friday changes a default timeout, or when traffic triples because a client's sales team finally did their job.
Reliability engineering is the discipline of the second question. Not "does it work" but "does it keep working, fast, and in a way you can trust — and will you know before your users do when it doesn't." That last clause is the whole game. Everything below is in service of learning about your own problems before the customer emails you about them.
An SLO turns "it feels slow" into a number you can act on
Most teams we walk into have no agreed definition of "up." Someone says the app is slow, someone else says it seems fine, and there's no way to settle it because there's no target. So the first thing we do is write down a Service Level Objective — a real one, tied to what users actually feel.
Not "99.9% uptime" in the abstract. Something like: 95% of search requests return in under 400ms, and 99.5% of them succeed, measured over a rolling 28 days. Now "it feels slow" is a measurable claim. And the inverse of an SLO — the error budget — is the most useful management tool we've ever handed a founder. If your target is 99.5% success, you've got 0.5% to spend. When you're burning budget fast, you stop shipping features and fix reliability. When you've got budget to spare, you ship aggressively. The argument about "should we slow down and harden things" stops being a vibe and starts being arithmetic.
You can't fix what you can't see — instrument before you need to
You cannot debug production from stack traces alone. You need the three signals: metrics (aggregate health — latency percentiles, error rates, throughput), logs (what happened on one specific request), and traces (how a single request flowed across every service and vendor call it touched). Miss any one and you're guessing.
The trap is that observability feels optional until the exact moment you desperately need it — and by then it's too late to add. You can't instrument the past. So we wire it in during the build, not after the fire. When a request is slow, a trace tells you in seconds that 380ms of a 500ms response was spent waiting on an LLM provider, not in your own code. Without it, you'd spend a day profiling the wrong service.
Alerts should wake a human only when a human is needed
The fastest way to make on-call useless is to alert on everything. We've inherited systems with 200 alerts a day, and the team had — rationally — muted all of them. A muted pager is worse than no pager, because it comes with the illusion of coverage.
Good alerting is ruthless. Page a human only for symptoms users feel and that need action now — SLO burn rate spiking, error rate crossing a threshold, the checkout path failing. Everything else is a dashboard or a ticket, not a 3am phone call. The test we use: if the alert fires and the on-call engineer can't do anything about it right now, it shouldn't have paged. Fewer, sharper alerts mean people actually respond when the pager goes off, because it means something.
Assume your dependencies will fail, because they will
This one bites AI products especially hard. If your product calls an LLM API, you've bolted your uptime to a vendor who will, on some random Tuesday, return 529s, get slow, or change behavior mid-quarter. Treat every external call as something that can and will fail.
That means timeouts on everything, retries with backoff and jitter, circuit breakers so one slow dependency doesn't cascade into total collapse, and — most importantly — a graceful degradation path. When the primary model provider is throttling, do you fail the whole request, or fall back to a secondary provider, a smaller cached model, or a "we're having trouble, try again" that doesn't lose the user's work? A product that degrades gracefully under vendor failure feels far more reliable than one that's binary up-or-down, even with identical raw uptime.
The incident that taught this the hard way
Here's the one that sticks with us. A client's app had a feature that enriched user records through a third-party LLM call. It ran great for months. Then their vendor rolled out a change that pushed p99 latency from about 900ms to roughly 6 seconds under load — not an outage, just slow. The client's code had no per-call timeout. Requests piled up, held database connections while they waited, and the connection pool — 20 connections — drained in about eleven minutes. The whole app went down, and the symptom (database connection exhaustion) pointed nowhere near the actual cause (a slow vendor two hops away).
They found out from a customer tweet. Recovery took ninety minutes, mostly spent looking in the wrong place, because there were no traces to show that the time was being spent waiting on the enrichment call.
What reliability practice would have caught it: a latency SLO on that endpoint would have burned budget and paged before the pool drained. A distributed trace would have pointed at the vendor call in thirty seconds instead of an hour. A per-call timeout plus a circuit breaker would have shed those slow requests and kept the connection pool alive — the feature degrades, the app stays up. None of this is exotic. It's just work that's invisible until you skip it.
The postmortem is where you buy the fix
After that incident we ran a blameless postmortem — the point is never who typed the bad line, it's what in the system let one slow vendor take down everything. The action items were concrete: timeouts on every external call, a circuit breaker on that path, a fallback provider, and a latency SLO with alerting. An incident you don't learn from you just get to have again.
Follow-the-sun means someone competent is already awake
Reliability is also a scheduling problem. A single-timezone team means every off-hours incident either wakes someone exhausted or waits until morning while the error budget bleeds. Our pod runs Irvine plus a Vietnam bench, which lands close to round-the-clock coverage across the two. When something breaks at 3am in California, it's mid-afternoon in Vietnam — an engineer who knows the system is at a desk, coffee in hand, not fumbling for a laptop in the dark. Incidents get shorter when the responder is actually awake.
Shipping is the easy part. If you've launched and you're now realizing the boring 80% has no owner, that's the conversation we like having. Add an SRE or reliability engineer to your pod and let's put the SLOs, the tracing, and the on-call in place before the next 2am — not after it.