What follows is one night on call, hour by hour — stitched together from several real ones. Some were ours; some belonged to teams we were dropped into to clean up. Every hour turns on the same question: did the boring work already exist, or did somebody have to invent it live, with an outage running?
5:40 p.m. — the release goes out
A Series A fintech pushed a "small" release. It changed a serialization format on a queue consumer. Within twenty minutes, half their payment webhooks were landing in a dead-letter queue and silently not retrying.
The on-call engineer knew exactly what to revert. He couldn't. Their "rollback" was a manual redeploy of the previous Git tag through a CI job that also ran migrations — and the bad release had already altered a column. Going backward meant the old code hit a schema it didn't understand. So the real fix took three hours: hand-patching rows, writing a one-off backfill script, and babysitting the DLQ drain until 1 a.m.
The fix we put in afterward took a day. Blue-green deploys behind a load balancer, so "rollback" is a traffic switch, not a rebuild. Migrations split into expand-and-contract phases so old and new code can run against the same schema for a full release cycle. The next time a deploy went bad — and one did, six weeks later — rollback was ninety seconds and nobody woke up. That's the whole point. The good version of this story is one you never hear about.
9:15 p.m. — grepping the repo while the queue drains
Onboarding onto an unfamiliar codebase, we did the thing we always do first: grep the history for anything that looks like a credential. `git log -p` piped through a few patterns. Thirty seconds later we had a live AWS access key, a Stripe secret, and a database password — committed eleven months earlier, "temporarily," and never rotated. The repo had been through two contractor handoffs. There was no way to know who'd cloned it.
Nothing was on fire. That's what made it dangerous. A secret in a repo doesn't page you; it just sits there being a breach that hasn't happened yet. We rotated all three keys that afternoon, moved them into a secrets manager with rotation policies, and added a pre-commit hook plus a CI scan so the next key never lands in history at all. Boring. Invisible. The kind of work that only shows up in the story that doesn't happen — the 2 a.m. call where someone's draining your S3 bucket with a key you leaked last spring.
If you take one thing from this hour: `git secrets --scan-history` on your main repo, tonight. We have never run it on a mature codebase and found nothing.
11:40 p.m. — the autoscaler scales the wrong thing
Earlier the same day, a B2B SaaS ran a big customer onboarding — a genuinely good day. Traffic tripled around 10 a.m. Their Kubernetes cluster dutifully scaled up the web tier from four pods to twelve. Latency kept climbing anyway. By noon, requests were timing out and the good day had become a support fire that was still smoldering at midnight.
The autoscaler had done exactly what it was told. The web tier wasn't the bottleneck. Postgres was — connection count was pinned at max, and every new web pod just opened more connections and made it worse. They'd scaled the part that was easy to scale instead of the part that was actually saturated. Nobody had load-tested past 2x, so nobody knew where the real ceiling was.
The unglamorous fix: a connection pooler in front of Postgres, a read replica for the reporting queries that were eating the primary, and HPA rules keyed to the metric that actually correlated with pain instead of raw CPU. Then a load test that pushed to 5x on purpose, in staging, on a Tuesday afternoon — so the ceiling gets found by us at 2 p.m. and not by your biggest customer at noon.
1:15 a.m. — the migration with no way back
The scariest ticket we've ever picked up said, in full: "run the schema migration on prod, low risk." The migration dropped a column and rewrote a table with ~40 million rows. There was no snapshot taken first. The plan was to run it live and "restore from the nightly backup if something goes wrong."
Two problems. The nightly backup had last succeeded nine days earlier — the job had been failing silently because nobody watched it. And restoring 40 million rows would have meant hours of downtime even if the backup were good. "We have backups" and "we have tested, recent, restorable backups" are different sentences that feel identical until 2 a.m.
We stopped the migration. Took a fresh snapshot, actually restored it into a scratch instance to prove it worked, fixed the backup job's alerting, and only then ran the migration — inside a transaction, on a replica first, with the whole thing rehearsed. The migration itself was uneventful. It always is, when the safety net underneath it is real. Untested backups are just a folder of hope.
2:10 a.m. — the model endpoint hangs
The modern version of all of these. A product's core feature was an LLM call — summarize this, classify that. It ran straight against one provider's API with no timeout, no retry budget, and no fallback. When that provider had a regional incident that evening, the feature didn't degrade. It hung. Requests piled up behind the stalled calls, thread pools filled, and a single upstream hiccup took down endpoints that had nothing to do with AI.
We treat a model endpoint like any other flaky dependency, because that's what it is. Hard timeouts. A circuit breaker that trips fast instead of letting requests stack. A secondary provider or a smaller local model as a graceful-degradation path, so "the AI is slow right now" replaces "the whole app is down." The feature should get worse when the model does — not disappear and take the checkout flow with it.
By dawn — what the night was actually about
None of this is exotic. CI/CD with real rollback. Infrastructure as code so recovery is a `git apply`, not archaeology. Staged rollouts. Secrets in a vault, not a commit. Observability that pages on the metric that matters. Backups you've actually restored. Runbooks so the 2 a.m. engineer follows a checklist instead of improvising. It's all boring, and boring is the entire value: boring means nobody's awake.
The teams that get burned aren't reckless. They're busy shipping product, and infra is the work that's easy to defer because deferring it costs nothing — right up until the night it costs everything. If your infra story is currently "it's fine, nobody's paged," that's either a system that's genuinely solid or one that hasn't been tested yet, and from the outside those look the same. A pair of eyes across your deploys, secrets, backups, and failure paths — split between Irvine and Vietnam, so someone's awake across your whole clock — can tell you which one you have before 2 a.m. does.