Nobody thanks the DevOps engineer. That's the job. When infra works, it's a non-event — deploys go out, dashboards stay green, and the product team ships features they think are the whole story. The infra is invisible. Then it's 2 a.m., a pager goes off, and for the next forty minutes the infra is the only thing that matters. Everything else — the roadmap, the demo, the board deck — is on hold until someone brings the system back.
We've been the someone. These are field notes from a few of those nights. Some are ours, some are from teams we were dropped into to clean up. Every one turns on a single question: did the boring work already exist, or did somebody have to invent it live at 2 a.m. with an outage running?
The deploy that couldn't go backward
A Series A fintech pushed a "small" release at 5:40 p.m. on a Thursday. It changed a serialization format on a queue consumer. Within twenty minutes, half their payment webhooks were landing in a dead-letter queue and silently not retrying.
The on-call engineer knew exactly what to revert. He couldn't. Their "rollback" was a manual redeploy of the previous Git tag through a CI job that also ran migrations — and the bad release had already altered a column. Going backward meant the old code hit a schema it didn't understand. So the real fix took three hours: hand-patching rows, writing a one-off backfill script, and babysitting the DLQ drain until 1 a.m.
The fix we put in afterward took a day. Blue-green deploys behind a load balancer, so "rollback" is a traffic switch, not a rebuild. Migrations split into expand-and-contract phases so old and new code can run against the same schema for a full release cycle. The next time a deploy went bad — and one did, six weeks later — rollback was ninety seconds and nobody woke up. That's the whole point. The good version of this story is one you never hear about.
The secret that was in the repo the whole time
We onboarded onto a codebase and did the thing we always do first: grep the history for anything that looks like a credential. `git log -p` piped through a few patterns. Thirty seconds later we had a live AWS access key, a Stripe secret, and a database password — committed eleven months earlier, "temporarily," and never rotated. The repo had been through two contractor handoffs. There was no way to know who'd cloned it.
Nothing was on fire. That's what made it dangerous. A secret in a repo doesn't page you; it just sits there being a breach that hasn't happened yet. We rotated all three keys that afternoon, moved them into a secrets manager with rotation policies, and added a pre-commit hook plus a CI scan so the next key never lands in history at all. Boring. Invisible. The kind of work that only shows up in the story that doesn't happen — the 2 a.m. call where someone's draining your S3 bucket with a key you leaked last spring.
If you take one thing from this post: `git secrets --scan-history` on your main repo, tonight. We have never run it on a mature codebase and found nothing.
The autoscaler that scaled the wrong thing
A B2B SaaS ran a big customer onboarding — a genuinely good day. Traffic tripled around 10 a.m. Their Kubernetes cluster dutifully scaled up the web tier from four pods to twelve. Latency kept climbing anyway. By noon, requests were timing out and the good day had become a support fire.
The autoscaler had done exactly what it was told. The web tier wasn't the bottleneck. Postgres was — connection count was pinned at max, and every new web pod just opened more connections and made it worse. They'd scaled the part that was easy to scale instead of the part that was actually saturated. Nobody had load-tested past 2x, so nobody knew where the real ceiling was.
The unglamorous fix: a connection pooler in front of Postgres, a read replica for the reporting queries that were eating the primary, and HPA rules keyed to the metric that actually correlated with pain instead of raw CPU. Then a load test that pushed to 5x on purpose, in staging, on a Tuesday afternoon — so the ceiling gets found by us at 2 p.m. and not by your biggest customer at noon.
The migration with no way back
The scariest ticket we've ever picked up said, in full: "run the schema migration on prod, low risk." The migration dropped a column and rewrote a table with ~40 million rows. There was no snapshot taken first. The plan was to run it live and "restore from the nightly backup if something goes wrong."
Two problems. The nightly backup had last succeeded nine days earlier — the job had been failing silently because nobody watched it. And restoring 40 million rows would have meant hours of downtime even if the backup were good. "We have backups" and "we have tested, recent, restorable backups" are different sentences that feel identical until 2 a.m.
We stopped the migration. Took a fresh snapshot, actually restored it into a scratch instance to prove it worked, fixed the backup job's alerting, and only then ran the migration — inside a transaction, on a replica first, with the whole thing rehearsed. The migration itself was uneventful. It always is, when the safety net underneath it is real. Untested backups are just a folder of hope.
The LLM outage with no plan B
Last one, and it's the modern version of all of these. A product's core feature was an LLM call — summarize this, classify that. It ran straight against one provider's API with no timeout, no retry budget, and no fallback. When that provider had a regional incident one evening, the feature didn't degrade. It hung. Requests piled up behind the stalled calls, thread pools filled, and a single upstream hiccup took down endpoints that had nothing to do with AI.
We treat a model endpoint like any other flaky dependency, because that's what it is. Hard timeouts. A circuit breaker that trips fast instead of letting requests stack. A secondary provider or a smaller local model as a graceful-degradation path, so "the AI is slow right now" replaces "the whole app is down." The feature should get worse when the model does — not disappear and take the checkout flow with it.
The pattern is the point
None of this is exotic. CI/CD with real rollback. Infrastructure as code so recovery is a `git apply`, not archaeology. Staged rollouts. Secrets in a vault, not a commit. Observability that pages on the metric that matters. Backups you've actually restored. Runbooks so the 2 a.m. engineer follows a checklist instead of improvising. It's all boring, and boring is the entire value: boring means nobody's awake.
The teams that get burned aren't reckless. They're busy shipping product, and infra is the work that's easy to defer because deferring it costs nothing — right up until the night it costs everything.
If your infra story is currently "it's fine, nobody's paged," that's either a system that's genuinely solid or one that hasn't been tested yet, and from the outside those look the same. We can tell you which one you have. Rent a DevOps engineer from our pod for a couple of weeks — Irvine and Vietnam, so someone's awake across your whole clock — and we'll walk your deploys, secrets, backups, and failure paths before 2 a.m. does it for you.