We've shipped a lot at Black Gibbon, and there's a particular kind of Slack message we've learned to dread. It usually arrives on a Tuesday, it's from a founder, and it says something like: "The AI thing is amazing, the board loved it, we're launching Thursday." That's not good news. That's a countdown.
This is the story of one of those Thursdays. Details are blurred to protect the guilty, but the failure was real and the numbers are close to true.
The demo was genuinely great, and that was the problem
A Series A company — think ops software for a mid-sized vertical — had built an AI feature that ingested a customer's messy uploaded document, extracted structured fields, and generated a draft record their users could accept or edit. In the demo it was magic. You dropped a PDF, waited four seconds, and a beautifully populated form appeared. Everyone in the room made the little "ooh" sound. The CEO recorded it on his phone.
Here's what nobody in that room understood: the demo was a single person, clicking one file, on a laptop, against a dev server with a warm cache and no other traffic. The "backend" was a Next.js API route that did, roughly, `const result = await openaiCall(fileText); return result;`. One function. No queue. No database beyond a `documents` table with an `id` and a `json` blob column. The LLM call blocked the request thread for the entire four seconds.
It worked perfectly. It was also not a product. It was a photograph of a product.
We got pulled in the week after launch, which is to say the week after it broke.
Day one of real traffic broke four things at once
They launched Thursday. By Sunday night the feature was effectively down, and the failure wasn't one bug — it was four, all stacked on top of each other, each masking the next.
The first thing that went was request timeouts. Their hosting platform killed any HTTP request that ran longer than 30 seconds. Under real load the model provider's latency went from a comfortable four seconds to fifteen, twenty, sometimes past thirty when the document was large. So requests started dying at the edge. The user saw a spinner, then a generic 504, then nothing. The document had actually been sent to the model — they were paying for the tokens — but the response evaporated because the thread that was waiting for it had already been shot.
The second thing was retries, or rather the absence of one and the presence of the wrong one. When the 504 hit, the frontend helpfully retried. So did the browser, automatically, on some connections. So one user uploading one contract generated three, four, five identical model calls, each one a fresh four-cents-to-a-dollar of tokens, each one racing to write to that same `documents` row. Their model bill for the weekend was roughly 6x what their per-document math had predicted. Nobody had put a number on "what if the same request runs five times," because in the demo it never did.
The third thing was the writes colliding. Two of those racing retries would finish near-simultaneously and both write to the same row. Whichever landed last won. We found records where a customer's extracted data had been silently overwritten by a partial result from a retry that the model hadn't even finished cleanly. No error. No log. Just quietly wrong data sitting in a table that the product treated as ground truth.
The fourth thing was the provider throttling them. Once volume climbed, the model API started returning 429s — rate limited. The code had no handling for a 429 at all. It treated it like any other failure, bubbled it up as a 500, and the user got "Something went wrong." At peak, something like one in five uploads was just failing, and the team couldn't tell whether it was a timeout, a rate limit, a bad document, or a bug, because everything failed the same anonymous way.
Four failures, and not one of them was about the AI. The model was doing its job. The problem was that there was no system around it. Somebody had wired a jet engine to a skateboard.
What the backend engineer actually built
We put one of our backend engineers on it — not a prompt person, not an "AI engineer," a backend engineer whose entire job that month was to make the magic survive contact with a crowd. Here's the unglamorous work, roughly in the order it mattered.
He got the LLM call off the request thread. The upload endpoint stopped calling the model at all. Now it did three cheap, fast things: validate the file, write a job row, and return a `202 Accepted` with a job ID. Total response time dropped from "fifteen to thirty seconds and praying" to under 200 milliseconds. The actual model work moved to a background worker pulling from a queue. This one change alone killed the timeout problem entirely, because the HTTP request no longer waited on anything slow.
He made the whole thing idempotent. Every upload got a client-generated idempotency key. If the same key showed up twice — retry, double-click, flaky network, browser being helpful — the API returned the existing job instead of starting a new one. The 6x token bleed became a non-event. You can hammer the endpoint as hard as you like now; you get one job per real request, full stop. This is maybe the least glamorous paragraph in this post and it saved them the most money.
He gave the data an actual shape. The `json` blob column became a real schema: a `documents` table, a `jobs` table with an explicit state machine (`queued → processing → succeeded / failed`), and an `extractions` table that stored each attempt with its model version and a confidence score. Now you could answer questions the blob could never answer — which documents are stuck, which model version produced this field, how many attempts a job took. The state machine also meant a crashed worker left a job in `processing` with a timestamp, so a sweeper could find stragglers and requeue them instead of losing them forever.
He treated the model provider like the flaky dependency it is. The worker got a real timeout on the model call (not the platform's 30-second guillotine — a deliberate 45-second budget it controlled). It got exponential backoff with jitter on 429s and 5xxs, capped at a few attempts, so a rate limit became a short wait instead of a user-facing error. And it got a dead-letter path: a job that genuinely couldn't complete landed in `failed` with a real reason attached, surfaced to the user as "we couldn't read this one, try again" instead of a blank 500. Suddenly the team could see, in a dashboard, exactly why anything failed.
None of this touched the prompt. The AI part — the actual clever bit everyone applauded — didn't change by a single token. What changed was everything holding it up.
The demo is the last 5%, not the first 95%
The thing we want engineers and founders to internalize is that the LLM call is the easy, finished, impressive part. It's also the small part. The demo shows you the last five percent of the work as if it were the whole thing, and that's a genuinely dangerous illusion, because it feels done. It looks done. Someone recorded it on their phone.
The other ninety-five percent is queues and timeouts and idempotency keys and a data model that can tell you what happened. It's the assumption that anything over the network will be slow sometimes, fail sometimes, and get called twice sometimes — and building for all three before real traffic teaches it to you at 2 a.m. It is boring, it does not demo, and it is the entire difference between a photograph of a product and a product.
After the rebuild, that feature ran through a Black Friday-scale spike without an incident. The AI didn't get smarter. It just finally had a backend underneath it.
If you've got an AI demo that dazzles in the room and you're not sure what's actually holding it up, that's exactly the gap our backend and infra engineers are built for. Rent the pod that makes the magic survive real traffic.