Part {n} of 8 in The Last Mile of Enterprise AI. Last week: human in the loop is a design, not a disclaimer.
A demo shows that something can work. Production requires that it work on the ten-thousandth document, on a Friday, after the vendor quietly updates the model. The distance between those two is where most enterprise AI projects stop.
The arithmetic of long tasks
Suppose each step an agent takes is right 95% of the time. That sounds strong. A twenty-step task then succeeds about 36% of the time, because the errors multiply: 0.95 to the twentieth power. At 99% per step, the same task succeeds about 82% of the time.
This is why agent work is unforgiving. Small gains in per-step accuracy produce large gains in whether the job finishes, and small losses do the reverse.
Benchmarks will not tell you
Published scores describe a model on a public test. They do not describe your system on your data. One summary of 2026 enterprise deployments puts the gap between benchmark scores and observed production success at about 37 percentage points, driven by brittleness on long tasks, compounding tool errors and silent degradation. Treat the exact figure loosely. The direction is not in doubt.
Failure is silent
Traditional software breaks loudly. As one engineering write-up puts it, a drop in faithfulness or relevance generates no stack trace and no alert, only quietly worse output. Things also change underneath you.
A Stanford study found one model's accuracy on a simple maths task falling from 84% to 51% within three months, with no announcement.
The fix is boring, and it works
Reliability comes from measurement, and the instrument is an evaluation suite, usually shortened to evals. It has four parts.
A golden dataset. A set of real inputs paired with the correct behavior, agreed by people who know the work. It does not need to be large. One production framework suggests 30 to 200 cases, split roughly 60% normal, 30% edge cases and 10% adversarial. Another guide recommends 200 to 500 drawn from real production failures. Quality matters more than size: these are the cases your business cares about, labeled by your experts.
Graders. Some checks are mechanical: is the output valid, were the right tools called, are the required fields present. Others need judgment, and there the common method is a second model acting as judge against a written rubric. Two rules apply. The judge should not be the same model that did the work. And the judge must be calibrated against people, because an uncalibrated judge may be measuring something other than what you think.
A gate in the build process. Every change to a prompt, a model version, a retrieval setting or a tool runs against the golden dataset before it ships. If quality drops past a threshold, the change does not go out.
Production monitoring. The golden dataset covers the failures you know. Real users find the ones you do not. So you sample live traffic, grade it, and send uncertain cases to human review.
The discipline that matters most
One rule recurs across the engineering literature: each production failure gets added to the golden dataset before the ticket is closed. After six months, the dataset is a record of every mistake the system has learned not to repeat. That is how reliability accumulates.
Error analysis before anything else
Teams often start by tuning prompts. Better to start by reading failures. Take a hundred bad outputs, sort them by cause, and count. Usually a few categories account for most of the damage: a document layout the parser mishandles, an ambiguous instruction, a missing piece of reference data. Fix the biggest category, re-run, repeat. This work is closer to quality control than to AI research, and it is where the gains are.
Engineering around the model
Several techniques raise reliability without a better model.
- Break long tasks into short ones with a check after each.
- Use ordinary code wherever the logic is fixed. Do not ask a model to add up an invoice.
- Constrain outputs to a schema so that malformed answers are rejected automatically.
- Have the system report uncertainty and send low-confidence cases to a person.
- Cap retries. One tutorial suggests two or three regeneration attempts before handing the case to a human.
What a senior pod does here
This is where senior judgment matters most. The tech lead decides what to measure, sets the thresholds, and writes the rubric the graders use. The client's experts supply the correct answers for the golden dataset, and the pod turns their review decisions into new cases every week. Engineers wire the suite into the build and the monitoring, calibrate the model judge against the experts' verdicts, and sort failures into categories so the biggest one gets fixed first. None of it needs a large team. It needs a few people who know what good evals look like and the discipline to keep them current.
A team that owns evals can change models when a cheaper one appears, because it can prove the new one is as good. A team without them is locked into whatever it has, because it cannot tell.
Next week: Productivity: redesign the workflow, don't add a tool.