Part 5 of 8 in The Last Mile of Enterprise AI. Last week: safeguards, build the brakes before the engine.
Inside many large models is a design called mixture of experts: many specialist sub-networks, with each piece of input sent to only a few. The same idea works one level up, as a way to organize a company's AI: several small specialist models, each good at one job, with a router that sends each request to the right one and passes hard cases to a large general model.
Why specialize
A frontier model is a generalist. Most enterprise work does not need that. Classifying a ticket, pulling fields from an invoice or summarizing a call are narrow, repetitive tasks with a known right answer. For those, a small model trained on your examples can match a large one at a fraction of the cost.
The evidence is suggestive, though much of it comes from vendors.
- One enterprise architecture review cited this year estimates that for about 80% of high-volume tasks, a single-GPU model performs as well as a frontier model.
- One guide reports organizations adopting the hybrid pattern seeing inference cost fall 60% to 80% within a quarter.
- Routing alone is said to cut token costs 20% to 60% at equal quality.
- Gartner projected spending on domain-specific models to grow 210% in 2026.
The four benefits
- Cost. Compute per token scales with model size.
- Speed. Small models respond faster, which matters when an agent makes dozens of calls per task.
- Control. A small model can run on hardware you own. For regulated data, that can decide whether a project is allowed.
- Stability. A model you host does not change until you change it.
How a specialist is made
The usual method is fine-tuning an open model on examples of your task. One guide puts the requirement at 1,000 to 10,000 domain examples, a single GPU for a few hours, and parameter-efficient methods such as LoRA. For simple classification, a few hundred examples can be enough.
The scarce ingredient is the examples. There are two sources, and neither needs a labeling workforce.
The review queue. The client's experts correcting the system's output (Part 1) produce labeled examples every day as a by-product of normal work.
Distillation. Have a large model do the task well on thousands of your inputs, have experts check a sample, and train a small model on the results. It is the method the big labs use for their own cheap tiers, applied to your domain.
The golden dataset from Part 2 then tells you whether the specialist is good enough to take over.
The router
It can be simple rules, a small classifier, or a confidence check where the small model tries first and escalates when unsure. It needs careful tuning. One analysis warns that a poorly tuned router either sends too much to the frontier model, erasing the savings, or too much to the small one, damaging quality.
The honest costs
CIOs interviewed by InformationWeek describe savings below expectation and costs that appear only after launch.
- Accuracy trade-offs. In one experiment, a fine-tuned 8-billion-parameter model reached 82% accuracy for $4.58, against 92% for $11.30 from a 70-billion-parameter one.
- Drift. A model trained on January's data may degrade by June as products and customers change. One source calls a monthly evaluation against a held-out test set the minimum.
- Operations. Someone must host, monitor, patch and retrain.
- Brittleness. Specialists fail abruptly on inputs outside their training. The router and the escalation path exist to catch that.
So the right measure is again cost per completed task, with retries and escalations counted.
When not to do this
Skip custom models when volume is low, when the task is open-ended reasoning, when you have no examples and no way to get them, or when nobody will maintain what you build. A frontier API with good prompts and good evals is a sound place to start and often to stay.
A sensible order
Start with a frontier model to prove the workflow. Build the evals. Collect review data. Then replace the high-volume, narrow steps one at a time with specialists, checking each swap against the golden dataset. The large model stays for hard cases.
Why this is senior work
Each decision here is a trade-off that someone has to defend: which steps to specialize, which base model, how much accuracy to give up for how much saving, where to host, when to retrain. The fine-tuning itself is now routine and largely automatable. The judgment about whether to do it at all is not.
What a senior pod does here
The tech lead runs the numbers and picks the steps worth specializing. Engineers fine-tune, build the router, deploy on the client's infrastructure and set up the monthly re-evaluation. The client ends up with models trained on its own data and running on its own hardware.
Next week: Forward deployed, remotely.