Before you fine-tune anything, walk these gates in order. Each one is cheaper and faster to change than the one after it, and most teams stop long before the last. Fine-tuning sits at the bottom on purpose: it's the most expensive, slowest-to-iterate, hardest-to-debug option on the menu, and it fixes the narrowest class of problems. It does not teach a model new facts reliably. It does not fix hallucination. It does not give the model access to your live data. It mostly changes behavior and format — tone, structure, adherence to a schema, a consistent style you couldn't get to stick with a prompt.
We do ship fine-tuned models and small trained classifiers in production for clients who genuinely need them. But "we need a custom model" is a guess, not a diagnosis. This is the decision path an ML engineer actually walks before touching a training run.
Gate 1: Have you exhausted the prompt?
If you haven't spent a serious week on prompting and evals, stop here — you don't yet know what the model can't do. The majority of what teams call a model limitation is a prompting limitation. The model can do the task; nobody has clearly told it how.
Exhaust the prompt before anything heavier: a sharp system message, three to five good few-shot examples, an explicit output schema, and a clear statement of what "wrong" looks like. Add structured output when the downstream code needs it. Split one overloaded prompt into two focused steps when a single call is trying to do too much. Then measure.
This is unglamorous and it works absurdly often. A support-triage client came to us convinced they needed a fine-tuned classifier for routing tickets. We spent an afternoon rewriting their prompt with real category definitions and a handful of edge-case examples. Accuracy went from "embarrassing" to "ship it." Total cost: one afternoon, zero training infrastructure, zero ongoing maintenance. If we'd fine-tuned, they'd still be relabeling data every quarter.
Pass this gate only if: prompting has genuinely plateaued and you can prove it with numbers, not vibes.
Gate 2: Is the problem knowledge, or behavior?
There's a clean line here, and most people blur it:
- If the model gives badly formatted or wrong-style answers → that's behavior → continue down the path.
Fine-tuning to inject knowledge is the single most common mistake we see. You cannot reliably cram your product docs, your policies, or last Tuesday's pricing into weights. You'll get a model that sounds like your docs and makes things up with total confidence. Knowledge that changes belongs outside the model, fetched at query time and dropped into context.
We won't relitigate how to build retrieval here — we've written a whole post on doing RAG without the hallucinations. The point for this decision is narrower: when the gap is "the model doesn't have the facts," retrieval is the answer, and it's dramatically cheaper and more maintainable than trying to train facts in. You update a document; the system is current the same second. No re-training, no drift, no dataset to babysit.
Pass this gate only if: the gap is behavior — consistent tone, format, or style — not fresh facts.
Gate 3: Does this task even need a generative model?
This is the one nobody wants to hear, and it's often the best engineering call. If your task is "put this input into one of N buckets" — spam or not, urgent or not, which of eight departments, is this transaction fraud — you may not need a generative model at all.
A logistic regression or a small fine-tuned encoder (think a few-hundred-million-parameter classifier) will often beat a 70B LLM on that job: higher accuracy, a hundred times cheaper per call, single-digit-millisecond latency, and a confidence score you can actually threshold and monitor.
We had a client running every inbound event through a large model to tag it. It worked, and it cost a fortune and took 900ms a call. We replaced it with a trained classifier on their labeled history. Same accuracy, roughly 1/50th the cost, 12ms latency, and a clean probability output their ops team could tune. The LLM was a Ferrari doing a forklift's job.
Ask the honest question: is this task generative (produce language, reason, synthesize) or is it classification/extraction/ranking? If it's the latter and you have labeled data, a purpose-built ML model usually wins on every axis that matters. LLMs are a spectacular default when you don't have labels or the task is open-ended. They're a lazy default when a cheap model would crush it.
Pass this gate only if: the task is genuinely generative, or you have no labels to train a classifier on.
Gate 4: The fine-tuning checklist
You've cleared the first three gates. Now the actual checklist. If you can't check most of these, you're still not ready:
- Prompting has plateaued. You've genuinely exhausted good prompts and few-shot examples, and you can prove it with numbers, not vibes.
That last case is the strongest one for fine-tuning today: take a small, cheap model and train it to match an expensive frontier model on your specific task. Done right, it cuts inference cost hard. But you only know it worked because of the eval harness. Notice the eval harness shows up in every serious version of this — it's the thing teams skip and the thing that determines whether any of this succeeds.
Where most teams actually land
Read the path from the top and the ordering is the whole point:
1. Better prompt — fixes most "model" problems in an afternoon.
Most teams that ask us to fine-tune leave with a sharper prompt, a retrieval layer, and — for the first time — a real eval harness that tells them what's actually working. They pass Gate 1 or Gate 2 and never reach the bottom. That's usually the whole win: not a training pipeline, but knowing which rung actually solves the problem before spending a dollar on the wrong one.