We've lost count of how many kickoff calls open with the same line: "We want to fine-tune a model on our data." It sounds like the serious, grown-up answer. It's a bigger budget, a bigger project, a bigger sense that you're building something proprietary.
And nine times out of ten, it's the wrong first move.
We're not anti-fine-tuning. We ship fine-tuned models and small trained classifiers in production for clients who genuinely need them. But "we need a custom model" is almost never the real diagnosis. It's a guess. Below is the actual decision tree we walk through before anyone touches a training run — the same order of operations one of our ML engineers uses on day one.
Fine-tuning is the last thing you should try, not the first
Here's the uncomfortable truth: fine-tuning is the most expensive, slowest-to-iterate, hardest-to-debug option on the menu, and it fixes the narrowest class of problems. It does not teach a model new facts reliably. It does not fix hallucination. It does not give the model access to your live data. It mostly changes behavior and format — tone, structure, adherence to a schema, a consistent style you couldn't get to stick with a prompt.
People reach for it because it feels like ownership. But you pay for that feeling with a training pipeline, a labeled dataset you have to build and maintain, an eval harness, and a re-training treadmill every time the base model updates or your data drifts. Start there and you've bought a maintenance burden before you've validated the idea.
So we go in the opposite order. Cheapest and fastest to change, first.
Try a better prompt first — most "model problems" die here
The majority of what teams call a model limitation is a prompting limitation. The model can do the task; nobody has clearly told it how.
Before we entertain anything heavier, we exhaust the prompt: a sharp system message, three to five good few-shot examples, an explicit output schema, and a clear statement of what "wrong" looks like. We add structured output when the downstream code needs it. We split one overloaded prompt into two focused steps when a single call is trying to do too much. We measure.
This is unglamorous and it works absurdly often. A support-triage client came to us convinced they needed a fine-tuned classifier for routing tickets. We spent an afternoon rewriting their prompt with real category definitions and a handful of edge-case examples. Accuracy went from "embarrassing" to "ship it." Total cost: one afternoon, zero training infrastructure, zero ongoing maintenance. If we'd fine-tuned, they'd still be relabeling data every quarter.
The rule: if you haven't spent a serious week on prompting and evals, you don't yet know what the model can't do.
Reach for retrieval when the problem is knowledge, not behavior
There's a clean line between the two heavy options, and most people blur it:
- If the model gives badly formatted or wrong-style answers → that's behavior → prompt, then maybe fine-tune.
Fine-tuning to inject knowledge is the single most common mistake we see. You cannot reliably cram your product docs, your policies, or last Tuesday's pricing into weights. You'll get a model that sounds like your docs and makes things up with total confidence. Knowledge that changes belongs outside the model, fetched at query time and dropped into context.
We won't relitigate how to build retrieval here — we've written a whole post on doing RAG without the hallucinations. The point for this decision is narrower: when the gap is "the model doesn't have the facts," retrieval is the answer, and it's dramatically cheaper and more maintainable than trying to train facts in. You update a document; the system is current the same second. No re-training, no drift, no dataset to babysit.
Prompt handles how to answer. Retrieval handles what to answer with. Get those two right and you've solved most of what people came to fine-tuning for.
Sometimes the right model isn't an LLM at all
This is the one nobody wants to hear, and it's often the best engineering call.
If your task is "put this input into one of N buckets" — spam or not, urgent or not, which of eight departments, is this transaction fraud — you may not need a generative model at all. A logistic regression or a small fine-tuned encoder (think a few-hundred-million-parameter classifier) will often beat a 70B LLM on that job: higher accuracy, a hundred times cheaper per call, single-digit-millisecond latency, and a confidence score you can actually threshold and monitor.
We had a client running every inbound event through a large model to tag it. It worked, and it cost a fortune and took 900ms a call. We replaced it with a trained classifier on their labeled history. Same accuracy, roughly 1/50th the cost, 12ms latency, and a clean probability output their ops team could tune. The LLM was a Ferrari doing a forklift's job.
Ask the honest question: is this task generative (produce language, reason, synthesize) or is it classification/extraction/ranking? If it's the latter and you have labeled data, a purpose-built ML model usually wins on every axis that matters. LLMs are a spectacular default when you don't have labels or the task is open-ended. They're a lazy default when a cheap model would crush it.
Reach for fine-tuning only when all of these are true
We do fine-tune. Here's the actual checklist. If you can't check most of these, you're not ready:
- Prompting has plateaued. You've genuinely exhausted good prompts and few-shot examples, and you can prove it with numbers, not vibes.
That last case is the strongest one for fine-tuning today: take a small, cheap model and train it to match an expensive frontier model on your specific task. Done right, it cuts inference cost hard. But you only know it worked because of the eval harness. Notice the eval harness shows up in every serious version of this — it's the thing teams skip and the thing that determines whether any of this succeeds.
The honest summary
Walk the ladder in order, cheapest first:
1. Better prompt — fixes most "model" problems in an afternoon.
Most teams that ask us to fine-tune leave with a sharper prompt, a retrieval layer, and — for the first time — a real eval harness that tells them what's actually working. That's usually the whole win.
If you're staring at a "we need a custom model" line in your own roadmap and you're not sure which rung you're actually on, that's a good conversation to have with an ML engineer before you spin up a training pipeline. Hire one of ours onto your team for a couple of weeks — we'll tell you honestly which rung solves your problem, then build it with you. That's cheaper than fine-tuning the wrong thing.