How to — decide if your model needs fine-tuning

When an AI feature misbehaves, "just fine-tune it" is the most popular and most expensive wrong answer. This is the order of operations that tells you whether the weights are actually the problem — before you spend a fraction of your budget finding out.

The tell is rarely dramatic. Your support bot starts inventing refund policies it was never given, or a summarizer keeps ignoring the output format your whole product depends on. Someone on the team says the model "just needs training on our data," and suddenly there is a GPU bill and a two-week plan. Slow down. Most of what reads as a model problem is a briefing problem, a retrieval problem, or a measurement problem — and each of those is cheaper to fix than retraining.

The mental model to carry through all six moves: a fine-tune changes the model itself, while prompting and retrieval change only what the model sees. Our explainer What is fine-tuning? covers the mechanics — the short version is that fine-tuning is a nudge to the weights, not a rebuild. This piece is about when to pull that lever and when to walk away from it.

The six moves

1. Name which of the three failures you actually have. Write the bad output down, then classify it. Is the model behaving wrong — wrong tone, wrong format, ignoring rules? Is it knowing wrong — missing facts about your product, your docs, last week? Or is it costing wrong — a fat prompt resent on every request, a bigger model than the task needs? Three diagnoses, three different treatments, and only one of them involves training. Teams that skip this step fine-tune their way out of a retrieval problem and wonder why the model still can't quote the price list.

2. Fix the briefing before the weights. Paste your clearest instructions into the system prompt and a handful of worked examples showing input and desired output, side by side. This is the cheapest intervention in AI engineering, and its hit rate is embarrassingly high. If one clean example cures the failure, stop — you are done, and the system prompt is the right home for the fix because you can change it on a Tuesday without a training run. Providers publish formal prompting guidance for exactly this reason; treat it as step one, not a consolation prize.

3. Run the paste test to separate knowledge from training. Take a real failing question, find the passage in your documents that answers it, and paste that passage directly into the prompt. If the model now answers correctly, it never lacked the capability — it lacked the page. That is a retrieval problem, and retrieval means RAG: the system looks facts up at answer time instead of relying on memorization. Fix the lookup. And once you ship it, verify the retrieval is actually firing — our guide How to — know if RAG is actually doing anything walks through the ablation check. If the model still gets it wrong with the exact source in front of it, that is information that training could genuinely help.

4. Build the eval set before you touch training. Collect 50 to 200 real cases — every failure mode you've seen plus the outputs that currently work and must keep working — and write down what a pass looks like. This step is the decision. If you cannot score today's model on your own cases, you cannot score a fine-tuned one either, and you'll be evaluating vibes against a bill. The eval also gives the fine-tune its raw material: the training examples are drawn from the same definition of "good," which is the only way to know the fix is real. Providers like OpenAI now build the tooling around this loop — upload examples, train, measure against held-out data — because the measurement, not the training, is where projects die.

5. Count the tokens you're re-paying forever. If your fix from step 2 means a 900-token instruction block and twelve examples resent on every request, that overhead is billed every single call, forever, and it also eats room from the user's actual content. A fine-tune burns the behavior into the weights so the prompt goes quiet. This is where volume decides: at a thousand calls a day, the fat prompt is pocket change; at a million, the same prompt is a line item that justifies training. Run the arithmetic on your own pricing page before deciding — and remember the arithmetic favors fine-tuning more when the per-request penalty is long and the traffic is high.

6. If you train, train small and look for collateral damage. Nobody with a modern option should reach for a full-model retrain first. Low-rank methods — the family of techniques our explainer What is LoRA? covers — freeze the original model and train a small adapter on top, at a fraction of the compute, and they are the default way teams customize open-weight models. Then run the second half of your eval set — the cases that already worked — and score them again. Fine-tuning degrades things on purpose by accident: a study from Zhejiang and Swansea universities, accepted at IJCAI 2026, found that the harder teams tuned a popular vision-language model's image side, the worse it got on new, unseen data, while tuning the text side often gained more for less. A separate continual-learning study documented the same pattern more bluntly — aggressive fine-tuning erases previously learned skills. A fine-tune that fixes your five bad cases and quietly breaks fifty good ones is a regression dressed as progress.

Don't do this

Don't fine-tune to teach the model facts. Retraining on your product catalog so the bot "knows" your prices is the most common misuse of the technique: the facts sit in weights where you can't audit, update, or date them, the model can forget old knowledge while absorbing new, and the next vendor model release strands your whole training run on an obsolete base. Facts change weekly; weights don't. Facts belong in retrieval. Training belongs in behavior.

How you'll know it worked

Three signals, all measurable. Your eval score on the failure cases moves — that's the fix. Your eval score on the keepers holds — that's the absence of collateral damage. And your live prompt gets shorter while your per-request cost drops — that's the fine-tune earning back what it cost. If any one of the three is missing, you don't have a tuned model; you have a more expensive guess.

What's the most expensive-looking AI problem you ever fixed with a cheap instruction change? Tell us in the comments.

Sources: OpenAI — Fine-tuning guide · arXiv — LoRA: Low-Rank Adaptation of Large Language Models · Hugging Face — Parameter-Efficient Fine-Tuning (PEFT) blog · arXiv — An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning · arXiv — A3B2: adaptive asymmetric adapter (IJCAI 2026)