Most problems that sound like fine-tuning are retrieval problems

A request that arrives regularly, in various forms: we want to train a model on our company’s data so it can answer questions about our business.

It is a reasonable thing to want and almost always the wrong way to get it. The distinction that matters is simple, and getting it wrong costs months.

Behaviour versus knowledge

Fine-tuning adjusts a model’s weights on examples of desired behaviour. It is how you teach a model to respond in a particular format, adopt a house style, or handle a specialised task the base model handles clumsily. It changes how the model responds.

Retrieval supplies relevant information at the moment of the request, in the prompt. It changes what the model knows when it answers.

“Answer questions about our business” is a knowledge problem. Your policies, prices, inventory and documentation are facts. Facts belong in retrieval.

Why fine-tuning on facts goes wrong

Your facts change; the weights do not. Fine-tune on this quarter’s pricing and the model knows this quarter’s pricing until you retrain. Retrieval reads from a source of truth, so when the price changes, the answer changes.

It cannot cite. A fine-tuned model produces an answer with no provenance. A retrieval system can show which document a claim came from, which is often the difference between a system people trust and one they quietly stop using. In regulated work it is not optional.

It still invents things. Fine-tuning does not install a fact lookup; it shifts a distribution. A model trained on your documents will produce plausible-sounding statements about your business that are not in any document, and now they carry your house style, so they are harder to spot.

Access control disappears. Facts baked into weights cannot be filtered per user. With retrieval, permissions are applied at the retrieval step, and a user only ever gets context they were allowed to see. Anyone building an internal tool across departments with different access needs discovers this constraint eventually — better before the build than after.

It is far more expensive to iterate. Retraining and re-evaluating on every content change is a slow loop. Reindexing a document is a fast one.

When fine-tuning is right

It has real uses, and the pattern is consistent: it is about form, not content.

• Output format, when prompting alone does not hold a rigid structure reliably enough.
• Tone and voice, where the requirement is genuinely stylistic and hard to specify in words.
• Narrow, repetitive classification at volume, where a small fine-tuned model is cheaper and faster than a large general one.
• Domain language, where a field’s vocabulary is unusual enough that the base model misreads it.

Notice that none of these are “know our facts.” They are all “behave a particular way.” That is the line.

The combination

Serious systems often use both, and the division of labour is clear once you see it: retrieval supplies the facts, fine-tuning shapes the delivery. Retrieval first, always — establish that the right information reaches the model at the right moment, then ask whether the response format still needs work.

Doing it the other way around means debugging a system where two things could be wrong and no way to isolate which.

Where retrieval actually gets hard

Retrieval being the right answer does not make it an easy one. The failure modes just move somewhere more tractable.

Chunking decides what a retrievable unit is. Too small and you lose the context that makes a passage meaningful; too large and you dilute the match, burying the relevant sentence among irrelevant ones. Document structure usually beats fixed-size splitting — sections, clauses and headings are boundaries the author already decided for you.

Retrieval quality is where most systems actually fail, and the symptom is misleading. Pure vector similarity is good at topical closeness and bad at exact identifiers — part numbers, error codes, proper nouns. Combining keyword and vector search covers each other’s weaknesses. If your answers are wrong, measure retrieval separately from generation before touching the prompt, because a perfect prompt cannot rescue the wrong document.

Staleness. A retrieval system is only as current as its index. Reindexing needs to be automatic and monitored, or it becomes the thing that silently stops running in month four.

None of these are model problems. They are ordinary information-retrieval and data-pipeline problems, which is good news — they are well understood, debuggable, and fixable by engineers rather than by hoping a training run comes out better.

The scoping question

When someone describes a problem as needing a custom model, one question usually resolves it:

If a new employee had to answer this, would they need training, or would they need to look something up?

If they would look it up, you need retrieval. If they would need to learn a skill or a way of responding, fine-tuning might genuinely help.

Most of the time, they would look it up.