Fine-tuning vs RAG

Fine-tuning teaches behaviour — tone, format, a domain's conventions, a task the base model performs badly. RAG supplies facts at query time from a source you control. If your problem is that the model does not know something, RAG is almost always the answer, and fine-tuning facts into weights is unreliable and stale the moment the facts change. If your problem is that the model knows but answers wrongly shaped, fine-tune. Production systems frequently do both.

Fine-tuning vs RAG at a glance

Why fine-tuning is bad at facts

Training adjusts weights to make certain continuations more likely. Show a model a thousand statements about your company and it will become more likely to produce text in that shape — but there is no mechanism guaranteeing any individual fact survives intact, and no way to check which ones did short of testing each one.

Worse, it makes hallucination more confident rather than less. A model fine-tuned on your domain has learned the domain's style, so its invented answers sound exactly like its correct ones. Without retrieval there is nothing to check them against.

And facts change. A fine-tune is a snapshot: prices, policies, staff, product details all drift, and every drift is a retraining cycle. RAG updates by reindexing a document.

What fine-tuning is genuinely good at

Output shape. If you need every response in a specific JSON structure, or in a house style, or following a clinical or legal convention, a few hundred good examples teach that far more reliably than instructions in a prompt — and without spending context tokens on those instructions every single call.

Narrow task performance. A small model fine-tuned on one classification or extraction task routinely matches a much larger general model on that task, which matters when you are running locally and the large model does not fit.

And implicit conventions — the things domain experts do without being able to articulate them, which are therefore hard to write into a prompt but visible across enough examples.

Doing both, in the right order

The standard production shape is RAG for knowledge and a fine-tune for behaviour: retrieval finds the relevant documents, and a fine-tuned model has learned how to use retrieved context, when to say the answer is not in the sources, and what format to answer in.

The order matters for your own effort. Build RAG first. It is faster, it usually solves more of the problem than expected, and it tells you what is actually wrong — if answers are correct but badly shaped, you now know a fine-tune is worth it, and you have real examples to train on.

Starting with the fine-tune means spending days on GPU time before learning whether the problem was ever about behaviour. That is the expensive mistake, and it is common.

Work it out for your own setup

Common questions

Can I fine-tune facts into a model if I really want to?

You can train on factual statements, and some of them will stick. You will not know which, you cannot cite a source, and every change requires retraining. For a stable, closed body of knowledge it is occasionally defensible; for anything that moves it is the wrong tool.

Does RAG need a vector database?

Not always. For a few thousand chunks, an in-memory index or even a well-built keyword search can be enough, and hybrid retrieval combining both often beats vectors alone. A dedicated vector store earns its place at scale, not at the start.

How much data do I need to fine-tune?

For style and format, a few hundred high-quality examples often suffice with LoRA. For a genuinely new capability, thousands. Quality dominates quantity — a small consistent set beats a large noisy one.

Can I do either of these locally?

RAG, comfortably: an embedding model and a vector index run on modest hardware. Fine-tuning locally depends on method — QLoRA puts a 7B or 8B model within reach of a 12GB card, while full fine-tuning of the same model needs far more memory than any consumer GPU has.