Use retrieval-augmented generation (RAG) when the model needs your facts; use fine-tuning when it needs different behaviour. RAG fetches relevant documents at question time and gives them to the model as context. Fine-tuning continues training the model on your examples, so the change lives in the model's weights. They solve different problems, and many production systems use both.

What does RAG do well?

RAG is the right default when answers depend on knowledge that is large, private or changes often.

  • Fresh facts. Update a document and the next answer reflects it. No retraining.
  • Citations and traceability. Answers can point to the passages they came from.
  • Permissions. Retrieval can filter by who is asking, so people only get content they may see.
  • Lower risk to start. There is no training run to get wrong, and the base model can be swapped later.

RAG struggles when the task is about style or skill rather than knowledge: a strict output format, a specialised classification, a house tone of voice, or a task the base model simply does poorly even when given the right context.

What does fine-tuning do well?

Fine-tuning teaches the model a behaviour from examples. Typical reasons to do it:

  • Consistent format or tone that prompting only achieves unreliably
  • Specialised classification or extraction with domain terminology and labels
  • Smaller, cheaper, faster models. A small model fine-tuned on one task can match a much larger general model on that task, with lower latency and cost
  • Reducing prompt length. Behaviour learned into the model no longer needs pages of instructions and examples in every request

Fine-tuning is a poor way to add facts. Knowledge baked into weights goes stale, cannot cite its source, cannot be restricted by user permissions, and is hard to correct without another training run.

How do the two compare?

Question RAG Fine-tuning
Best for Facts from your documents Behaviour, format, style, specialised tasks
Updating knowledge Update the index Retrain
Citations Yes No
Per-user permissions Yes, at retrieval Not possible
Upfront effort Pipeline and evaluation Training data, training runs, evaluation
Main risk Poor retrieval Poor or too little training data; forgetting general skills

How do you fine-tune an LLM?

Modern fine-tuning rarely updates every weight of the model.

  1. Define the task and the success metric before touching data. "Better" must be measurable.
  2. Build the training data. Examples of input and ideal output, written or reviewed by people who know the domain. For many tasks a few hundred to a few thousand high-quality examples matter more than a large noisy set.
  3. Hold out an evaluation set the model never trains on, and score the base model on it first. That baseline is what the fine-tune must beat.
  4. Choose the method. Supervised fine-tuning trains on input–output pairs. Parameter-efficient methods such as LoRA train small adapter layers and leave the base weights frozen; QLoRA does this on a quantised base model to fit on less GPU memory. They are far cheaper than full fine-tuning and often close in quality for focused tasks.
  5. Train and track experiments: data version, parameters, metrics, and the resulting model.
  6. Evaluate against the baseline on the held-out set, and also on general-capability checks, because a model can improve on its task and get worse elsewhere.
  7. Deploy, monitor and retrain when data or requirements change.

When should you combine fine-tuning and RAG?

A common production pattern is to fine-tune for how to answer and use RAG for what to answer with. For example, a model fine-tuned to produce a structured, compliant summary format, fed with documents retrieved at question time. Each part does what it is good at.

A practical order of operations

  1. Write a clear prompt and test the base model against your evaluation set.
  2. Add retrieval if the failures are missing or outdated facts.
  3. Improve retrieval quality (chunking, hybrid search, re-ranking) before reaching for training.
  4. Fine-tune only where evaluation shows a behavioural gap that prompts and retrieval do not close.

This order keeps the simplest approach that meets the quality target, and gives fine-tuning a measured, justified role instead of a hopeful one.

Common mistakes

  • Fine-tuning to add knowledge that belongs in a retrieval index
  • No baseline, so nobody can say whether the fine-tuned model is actually better
  • Training on messy data: inconsistent labels teach the model inconsistency
  • Evaluating on training data, which overstates results
  • Ignoring licences and data rights for both the base model and the training data
  • No plan to retrain when data drifts

Frequently asked questions

Does fine-tuning reduce hallucinations? Not reliably. Grounding answers in retrieved sources, and training the model to say when it does not know, usually matters more.

How much data do we need? It depends on the task. Narrow, well-defined tasks can improve with a few hundred excellent examples; broader behaviour needs more. Quality and consistency beat volume.

Can we fine-tune a model and keep our data private? Yes. Open models can be fine-tuned and hosted in your own cloud account, and several commercial providers offer private fine-tuning. Match the choice to your data residency and security requirements.

Is RAG enough for an enterprise knowledge assistant? Often, yes. Start there, measure, and add fine-tuning only for a gap you can demonstrate.

Marquedata designs and builds data science, machine learning, RAG and LLM fine-tuning solutions, from training data and evaluation through MLOps and monitoring. To discuss a use case, see Data Science & ML or get in touch.