RAG vs Fine-Tuning: Which One Should You Use for Your LLM?

Teams building with LLMs hit the same fork early: should the model look things up (RAG), or should it be trained on them (fine-tuning)? The two techniques solve different problems, and picking the wrong one can waste weeks of work and a lot of money. This guide explains how each approach works, compares them honestly, and gives you a simple framework and real scenarios so you can decide for your own project.

The one-sentence difference

• RAG (retrieval-augmented generation) changes what the model sees at question time. It retrieves relevant documents and puts them in the prompt.
• Fine-tuning changes the model itself by updating its weights on example data.

A useful rule to remember: RAG is for knowledge, fine-tuning is for behavior.

How RAG works

A basic RAG pipeline has five steps:

1. Chunk your documents into passages, often a few hundred tokens each, sometimes with a small overlap so ideas are not cut in half.
2. Embed each chunk, which means turning it into a vector that captures its meaning.
3. Store the vectors in an index or vector database.
4. Retrieve at question time: embed the question and find the closest chunks by similarity, often combined with keyword search for better recall.
5. Generate: paste the retrieved chunks into the prompt and let the model answer from them.

Because the knowledge lives in an index, updating it means re-indexing a document, not retraining a model. You can also show users the source passages, which builds trust and makes answers easy to verify.

How fine-tuning works

Fine-tuning continues training a pretrained model on your own examples. For chat models those examples are usually conversations that show the exact behavior you want:

{"messages": [
  {"role": "system", "content": "You are a support assistant. Reply in two sentences."},
  {"role": "user", "content": "Where is my invoice?"},
  {"role": "assistant", "content": "You can download invoices under Billing in your account settings. If it is missing, reply here and we will resend it."}
]}

Full fine-tuning updates every weight, which is expensive in memory. LoRA freezes the original weights and trains small low-rank matrices instead.

Worked example: one 4096 × 4096 weight matrix has 16,777,216 parameters. LoRA with rank 8 adds two small matrices, 4096 × 8 and 8 × 4096, which gives 65,536 trainable parameters, about 0.39% of the original. That is why LoRA fits on much smaller hardware. See for the full comparison with full training.

Side-by-side comparison

How many examples is enough? Read for realistic ranges.

A four-question decision framework

1. Is the problem missing or changing knowledge? Choose RAG.
2. Is the problem style, format or consistent behavior? Choose fine-tuning.
3. Do users need to see where an answer came from? Choose RAG.
4. Do you need a smaller, faster model that does one task well? Consider fine-tuning.

If you answered yes to both knowledge and behavior, combine them: fine-tune for the format, and use RAG for the facts.

Four scenarios

Scenario 1: A support bot for a product with weekly releases. The answers depend on documentation that changes constantly, and users want links to the source. This is a RAG problem. Fine-tuning would be out of date within weeks.

Scenario 2: A tool that must always return valid JSON in a strict schema. The model already knows how to write JSON, but it is inconsistent. Start with a better prompt and examples. If errors remain, fine-tuning on a few hundred correct examples is a good fit, because the problem is behavior, not knowledge.

Scenario 3: A writing assistant that must match a company's brand voice. Voice is style, so this is a fine-tuning candidate. Add RAG only if the assistant also needs product facts.

Scenario 4: An internal assistant over a large archive of contracts. The knowledge is private, large and sometimes updated, and lawyers need to see the exact clause. This is RAG, with careful chunking and evaluation.

The cost picture (illustrative)

RAG makes every request longer. Suppose retrieval adds 3 chunks of 400 tokens, so 1,200 extra input tokens per request. At 100,000 requests a month that is 120 million extra input tokens.

• At an assumed $0.15 per million input tokens: $18 per month.
• At an assumed $3 per million input tokens: $360 per month.

These prices are examples only, so plug in your own with the . If your prompt has a large stable prefix, caching can reduce the bill, as explained in .

Fine-tuning shifts the cost the other way. You pay once for data preparation and a training run, then you pay to serve the tuned model. Some providers charge a different rate for fine-tuned models, and self-hosting needs GPU memory, so estimate it with the before you commit.

Combining both

The two approaches are not rivals. A common production pattern is to fine-tune a small model so it follows your output format and tone reliably, then feed it retrieved documents at question time. The tuned model handles behavior, and the index handles up-to-date knowledge. This can let a smaller, cheaper model match the quality of a larger one on your specific task.

Common mistakes

• Fine-tuning to inject facts. The model may memorize them unreliably, cannot cite them, and the facts go stale.
• Weak chunking in RAG. Chunks that are too small lose context, and chunks that are too large dilute the match. Test a few sizes.
• Never checking retrieval. If the right passage is not retrieved, the model cannot answer correctly, however good it is.
• Skipping the baseline. Try a well-written prompt with a few examples first. It is the cheapest experiment.
• Training on messy data. A small set of clean, consistent examples beats a large noisy one.

How to evaluate either approach

You cannot improve what you do not measure. Build an evaluation set of 50 to 100 realistic questions with known good answers before you build anything else.

For RAG, evaluate two things separately. First, retrieval: does the correct chunk appear in the top k results? Second, generation: given the right chunk, does the model answer faithfully? For fine-tuning, compare the tuned model against the untuned model with the same prompt on held-out questions, and check for regressions on general tasks. If a change does not move these numbers, it is not worth shipping.

Recommended order

1. Improve the prompt and add a few examples.
2. Add RAG if the model lacks knowledge.
3. Fine-tune with LoRA if you still need consistent style, format or a smaller model.

Want to try the training side hands-on? Work through .

Frequently asked questions

Can fine-tuning replace RAG? Not for changing or private facts. It can teach a model a style or a skill, but it is a poor database.

Does RAG stop hallucinations? It reduces them when retrieval is good, because the model has real text to rely on. It does not remove them, so instruct the model to say when the context does not contain the answer.

Do I need a vector database? Not always. For a small set of documents, a simple in-memory index is enough. A dedicated database helps when the collection is large.

How big should chunks be? There is no universal number. Start with a few hundred tokens, then test different sizes against your evaluation set.

Checklist

• RAG for fresh, private or citable knowledge.
• Fine-tuning for style, format and narrow tasks.
• Start with prompting, then RAG, then fine-tuning.
• Build an evaluation set before you build anything else.
• Measure retrieval and generation separately.