RAG vs Fine-Tuning vs Long Context

The one-line verdict

• Facts that change, and answers you need to cite? RAG: look up the right passages at question time.
• A consistent style, format or specialized behavior? Fine-tuning: teach the model by example.
• A small, fixed set of documents and a deadline? Long context: paste it all in and ask.

These are not rivals as much as tools for different problems. Many production systems use two of them together.

Three ways to give a model your data

Every LLM knows only what was in its training data, which stops at a cutoff and never included your company's documents. There are three ways around that:

1. Retrieval (RAG): keep your documents in a searchable index. For each question, find the most relevant passages and put just those into the prompt.
2. Fine-tuning: continue training the model on your examples, so the knowledge or behavior ends up in its weights.
3. Long context: skip the search and put whole documents into the prompt, relying on models that accept very large inputs.

Side by side

	RAG	Fine-tuning	Long context
Where your data lives	An index you search	The model's weights	The prompt, every time
Updating data	Re-index the changed document	Retrain	Just send the new version
Can cite sources	Yes, the retrieved passages	No	Partly, if you ask it to quote
Best at	Fresh, large, changing knowledge	Style, format, specialized tasks	Small document sets, quick prototypes
Main cost	Building and tuning retrieval	Training data and training runs	Tokens on every single request
Typical failure	The right passage isn't retrieved	Outdated or mixed-up facts	Missed details, high cost and latency

RAG: look it up, then answer

RAG was introduced in a 2020 paper by Lewis and colleagues, which combined a model's built-in "parametric" memory with an external, searchable "non-parametric" memory. The paper pointed at exactly the problems RAG still solves: showing where an answer came from, and updating the model's knowledge without retraining.

The practical shape is: split documents into chunks, embed them, store them in a vector index, retrieve the top matches for each question, and tell the model to answer only from them. The strengths follow directly:

• Freshness: change a document and re-index it; the next answer uses the new version.
• Citations: you know exactly which passages the model saw.
• Access control: retrieve only documents this user is allowed to see.

The weak spot is retrieval itself. If the right chunk isn't found, the model can't use it, so chunking, search quality and evaluation decide how good a RAG system is.

Fine-tuning: teach by example

Fine-tuning continues training on your own examples, changing the model's weights. Full fine-tuning updates every parameter, which is expensive for large models. Parameter-efficient methods such as LoRA freeze the original weights and train small added matrices instead. For GPT-3 175B, the LoRA paper reports about 10,000 times fewer trainable parameters and three times less GPU memory than full fine-tuning.

Where fine-tuning helps most is behavior: a house style, a strict output format, a classification task, or domain phrasing the base model handles poorly. It's a weak tool for facts. A 2023 study comparing knowledge injection found that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen before and for entirely new knowledge. Facts in weights also go stale, can't be cited, and need a new training run to change.

Long context: paste it all

Modern models accept very long inputs, so for a few documents the simplest approach is to include them in full. There's no index to build and no retrieval to tune, which makes it ideal for prototypes.

Two things limit it:

• Cost and latency grow with every request. You pay for the whole document each time, though prompt caching, which many APIs now offer, reduces the cost of repeating the same prefix.
• Long inputs aren't used evenly. The "Lost in the Middle" study found models often did best when the relevant information was at the start or end of the input, and noticeably worse when it sat in the middle, even for models built for long contexts.

A 2024 study comparing the two directly found that, with enough resources, long context outperformed RAG on average, while RAG's far lower cost remained a distinct advantage. Its proposed Self-Route lets the model decide per question whether retrieved chunks suffice or the full context is needed, keeping most of the quality at much lower cost.

A worked example: an internal HR assistant

The goal: employees ask questions about leave, travel and benefits policies.

• Start with long context. Twenty policy documents fit in one prompt. Ship a prototype in a day, and collect real questions.
• Move to RAG as it grows. Two hundred documents, updated monthly, with answers that must cite the policy section. Retrieval keeps cost flat per question, and citations build trust.
• Add fine-tuning only if needed. If answers must always follow a fixed template, such as an eligibility verdict, then the policy clause, then next steps, fine-tune a small model on examples of that format, and keep feeding it facts through RAG.

Five questions that decide it

1. Does the knowledge change? Often: RAG or long context. Rarely: any option works.
2. Must answers cite sources? Then RAG, or long context with quoting.
3. Is the problem knowledge or behavior? Knowledge points to RAG; behavior and format point to fine-tuning.
4. How much data is there? A few documents: long context. Thousands: RAG.
5. What does each request cost at your volume? Multiply long-context tokens by your daily questions before you commit.

Honest limits

• Every result here depends on the model and the task. The studies above used specific models and benchmarks; measure on your own questions.
• RAG is only as good as its retrieval. Most "the model hallucinated" reports in RAG systems are really "the right chunk wasn't retrieved".
• Fine-tuning needs good examples. A few hundred clean, consistent examples beat thousands of messy ones.

Remember this

• RAG = look it up per question: fresh, citable, cheap per request.
• Fine-tuning = teach behavior and format, not facts that change.
• Long context = simplest to start, costliest per request, and weakest in the middle.
• The best systems often combine them: fine-tune the format, retrieve the facts.