RAG vs. long context vs. fine-tuning

Before building RAG, know when not to. Three techniques get confused, and picking wrong wastes weeks. The durable mental model: RAG = knowledge; fine-tuning = behavior; long context = simplicity.

Long context — just put everything in the prompt. Modern models have enormous windows (hundreds of thousands to millions of tokens), so if your whole corpus fits, you may not need retrieval at all. Anthropic's own rule of thumb: if your knowledge base is under ~200K tokens (~500 pages), skip RAG and put it all in the prompt, using prompt caching for cost. Great for small, cohesive, rarely-changing corpora and for prototyping.

RAG — retrieve the relevant slice and add it to the prompt. Use it when your knowledge is dynamic, proprietary, larger than the context window, or needs citations. This is the default for the large majority of enterprise applications, because you can change source data without touching the model.

Fine-tuning — train the model on examples to change its behavior: tone, format, style, or to distill a big model into a cheaper one. The crucial point: fine-tuning teaches behavior, not facts. It does not reliably install new knowledge, and it can't cite sources. Don't fine-tune to "make it know our docs" — that's RAG's job.

The 2026 nuance is the huge context window. Doesn't it kill RAG? No — it reshapes it. Two facts: (1) models attend poorly to the middle of very long prompts ("lost in the middle"), so stuffing a million tokens isn't free, and (2) RAG stays competitive while using a fraction of the tokens, which is the cost and latency argument. The emerging consensus is a hybrid: retrieve to select the right evidence, then use long context to reason over it in full.

And these compose. The winning path is usually staged: exhaust good prompting first, add RAG when you hit the knowledge wall, and layer fine-tuning only if production data shows a behavioral gap. Choose the simplest thing that answers your questions accurately — then add machinery only when you can measure that it helps.