The Core Idea: A language model is a frozen snapshot of the text it was trained on. It cannot know what happened after training, it cannot see your private files, and it sometimes fills gaps with confident guesses. RAG exists because these four limits are impossible to remove by making the model bigger, but easy to work around by letting the model look things up. This lesson explains each limit and shows when RAG, fine-tuning or a long context window is the better tool.
1. Beginner: What an LLM Really Knows
Key point: A model is excellent at language, reasoning patterns and widely repeated facts. It is unreliable as a database of exact, recent or private facts. RAG moves those facts out of the parameters and into documents that can be searched.
2. Beginner: The Four Limitations RAG Is Built to Fix
| Customer asks | Plain LLM problem | With RAG |
|---|---|---|
| What is the Pro plan price this month? | Old price from training data | Current price from the pricing page |
| How long is the hardware warranty? | Confident guess | Exact term from the warranty document |
| How do I reset my account in the new app? | Never saw the new app | Steps from the latest help article |
3. Intermediate: RAG vs Fine-Tuning
| Question | Fine-tuning | RAG |
|---|---|---|
| What changes? | The model parameters | The prompt content |
| Updating a fact | Prepare data and train again | Edit the document |
| Citing a source | Hard; knowledge is blended into weights | Natural; the passage is known |
| Per-user access rules | Difficult; everyone shares the same weights | Possible by filtering documents |
| Teaching style or format | Strong | Weak; prompts help only partly |
| Main extra effort | Training data and compute | Retrieval pipeline and index |
They are not rivals. Many production systems use both: fine-tune for tone and format, and use RAG for the facts. A useful rule of thumb is fine-tune for how to answer, retrieve for what to answer with.
4. Intermediate: RAG vs Long Context
When long context wins: one document or a small set that fits comfortably, one-off analysis of a whole report, or tasks that really need to read everything. Often best: use RAG to pick the relevant material, then use a generous context window to read more of it.
5. Advanced: Choosing the Right Approach, and What RAG Does Not Fix
| Your situation | Best starting point |
|---|---|
| Answers must use recent or changing facts | RAG |
| Answers must use private documents and show sources | RAG |
| Model must follow a strict style or output format | Prompting first, then fine-tuning |
| One short document, one-off question | Paste it into the context window |
| Large changing knowledge base plus a specific brand voice | RAG plus fine-tuning |
Good RAG behavior includes saying I do not know. When retrieval finds nothing relevant, the best answer is an honest refusal, not a guess. The code in section 7 shows a simple way to do this with a minimum similarity score.
6. Visual Explanation: The Knowledge Gap, Four Problems and the Cost of Context
Click to see why fewer tokens is not the only benefitExplain ▼
7. Python Implementation: Stale Memory vs Live Documents
import mathfrom collections import Counter # 1. A pretend LLM: it only knows a frozen snapshot from training timetraining_snapshot = [ ("pro plan", "The Pro plan costs 20 dollars per month."), ("warranty", "Hardware usually comes with a 12 month warranty."),] def closed_book(question): for topic, fact in training_snapshot: if topic in question.lower(): return fact return "It was probably the manager." # a confident guess with no evidence # 2. The live knowledge base: new, private and changing informationlive_documents = [ "Update 2026: The Pro plan now costs 25 dollars per month.", "Internal policy: employees get 5 extra vacation days after 3 years.", "Warranty: all hardware products include a 24 month warranty.",] stop_words = set(["the", "is", "a", "of", "how", "much", "does", "per", "who", "to"]) def vectorize(text): cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ") words = [w for w in cleaned.split() if w not in stop_words] return Counter(words) def cosine(a, b): dot = sum(a[w] * b[w] for w in set(a) & set(b)) norm_a = math.sqrt(sum(v * v for v in a.values())) norm_b = math.sqrt(sum(v * v for v in b.values())) if norm_a == 0 or norm_b == 0: return 0.0 return dot / (norm_a * norm_b) # 3. RAG: retrieve the best document, or refuse if nothing matches welldef rag_answer(question, docs, min_score=0.2): q = vectorize(question) best_score, best_doc = max((cosine(q, vectorize(d)), d) for d in docs) if best_score < min_score: return "I could not find this in the documents." return best_doc questions = [ "How much does the Pro plan cost per month?", "How long is the hardware warranty?", "Who won the office chess tournament?",] for q in questions: print("Question:", q) print(" Without RAG:", closed_book(q)) print(" With RAG: ", rag_answer(q, live_documents)) print("") # 4. The cost of pasting everything into the prompt insteadkb_tokens = 5000000rag_tokens = 1500print("Tokens if the whole knowledge base is pasted in:", kb_tokens)print("Tokens with RAG (question + top chunks):", rag_tokens)print("Reduction factor:", round(kb_tokens / rag_tokens))Question: How much does the Pro plan cost per month? Without RAG: The Pro plan costs 20 dollars per month. With RAG: Update 2026: The Pro plan now costs 25 dollars per month. Question: How long is the hardware warranty? Without RAG: Hardware usually comes with a 12 month warranty. With RAG: Warranty: all hardware products include a 24 month warranty. Question: Who won the office chess tournament? Without RAG: It was probably the manager. With RAG: I could not find this in the documents. Tokens if the whole knowledge base is pasted in: 5000000Tokens with RAG (question + top chunks): 1500Reduction factor: 3333Knowledge Check
Key Points
The Big Picture
Up Next: RAG Architecture and the End-to-End Pipeline. You will see every component in one picture: the offline indexing pipeline, the online query pipeline, and how retrieval, context and generation connect.