RAG Architecture and the End-to-End Pipeline

See the complete RAG system in one picture: the offline indexing pipeline, the online query pipeline, and how retrieval, context and generation connect.

35 minBeginnerCode Examples

The Core Idea: Every RAG system is really two pipelines joined by one shared store. An indexing pipeline runs ahead of time and prepares your documents so they can be searched. A query pipeline runs every time a user asks a question: it searches the prepared store, builds a prompt from what it finds, and asks the model to answer. Once you can picture these two pipelines, every later lesson is just a closer look at one box.

1. Beginner: One System, Two Pipelines

Think of a restaurant. Before opening, the kitchen does the prep: washing, chopping and storing ingredients so they are ready. When a customer orders, the kitchen does the service: it grabs the right prepared ingredients, cooks the dish and serves it fast. Nobody chops onions while the customer waits.
Indexing pipeline = prep kitchen
Collects documents, cleans them, cuts them into chunks, converts each chunk into numbers, and stores everything in a searchable index. It is offline: it runs before users ask anything, or in the background when documents change.
Query pipeline = service
Takes one user question, finds the best chunks in the index, builds a prompt and generates the answer. It is online: a real person is waiting, so it must be fast.

The meeting point: the index. The indexing pipeline writes to it and the query pipeline reads from it. Keeping the heavy work on the writing side is what makes answering quick.

2. Beginner: The Indexing Pipeline (Offline)

Indexing has five stages. Each one turns the data into a form that is a little more ready for search.
Stage 1 · Load. Read documents from their sources: PDFs, web pages, wikis, databases. The output is raw text plus information about where it came from.
Stage 2 · Clean. Remove junk such as page headers, menus and duplicate copies, and normalize the text so it is consistent.
Stage 3 · Chunk. Split long documents into smaller pieces called chunks. A model cannot search or read a whole book at once, and small focused pieces match questions better. A 3,000-word document cut into pieces of about 200 words gives roughly 15 chunks.
Stage 4 · Embed. Convert each chunk into an embedding: a list of numbers that captures its meaning, so that chunks about similar things end up with similar numbers.
Stage 5 · Store. Save each embedding together with its original chunk text and metadata (the source file, a document id, a date) in a vector store, so results can later be shown with their sources.

Why do it offline? Embedding thousands of chunks takes time and money. Doing it once, and again only for new or changed documents, means no user ever waits for it.

3. Beginner: The Query Pipeline (Online)

When a user types a question, the system runs six steps, usually in well under a few seconds:
1 · Receive
Accept the question and any extra information, such as which user is asking.
2 · Embed the query
Turn the question into an embedding, using the same embedding model as the documents.
3 · Retrieve
Search the index and return the top-k chunks whose embeddings are closest to the question.
4 · Build the prompt
Combine instructions, the retrieved chunks and the question into one prompt.
5 · Generate
Send the prompt to the LLM, which writes an answer using the supplied context.
6 · Return
Show the answer to the user, ideally with the sources it came from.

One rule you must never break: the question and the documents must be embedded by the same embedding model. Embeddings from two different models live in different number spaces, so comparing them is meaningless.

4. Intermediate: Offline vs Online Processing

Because the two pipelines run at different times, they have different goals. Good engineers design them separately.
AspectIndexing (offline)Query (online)
When it runsBefore launch, then on a schedule or when documents changeEvery time a user asks
Main goalQuality and completeness of the indexLow latency and good answers
Speed pressureLow: can take minutes or hoursHigh: users wait
Typical costEmbedding every chunkOne query embedding, one search, one LLM call
If it failsIndex is missing or staleUser sees an error or a bad answer right now
Touches the indexWritesReads

Design consequence: any step that can be moved from online to offline usually should be. Pre-computing embeddings, cleaning text and attaching metadata all happen in indexing so the query path stays short.

5. Advanced: The Context Budget and Where the Pipeline Grows

The heart of the query pipeline is the hand-off retrieval → context → generation. Retrieval returns a ranked list of chunks with scores. Context construction decides which chunks to keep, in what order, and how to format them. Generation then reads that context together with the instructions and the question.
Everything the model reads, plus the answer it writes, must fit inside the context window. This gives a simple budget rule:
Tprompt+Tanswer≤TwindowT_{prompt} + T_{answer} \le T_{window}
Here TT counts tokens. For example, take 4 chunks of about 300 tokens each, a 100-token instruction and a 30-token question. The prompt is 4×300+100+30=1,3304 \times 300 + 100 + 30 = 1{,}330 tokens. If you reserve 500 tokens for the answer, the total is 1,830 tokens, comfortably inside typical windows. The budget matters because retrieving more chunks is not free: extra text costs money, adds delay, and can bury the one passage that matters.
The basic pipeline above is the foundation. Real systems add stages around it, and each one has its own lesson later in this course:
Added stageWhere it goesWhat it improves
Query rewritingBefore retrievalVague or messy questions
Metadata filteringDuring retrievalRelevance, permissions, freshness
Hybrid search and rerankingRetrieval and after itRanking quality
CitationsGeneration and outputTrust and checking
Evaluation and monitoringAround the whole systemFinding and fixing failures

6. Visual Explanation: Two Lanes, One Index, and Where Things Break

Panel A draws the architecture as two lanes that meet at the index. Panel B shows what the data looks like after each stage. Panel C is a debugging explorer: click a stage to see what goes wrong when it fails and what to check first.
Panel A · The ArchitectureBoth lanes meet at the index
Offline lane · indexingRuns ahead of time, nobody waits
Load→Clean→Chunk→Embed→Store
↓ writeVector index + chunk text + metadata↑ read
Online lane · queryRuns per question, user is waiting
Question→Embed→Retrieve→Prompt→LLM→Answer + sources
Panel B · The Data at Each StageFollow one fact through the system
StageWhat the data looks like
Loadhandbook.txt, a whole file of text
ChunkEmployees get 20 vacation days per year.
Embed and storeA list of numbers, saved with the text and source name
QuestionHow many vacation days do employees get?
RetrieveThe best chunks with similarity scores
PromptInstruction, then context chunks, then the question
Answer20 vacation days per year, source: handbook.txt
Panel C · If This Stage BreaksClick a stage to open it
Click to see the golden debugging ruleExplain ▼
Always debug from the left. If retrieval did not find the right chunk, no amount of prompt tuning will fix the answer. Check the chunks first, then retrieval, then the prompt, and only then blame the model.
How to read it: Panel A is the map, Panel B is one fact walking along that map, and Panel C is the list of places where the walk can go wrong.

7. Python Implementation: A Complete Two-Pipeline RAG System

The program below follows the architecture exactly. build_index is the offline pipeline: it chunks documents by sentence, vectorizes each chunk with word counts, and stores the source name next to it. retrieve, build_prompt and generate form the online pipeline. A real system would send the prompt to an LLM; here generate is a simple stand-in that reads the first context line from the prompt, so the whole program runs with the standard library only.
rag_pipeline.pyPython 3.10+ · Standard library only
import mathfrom collections import Counter documents = [    ("handbook.txt", "Employees get 20 vacation days per year. Unused days can be carried over for 3 months. Remote work is allowed on Tuesdays and Thursdays."),    ("pricing.txt", "The Basic plan costs 10 dollars per month. The Pro plan costs 25 dollars per month. Annual billing gives a 15 percent discount."),    ("support.txt", "Support is available Monday to Friday. Premium customers get a reply within 4 hours. Standard customers get a reply within 24 hours."),] stop_words = set(["the", "is", "a", "of", "how", "many", "much", "does", "do", "per", "to", "can", "for", "on", "get", "gets"]) # Shared tools: used by BOTH pipelines (same vectorizer for docs and queries)def vectorize(text):    cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ")    return Counter(w for w in cleaned.split() if w not in stop_words) def cosine(a, b):    dot = sum(a[w] * b[w] for w in set(a) & set(b))    norm_a = math.sqrt(sum(v * v for v in a.values()))    norm_b = math.sqrt(sum(v * v for v in b.values()))    if norm_a == 0 or norm_b == 0:        return 0.0    return dot / (norm_a * norm_b) # ===== OFFLINE PIPELINE: chunk, embed, store =====def chunk(text):    return [s.strip() + "." for s in text.split(".") if s.strip()] def build_index(documents):    index = []    for source, text in documents:        for piece in chunk(text):            index.append((source, piece, vectorize(piece)))    return index # ===== ONLINE PIPELINE: retrieve, build prompt, generate =====def retrieve(question, index, k=2):    q = vectorize(question)    scored = [(cosine(q, vec), source, piece) for source, piece, vec in index]    scored.sort(reverse=True)    return scored[:k] def build_prompt(question, hits):    lines = ["Answer using ONLY the context.", "Context:"]    for score, source, piece in hits:        lines.append("- " + piece + " (" + source + ")")    lines.append("Question: " + question)    return lines # Stand-in for the LLM call: it just reads the first context linedef generate(prompt):    first = prompt[2]    text, source = first[2:].rsplit(" (", 1)    return text + " [source: " + source.rstrip(")") + "]" # Run the offline pipeline onceindex = build_index(documents)print("Chunks in index:", len(index))print("") # Run the online pipeline for each questionquestions = [    "How many vacation days do employees get?",    "How fast do Premium customers get a reply?",] for question in questions:    hits = retrieve(question, index, k=2)    print("Question:", question)    for score, source, piece in hits:        print("  hit", round(score, 3), source, "|", piece)    prompt = build_prompt(question, hits)    print("  Prompt:")    for line in prompt:        print("    " + line)    print("  Answer:", generate(prompt))    print("")
Expected output
Chunks in index: 9 Question: How many vacation days do employees get?  hit 0.775 handbook.txt | Employees get 20 vacation days per year.  hit 0.218 handbook.txt | Unused days can be carried over for 3 months.  Prompt:    Answer using ONLY the context.    Context:    - Employees get 20 vacation days per year. (handbook.txt)    - Unused days can be carried over for 3 months. (handbook.txt)    Question: How many vacation days do employees get?  Answer: Employees get 20 vacation days per year. [source: handbook.txt] Question: How fast do Premium customers get a reply?  hit 0.612 support.txt | Premium customers get a reply within 4 hours.  hit 0.408 support.txt | Standard customers get a reply within 24 hours.  Prompt:    Answer using ONLY the context.    Context:    - Premium customers get a reply within 4 hours. (support.txt)    - Standard customers get a reply within 24 hours. (support.txt)    Question: How fast do Premium customers get a reply?  Answer: Premium customers get a reply within 4 hours. [source: support.txt]
Notice that build_index ran once and was never called again, while retrieve, build_prompt and generate ran for every question. That is the offline and online split in code form. Also notice that the metadata (the file name) travels with every chunk from the index into the prompt and finally into the answer, which is what makes citations possible.

Knowledge Check

Q1. Which of these steps belongs to the offline indexing pipeline?
Q2. Why must the question and the documents use the same embedding model?
Q3. The right chunk was retrieved, but the final answer is still wrong. Where should you look first?

Key Points

✓
A RAG system is two pipelines, indexing (offline) and query (online), connected by a shared index.
✓
Indexing: load, clean, chunk, embed, store. The index keeps vectors, chunk text and metadata.
✓
Query: receive, embed the question, retrieve top-k, build the prompt, generate, return with sources.
✓
Heavy work goes offline so the online path stays fast for waiting users.
✓
Queries and documents must use the same embedding model.
✓
The prompt plus the answer must fit the context window, so retrieving more is not always better.
✓
Debug from left to right: chunks, retrieval, prompt, and only then the model.

The Big Picture

You now have the map for the whole RAG track. The foundations are done: what RAG is, why it exists, and how the pieces fit. The next lessons walk the offline lane stage by stage, then the online lane, then the improvements and the production concerns that surround both.
Documents → Chunks → Embeddings → Index → Retrieve → Prompt → Answer

Up Next: Data Sources and Document Processing. You will start at the very beginning of the offline lane: how to load PDFs, web pages, Markdown, HTML, CSV, JSON and databases, and how to extract clean text from them.