The Core Idea: Searching chunks of text answers many questions, but not all. Questions about relationships need a graph. Questions about numbers need a database. Questions across a huge corpus need a hierarchy of summaries. Questions that touch several documents or systems need routing and synthesis. Specialized RAG architectures match the structure of the knowledge to the structure of the question.
1. Intermediate: Hierarchical RAG
Hierarchical RAG organizes knowledge in levels, such as document summaries on top, section summaries below, and detailed chunks at the bottom. The system searches top-down: first find the relevant documents using their summaries, then the relevant sections, and finally the exact chunks.
This narrows a huge corpus quickly and helps with questions that need an overview, since higher levels hold condensed information. Approaches such as RAPTOR build a tree of recursive summaries, so the retriever can pick the right level of detail for each question.
Costs: summaries must be generated and refreshed when documents change, and a summary can drop details that a question needs. Keep a path from every summary to the chunks beneath it.
2. Intermediate: Graph RAG and Knowledge Graphs
A knowledge graph stores facts as entities connected by relations, for example Billing Service owned by Payments Team, and Payments Team managed by Dana Lee. Graph RAG retrieves by following those links, so a question that needs several connected facts can be answered by walking the graph instead of hoping that one chunk contains everything.
| Aspect | Vector search | Graph retrieval |
|---|---|---|
| Finds | Passages with similar meaning | Entities and their relations |
| Multi-hop questions | Weak, facts are scattered | Strong, follow the links |
| Setup cost | Low | High, entities and relations must be extracted |
| Explainability | Moderate | High, the path is visible |
Some systems also build a graph automatically from text with a language model, then summarize clusters of related entities so they can answer broad, global questions about the whole corpus. Knowledge graph plus RAG combines both worlds: the graph supplies reliable relations, and text retrieval supplies the detailed passages.
3. Intermediate: Multi-Document and Multi-Source RAG
Multi-document RAG answers questions that span several documents, such as comparing two contracts. The system retrieves from each relevant document, keeps track of which passage came from where, and lets the model synthesize.
Multi-source RAG goes further: the knowledge lives in different kinds of systems such as a document index, a database, a wiki and an API. A router decides which source or sources to query for each question, and the results are merged with their provenance.
Pitfalls: sources disagree, have different freshness, and use different identifiers. Label every result with its source, apply conflict rules, and avoid querying every source for every question.
4. Intermediate: Structured Data and SQL RAG
Numbers, filters and aggregates belong in a database, not in text chunks. In SQL RAG, also called text-to-SQL, a model reads the question and the table schema, writes a query, the database runs it, and the result is turned into an answer.
1. Give the model the schema: table and column names, with short descriptions.
2. The model writes a SQL query for the question.
3. Validate the query, then run it with read-only access.
4. Pass the result rows to the model, which phrases the answer and shows the query as evidence.
Safety: allow only read operations, restrict to approved tables, limit rows and run time, and never run a generated query with write permissions. The correct sum comes from the database, so the model must not recompute it by guessing.
5. Advanced: Matching Architecture to Question
| Question looks like | Best fit | Why |
|---|---|---|
| Summarize or give an overview of a large corpus | Hierarchical RAG | Higher levels hold condensed content |
| Who is connected to whom, or which thing depends on which | Graph RAG | Relations are explicit |
| Compare these two documents | Multi-document RAG | Evidence from each, then synthesis |
| What was total revenue last month | SQL RAG | Exact aggregation |
| A mix of the above | Multi-source RAG with a router | Each part goes to the right system |
These architectures add real engineering cost: graphs must be built and maintained, summaries refreshed, schemas documented. Add one only when a clear class of questions fails with plain retrieval, and evaluate it separately on that class.
6. Visual Explanation: The Right Store for the Right Question
Panel A maps question types to stores. Panel B follows a graph walk and a router from the code below. Panel C is an architecture explorer.
Panel A · Four Kinds of KnowledgeMatch the store to the question
Text
Document index
What does the policy say?
Relations
Knowledge graph
Who manages whom?
Numbers
SQL database
What was the total?
Overview
Summary tree
What is this corpus about?
Panel B · From the CodeA graph walk and a router
Billing Service→Payments Team→Dana Lee→Priya Rao
Three hops answer: who does the manager of the billing team report to
| Question | Routed to |
|---|---|
| What was total revenue in March? | sql database |
| Who manages the team that owns billing? | knowledge graph |
| How do refunds work? | document index |
Panel C · Architecture ExplorerClick an architecture to open it
Click to see why SQL RAG must be read-only▼
A model writes the query, and models can be wrong or be manipulated by hostile input. Read-only access and approved tables ensure that a mistake cannot change or leak data.
How to read it: Panel A is the map of stores, Panel B shows two of them working, and Panel C compares the costs.
7. Python Implementation: A Graph Walk, a SQL Answer and a Source Router
The program shows three specialized retrievers in a few lines each. First, facts are stored as triples and a walk follows relations from a starting entity to answer a multi-hop question. Second, a small SQLite table answers a numeric question with an exact aggregate. Third, a router inspects the question and picks the right source. The language model steps that would write the query or choose the relations are replaced by fixed rules to keep the example self-contained.
specialized_rag.pyPython 3.10+ · Standard library only
import sqlite3 # Graph RAG: facts stored as (subject, relation, object) triplestriples = [ ("Billing Service", "owned_by", "Payments Team"), ("Payments Team", "managed_by", "Dana Lee"), ("Dana Lee", "reports_to", "Priya Rao"), ("Search Service", "owned_by", "Discovery Team"),] def neighbors(entity): return [(r, o) for s, r, o in triples if s == entity] def walk(start, relations): path = [start] current = start for relation in relations: found = [o for r, o in neighbors(current) if r == relation] if not found: return path, None current = found[0] path.append(current) return path, current path, answer = walk("Billing Service", ["owned_by", "managed_by", "reports_to"])print("Graph walk:", " -> ".join(path))print("Who does the manager of the billing team report to?", answer) # SQL RAG: the question becomes a query over a structured tableconn = sqlite3.connect(":memory:")conn.execute("CREATE TABLE sales (region TEXT, month TEXT, revenue INTEGER)")conn.executemany("INSERT INTO sales VALUES (?, ?, ?)", [("North", "March", 120), ("South", "March", 90), ("North", "April", 150)]) def total_revenue(month): row = conn.execute("SELECT SUM(revenue) FROM sales WHERE month = ?", (month,)).fetchone() return row[0] print("")print("SQL answer, total revenue in March:", total_revenue("March")) # Multi-source routing: pick the source that fits the questiondef route(question): q = question.lower() if "revenue" in q or "total" in q: return "sql database" if "report" in q or "owns" in q or "manager" in q: return "knowledge graph" return "document index" print("")for question in ["What was total revenue in March?", "Who manages the team that owns billing?", "How do refunds work?"]: print(question.ljust(42), "->", route(question))Expected output
Graph walk: Billing Service -> Payments Team -> Dana Lee -> Priya RaoWho does the manager of the billing team report to? Priya Rao SQL answer, total revenue in March: 210 What was total revenue in March? -> sql databaseWho manages the team that owns billing? -> knowledge graphHow do refunds work? -> document indexThe graph walk reaches the answer in three hops, and the path itself is the explanation. The SQL query returns 210, the exact sum of the March revenue rows, which no text retriever could compute reliably. The router sends each of the three questions to a different source. In production, a model would translate the questions into relations and SQL, and the validation and read-only rules from the lesson would apply.
Knowledge Check
Q1. Which architecture suits a question that needs several connected facts, such as who manages the team that owns a service?
Q2. Why use a database for the question what was total revenue last month?
Q3. What does a router do in multi-source RAG?
Key Points
✓
Match the retrieval structure to the structure of the question: text, relations, numbers or overview.
✓
Hierarchical RAG searches summaries top-down to narrow a large corpus and support overview questions.
✓
Graph RAG follows entity relations and excels at multi-hop questions, at the price of building the graph.
✓
Multi-document and multi-source RAG retrieve from several places, keep provenance, and use a router.
✓
SQL RAG turns questions into queries; the database computes the numbers, never the model.
✓
Generated SQL must be validated and run read-only with limits.
✓
Add a specialized architecture only for a class of questions where plain retrieval measurably fails.
The Big Picture
Not every question is a search for similar text. Specialized architectures acknowledge that knowledge has shapes, such as networks, tables and hierarchies, and that the best retrieval respects the shape. The skill is in recognizing the question type and keeping the added machinery proportionate.
Classify the question → Route to the right store → Retrieve → Combine with provenance
Up Next: Multimodal and Specialized Data RAG. You will extend RAG to PDFs and tables, images, code, audio transcripts and knowledge bases that mix several formats.