Agents with RAG

Combine agents with retrieval: use retrieval as an agent tool, let the agent control when and how to retrieve, build knowledge-based agents, and understand the difference between classic RAG and agentic RAG.

30 minIntermediateCode Examples

The Core Idea: In classic RAG the system always retrieves once and then answers. In agentic RAG the agent decides whether to retrieve, rewrites the query, checks the results and tries again, so it handles harder questions.

  1. Beginner: Agent + Retrieval

RAG (Retrieval-Augmented Generation) gives a model relevant documents to read before answering, so it can use private or recent knowledge. An agent adds decision-making on top. Combining them gives an agent that can look things up when it needs to.

RetrievalFind
Search a document collection for the most relevant pieces.
GenerationAnswer
The model writes the answer using the retrieved text.
AgentDecide
Chooses whether to retrieve, what to search for and whether the result is good enough.

Simplest view: Retrieval becomes one more tool the agent can call, such as search_knowledge_base(query).

  1. Beginner: Retrieval as an Agent Tool

To the agent, retrieval is just a tool with a name, a description and parameters. It returns a few passages with their sources, and the agent reads them like any other tool result.

Tool partExample
Namesearch_knowledge_base
DescriptionSearch the company help documents. Use for policy and product questions.
Parameterquery (a short search phrase)
ReturnsThe top passages and their document names and scores

Tip: Return scores and sources. Scores let the agent judge quality, and sources let it cite.

  1. Medium: Agent-Controlled Retrieval

Instead of retrieving blindly, the agent controls the process. It can:

1
Decide: Skip retrieval for greetings or math, retrieve for factual questions.
2
Rewrite: Turn how long until my package arrives? into a better query such as shipping time.
3
Check: Look at the retrieval score. If it is low, the results are probably irrelevant.
4
Retry: Reformulate and search again, up to a limit.
5
Combine: For multi-part questions, search several times and merge the findings (multi-hop).

In the code: Plain RAG always picks the top result even with a score of 0.0. The agent retries with a rewritten query and succeeds, or honestly says it could not find the answer.

  1. Medium: Knowledge-Based Agents

A knowledge-based agent answers from a defined body of knowledge, such as product manuals or company policies. Good ones follow four habits:

HabitWhy it matters
Ground answers in retrieved textReduces invented facts
Cite the source documentLets users verify
Say I could not find that when nothing relevant is foundHonesty beats a confident guess
Keep knowledge freshRe-index when documents change
Out of scope questions
The agent should be able to recognize questions its knowledge base cannot answer, instead of forcing a match from irrelevant text.

  1. Advanced: RAG vs. Agentic RAG

Classic RAGAgentic RAG
RetrievalAlways once, with the user's questionWhen and as often as the agent decides
QueryThe original questionRewritten and refined
Quality checkNoneLooks at scores and relevance
Multi-part questionsWeakSearches several times
Cost and latencyLow, predictableHigher, varies
Best forSimple lookup questionsComplex, ambiguous or multi-step questions

Choose wisely: Start with classic RAG. Move to agentic RAG only when you can show that simple retrieval fails on real questions, because it costs more and is harder to predict.

  1. Visual Explanation: Classic RAG and Agentic RAG Side by Side

The figure shows one question processed by classic RAG and by agentic RAG, then lists the five controls an agent has over retrieval.

Part 1 · Classic RAG vs. Agentic RAGSame question, different process
Classic RAGAlways once
Question: How long until my package arrives?
↓
Search the question as written
↓
Top result is the warranty text (score 0.0)
↓
Wrong document used
Agentic RAGDecide, rewrite, check
Search: score is too low
↓
Rewrite the query to add shipping
↓
Second search finds the shipping text
↓
Correct answer found
Part 2 · The Agent's Retrieval ControlsFive decisions
Decide
Retrieve?
Skip it for greetings.
Rewrite
Better query
Add the right words.
Check
Score
Is the result relevant?
Retry
Again
Up to a limit.
Admit
Not found
Say so honestly.
Click to see why scores matterCompare ▼
A retrieval system always returns something, even when nothing relevant exists. The score tells the agent whether to trust the result, retry, or admit it cannot answer.

Part 1 shows classic RAG using the wrong document while agentic RAG recovers. Part 2 lists the controls. Open the bar to see why scores matter.

  1. Python Implementation: Plain RAG vs. Agentic RAG

This script builds a tiny knowledge base and runs four questions through plain RAG (always use the top result) and agentic RAG (decide, rewrite, check the score, retry, or admit failure). Word overlap stands in for embeddings.

agentic_rag.pyPython 3.10+ · standard library only
DOCS = {    "refund": "Refunds are accepted within 30 days of purchase with a receipt.",    "shipping": "Standard shipping takes 3 to 5 business days. Express takes 1 to 2 days.",    "warranty": "All devices include a 12 month warranty covering manufacturing defects.",}STOP = {"the", "a", "an", "is", "of", "to", "my", "i", "can", "how", "what", "until", "does", "do", "for", "in"} def words(text):    tokens = [w.strip(".,?!'").lower() for w in text.split()]    tokens = [w[:-1] if len(w) > 3 and w.endswith("s") else w for w in tokens]    return {w for w in tokens if w not in STOP} def retrieve(query):    q = words(query)    scored = [(len(q & words(text)) / max(len(q), 1), doc_id) for doc_id, text in DOCS.items()]    return max(scored) def plain_rag(question):    score, doc_id = retrieve(question)           # always uses the top result, good or bad    return doc_id, round(score, 2) SYNONYMS = {"arrive": "shipping", "send back": "refund", "guarantee": "warranty", "broken": "warranty"} def rewrite(question):    extra = [target for phrase, target in SYNONYMS.items() if phrase in question.lower()]    return question + " " + " ".join(extra) def needs_retrieval(question):    return not question.lower().startswith(("hello", "hi", "thanks")) def agentic_rag(question, threshold=0.2, max_tries=2):    if not needs_retrieval(question):        return "no retrieval needed, answer directly"    query = question    for attempt in range(1, max_tries + 1):        score, doc_id = retrieve(query)        if score >= threshold:            return "found '" + doc_id + "' (score " + str(round(score, 2)) + ") on attempt " + str(attempt)        query = rewrite(query)                   # the agent reformulates and tries again    return "could not find it, will say so honestly" for q in ["Hello!", "How long until my package arrives?", "Can I send back a gadget?", "What is the CEO's name?"]:    print(q.ljust(38), "| plain RAG:", plain_rag(q), "| agentic RAG:", agentic_rag(q))
EXPECTED OUTPUT (your numbers may differ slightly)
Hello! | plain RAG: ('warranty', 0.0) | agentic RAG: no retrieval needed, answer directly
How long until my package arrives? | plain RAG: ('warranty', 0.0) | agentic RAG: found 'shipping' (score 0.25) on attempt 2
Can I send back a gadget? | plain RAG: ('warranty', 0.0) | agentic RAG: found 'refund' (score 0.25) on attempt 2
What is the CEO's name? | plain RAG: ('warranty', 0.0) | agentic RAG: could not find it, will say so honestly

Plain RAG returns the warranty document for every question, even with a score of 0.0. The agent skips retrieval for the greeting, finds the right document on the second attempt for shipping and refunds, and correctly admits it has no answer about the CEO.

Knowledge Check

Question 1: What is the main difference between classic RAG and agentic RAG?
Question 2: Why should a knowledge-based agent be able to say it could not find an answer?

Key Points

✓RAG gives a model relevant documents to read before answering.
✓Retrieval becomes a tool the agent can call, returning passages, sources and scores.
✓Agent-controlled retrieval means deciding whether to retrieve, rewriting queries, checking scores and retrying.
✓Knowledge-based agents ground answers in text, cite sources and admit when they do not know.
✓Classic RAG retrieves once and is cheap. Agentic RAG adapts but costs more and is less predictable.
✓Start simple and add agentic behavior only where simple retrieval fails.

The Big Picture

Agentic RAG in One Line

Decide → search → check the score → rewrite and retry → answer with sources or say it is unknown

Retrieval is one of the most valuable tools an agent can have, and controlling it well is a skill of its own.

Up Next: Your agent can now act, remember and look things up. In Agent Reasoning and Decision Making you will study how it chooses what to do next.