Advanced Agentic Systems

Explore advanced agent types: agentic RAG, tool-rich agents that retrieve the tools they need, computer-use and browser agents, multimodal agents and code agents, with working examples of tool retrieval and a browser-style action loop.

30 minAdvancedCode Examples

The Core Idea: Advanced agents extend the same loop with richer inputs and bigger toolboxes: they retrieve knowledge and tools on demand, see and operate screens, work with images, audio and documents, and write and run code. The loop is the same. The observations and actions get richer.

  1. Beginner: Agentic RAG, Revisited

You met agentic RAG earlier: the agent decides when to retrieve, rewrites queries, checks scores and retries. Advanced systems add more control:

TechniqueIdea
Multi-hop retrievalUse the first answer to form the next search, for questions that need several facts
Source routingChoose between several knowledge bases or tools, such as documents, a database or the web
Relevance and support checksVerify that retrieved text really supports the answer before writing it
CitationsAttach the source to every claim
Fallback to the userAsk a clarifying question when retrieval keeps failing

Remember: Each extra control improves hard questions but adds latency and cost. Add them only where simple retrieval fails.

  1. Beginner: Tool-Rich Agents

A real company may have hundreds of tools. Putting every tool description in the prompt wastes tokens and confuses the model. A tool-rich agent first retrieves the few tools relevant to the request, then shows only those.

In the code: 30 tools would need 330 tokens in every prompt. For the request show me the list of orders, the agent retrieves just three order tools and uses 33 tokens, a tenth of the size.

Way to find toolsHow
Keyword or embedding searchMatch the request to tool descriptions
Categories (routing)Pick a group such as billing, then show its tools
Learned from usagePrefer tools that worked in similar tasks

  1. Medium: Computer-Use and Browser Agents

A browser agent works with web pages, and a computer-use agent works with a whole desktop. Instead of calling an API, it looks at the screen and performs actions: click, type, scroll, press a key.

1
Observe: Get the page as a screenshot, or as a structured list of elements (an accessibility tree) with ids.
2
Decide: The model picks an action, such as type in element 1 or click element 3.
3
Act: A controller performs the action in a real or virtual browser.
4
Observe again: The new page shows whether it worked.

In the code: The page is a list of elements with ids. The agent types the email, fills the password from a secret store and clicks Sign in. The page then changes to a dashboard.

Risks: These agents can click the wrong thing and can be misled by hostile content on a page. Run them in a sandboxed browser, use limited accounts, never expose real passwords to the model and ask for approval before purchases or messages.

  1. Medium: Multimodal Agents

A multimodal agent understands more than text: images, screenshots, charts, PDFs, audio and sometimes video. This opens new tasks:

InputExample task
ScreenshotOperate an app or find a button
Chart or photoDescribe a trend or check a damaged product
PDF or scanned formExtract fields and fill a system
AudioTranscribe a call and create follow-up tasks
Practical notes
Images use many tokens, so resize and crop where you can. Always verify extracted numbers, because reading text from images can fail in small ways.

  1. Advanced: Code Agents and Choosing the Design

A code agent writes code, runs it and reads the results. This ReAct-and-reflection pattern is why coding assistants work well: the test runner provides exact, objective feedback.

Code agent ingredientWhy
Repository search and file toolsUnderstand a large codebase
A sandbox to run code and testsSafe execution
Test results as feedbackSelf-correction with real errors
Version controlReview, undo and compare changes
Human review of the resultSafety for important changes

Choosing: Pick the narrowest agent type that solves the problem. A plain API tool is faster, cheaper and safer than a browser agent, which is itself safer than controlling a whole computer.

  1. Visual Explanation: Four Advanced Agents Share One Loop

The figure shows what each advanced agent observes and does, then orders the approaches from the narrowest and safest to the broadest.

Part 1 · Four Advanced Agents, One LoopObserve, decide, act
Agentic RAGKnowledge
Observe: retrieved passages and scores
↓
Act: rewrite and search again
↓
Grounded answer
Tool-richMany tools
Observe: the request
↓
Act: retrieve the 3 best tools
↓
330 tokens become 33
BrowserScreens
Observe: element list or screenshot
↓
Act: type, click
↓
Login becomes dashboard
CodePrograms
Observe: test output
↓
Act: edit and rerun
↓
Objective feedback
Part 2 · Choose the Narrowest ToolFrom safest to broadest
API tool
Most reliable
Structured, fast, cheap.
Browser agent
Flexible
Works where no API exists.
Computer-use
Broadest
Any app, most risk.
Human check
Always
For purchases and messages.
Click to see why screens are a last resortCompare ▼
Screen control works with almost any software, but it is slower, costs more tokens and is easier to fool. Use a proper API whenever one exists.

Part 1 compares the observations and actions of four agent types. Part 2 is a ladder from API tools up to computer use. Open the bar to see why screens are a last resort.

  1. Python Implementation: Tool Retrieval and a Browser-Style Action Loop

This script shows two ideas. First, a tool-rich agent retrieves the three most relevant tools from a catalog of 30 and compares the prompt sizes. Second, a browser-style agent observes a page as a list of elements, acts by typing and clicking, and observes the changed page.

advanced_agents.pyPython 3.10+ · standard library only
# 1. A tool-rich agent: retrieve only the few relevant tools instead of showing all of themverbs = ["get", "create", "delete", "update", "list"]nouns = ["invoice", "ticket", "customer", "order", "report", "calendar"]TOOLS = [(v + "_" + n, v + " a " + n + " record in the system") for v in verbs for n in nouns] def words(text):    return {w.rstrip("s") for w in text.lower().replace("_", " ").split()} def select_tools(query, k=3):    q = words(query)    scored = sorted(TOOLS, key=lambda tool: len(q & words(tool[0] + " " + tool[1])), reverse=True)    return [name for name, _ in scored[:k]] def prompt_tokens(tools):    return sum(round(len((n + " " + d).split()) / 0.75) for n, d in tools) query = "show me the list of orders"print("Tools available:", len(TOOLS), "| tokens if all are shown:", prompt_tokens(TOOLS))chosen = select_tools(query)print("Query:", query, "-> shown to the model:", chosen, "| tokens:", prompt_tokens([t for t in TOOLS if t[0] in chosen])) # 2. A browser agent: the page is turned into a list of elements the agent can act onpage = {"title": "Login", "elements": [    {"id": 1, "role": "input", "label": "Email", "value": ""},    {"id": 2, "role": "input", "label": "Password", "value": ""},    {"id": 3, "role": "button", "label": "Sign in"}]} def observe(page):    return page["title"] + ": " + ", ".join("[" + str(e["id"]) + "] " + e["role"] + " " + e["label"] for e in page["elements"]) def act(page, action):    if action["type"] == "type":        next(e for e in page["elements"] if e["id"] == action["id"])["value"] = action["text"]    elif action["type"] == "click" and all(e.get("value", "x") for e in page["elements"]):        return {"title": "Dashboard", "elements": [{"id": 4, "role": "text", "label": "Welcome, Ana"}]}    return page plan = [{"type": "type", "id": 1, "text": "ana@example.com"}, {"type": "type", "id": 2, "text": "(from the secret store)"}, {"type": "click", "id": 3}]print("Observe:", observe(page))for action in plan:    page = act(page, action)    print("Act:", action, "->", observe(page))
EXPECTED OUTPUT (your numbers may differ slightly)
Tools available: 30 | tokens if all are shown: 330
Query: show me the list of orders -> shown to the model: ['list_order', 'get_order', 'create_order'] | tokens: 33
Observe: Login: [1] input Email, [2] input Password, [3] button Sign in
Act: {'type': 'type', 'id': 1, 'text': 'ana@example.com'} -> Login: [1] input Email, [2] input Password, [3] button Sign in
Act: {'type': 'type', 'id': 2, 'text': '(from the secret store)'} -> Login: [1] input Email, [2] input Password, [3] button Sign in
Act: {'type': 'click', 'id': 3} -> Dashboard: [4] text Welcome, Ana

The retrieval step cuts the tool prompt to one tenth. In the browser loop the observation is a compact text list of elements rather than a picture, which is cheaper and easier for the model, and the final observation shows the dashboard. Real browser agents use a browser automation library for the act step.

Knowledge Check

Question 1: Why do tool-rich agents retrieve tools instead of listing all of them?
Question 2: When should you prefer an API tool over a browser agent?

Key Points

✓Advanced agents use the same loop with richer observations and actions.
✓Agentic RAG can add multi-hop search, source routing, support checks and citations.
✓Tool-rich agents retrieve only the few relevant tools instead of showing hundreds.
✓Browser and computer-use agents observe screens and act by clicking and typing. Sandbox them and limit their accounts.
✓Multimodal agents read images, PDFs and audio, and their extractions need verification.
✓Code agents rely on test results for objective feedback. Always choose the narrowest agent type that works.

The Big Picture

Advanced Agents in One Line

Same loop, richer senses and hands, always the narrowest tool that works

Each new capability brings new risks, so add power only where it earns its place.

Up Next: The last lesson looks at the research frontier. In Advanced and Research-Level Agents you will see self-improving agents, trajectory optimization, agent fine-tuning, simulation and swarms.