Agent Evaluation and Observability

Learn how to measure and debug agents: task success, tool-use evaluation, trajectory evaluation, structured logging, tracing, and a practical debugging method.

30 minIntermediateCode Examples

The Core Idea: An agent is judged on what it achieved (task success), which tools it used and how it got there (trajectory). Logs and traces record every step so you can debug failures instead of guessing.

  1. Beginner: Agent Evaluation

Evaluating an agent is harder than evaluating a single answer, because an agent takes many steps and can reach the goal in different ways. You need to look at three levels:

LevelQuestionExample metric
OutcomeDid it achieve the goal?Task success rate
Tool useDid it use the right tools correctly?Tool precision
TrajectoryWas the path sensible and efficient?Extra steps vs. a reference path

Remember: An agent can succeed by luck or waste ten steps getting there. Outcome metrics alone hide both problems.

  1. Beginner: Task Success

Task success is the share of tasks the agent completes correctly. To measure it you need a test set of realistic tasks with a clear way to check each result.

Way to checkExample
Exact checkThe right order id, the right number
Programmatic testThe generated code passes its unit tests
State checkThe database now contains the new record
Judge model or humanIs the summary accurate and complete?

In the code: Three of four tasks succeeded, so the success rate is 75%. Always report it on many tasks, and run each task several times because agents are not fully repeatable.

  1. Medium: Tool-Use and Trajectory Evaluation

Tool-use evaluation checks the calls: right tool, right arguments, no needless calls. Trajectory evaluation compares the whole sequence of steps with a reference or with rules.

MetricMeaningWarning sign
Tool precisionShare of calls that used an appropriate toolt3 scored 0.0: it called only the calculator
Exact matchSame steps as the referenceToo strict, since several paths can be valid
Extra stepsHow many more steps than neededt2 repeated a search
Repeated callsSame call again and againA loop in the making
Reference paths
Reference trajectories are useful but not sacred. Several different paths can be correct, so combine them with outcome checks.

  1. Medium: Logging and Tracing

Logging records events. Tracing links the events of one run into a timeline, so you can follow a task from the user request to the final answer, including every model call and tool call.

RecordWhy
Run id on every lineLinks all steps of one task
Step, tool and argumentsShows what the agent did
Latency in msFinds slow steps
Status and errorFinds failures
Tokens and costFinds expensive steps
Model input and output (carefully)Explains decisions

Privacy: Logs can contain personal data and secrets. Redact them and set retention limits.

  1. Advanced: Debugging Agents

When an agent fails, use a repeatable method instead of guessing:

1
Find the failed run: Search by run id or by a failed status.
2
Read the trace from the top: Find the first step where reality differed from what you expected.
3
Classify the cause: Bad tool, bad prompt, bad context, bad data, or a system failure.
4
Reproduce it: Replay the same inputs, ideally from a recorded trace.
5
Fix and add a test: Add the case to the evaluation set so it never comes back.

In the code: The first failing step of the failed run is step 3, and it is also the slowest at 2100 ms. The trace points straight at the problem.

  1. Visual Explanation: Measuring Agents and Debugging Them

The figure shows the three evaluation levels with the numbers from the code, then a simple debugging loop.

Part 1 · Three Levels of EvaluationOutcome, tools, path
OutcomeSuccess rate
3 of 4 tasks succeeded
↓
75%
Tool usePrecision
t3 used only the wrong tool
↓
t3 scored 0.0
TrajectoryExtra steps
t2 and t3 took one step more than needed
↓
Waste found
Part 2 · From Trace to FixDebugging loop
Find
Failed run
By run id.
Read
The trace
Find the first surprise.
Cause
Classify
Tool, prompt, context, data.
Fix
Add a test
Keep it fixed.
Click to see why one run is not enoughCompare ▼
Agents vary between runs. A task that passed once may fail the next time, so run each test several times and report the success rate with its uncertainty.

Part 1 gives the result for outcome, tool use and trajectory. Part 2 shows the debugging loop. Open the bar to see why repeated runs matter.

  1. Python Implementation: Success, Tool Precision, Trajectories and Traces

This script evaluates four recorded agent runs. It computes the success rate, the tool precision, exact trajectory matches and extra steps, prints a structured trace as JSON lines, and finds the first failing and the slowest step of a failed run.

agent_evaluation.pyPython 3.10+ · standard library only
import json runs = [    {"task": "t1", "success": True,  "tools": ["search", "read", "answer"], "reference": ["search", "read", "answer"]},    {"task": "t2", "success": True,  "tools": ["search", "search", "read", "answer"], "reference": ["search", "read", "answer"]},    {"task": "t3", "success": False, "tools": ["calculator", "calculator", "calculator"], "reference": ["search", "answer"]},    {"task": "t4", "success": True,  "tools": ["search", "answer"], "reference": ["search", "answer"]},] # 1. Task success: did the agent reach the goal?success_rate = sum(r["success"] for r in runs) / len(runs)print("Task success rate:", round(success_rate * 100), "%") # 2. Tool-use evaluation: were the tools it called the right ones?def tool_precision(run):    right = sum(1 for tool in run["tools"] if tool in run["reference"])    return right / len(run["tools"]) print("Tool precision per task:", {r["task"]: round(tool_precision(r), 2) for r in runs}) # 3. Trajectory evaluation: how close was the path to the ideal one?for r in runs:    exact = r["tools"] == r["reference"]    extra = len(r["tools"]) - len(r["reference"])    print(r["task"], "| exact match:", exact, "| extra steps:", extra) # 4. Logging and tracing: one structured line per step, tied together by a run idtrace = [    {"run": "t3", "step": 1, "tool": "calculator", "ms": 40, "status": "ok"},    {"run": "t3", "step": 2, "tool": "calculator", "ms": 35, "status": "ok"},    {"run": "t3", "step": 3, "tool": "calculator", "ms": 2100, "status": "timeout"},]for line in trace:    print(json.dumps(line)) # 5. Debugging: find the first problem and the slowest step in a failed runfirst_bad = next(s for s in trace if s["status"] != "ok")slowest = max(trace, key=lambda s: s["ms"])print("First failing step:", first_bad["step"], first_bad["tool"], "| slowest step:", slowest["step"], str(slowest["ms"]) + " ms")
EXPECTED OUTPUT (your numbers may differ slightly)
Task success rate: 75 %
Tool precision per task: {'t1': 1.0, 't2': 1.0, 't3': 0.0, 't4': 1.0}
t1 | exact match: True | extra steps: 0
t2 | exact match: False | extra steps: 1
t3 | exact match: False | extra steps: 1
t4 | exact match: True | extra steps: 0
{"run": "t3", "step": 1, "tool": "calculator", "ms": 40, "status": "ok"}
{"run": "t3", "step": 2, "tool": "calculator", "ms": 35, "status": "ok"}
{"run": "t3", "step": 3, "tool": "calculator", "ms": 2100, "status": "timeout"}
First failing step: 3 calculator | slowest step: 3 2100 ms

The headline number (75%) hides the story: t3 failed because it used the wrong tool three times, and t2 succeeded but wasted a step. The structured trace of the failed run shows that its last step timed out and was also the slowest.

Knowledge Check

Question 1: Why is task success alone not enough to judge an agent?
Question 2: What is the purpose of a run id in logs?

Key Points

✓Evaluate agents at three levels: outcome, tool use and trajectory.
✓Task success needs a realistic test set and a reliable check for each task.
✓Tool precision and extra steps reveal wrong or wasteful behavior that success rates hide.
✓Logs record events, and traces link them with a run id into one timeline.
✓Record latency, status, errors, tokens and cost, and redact sensitive data.
✓Debug by finding the first surprise in the trace, classifying the cause and adding a test.

The Big Picture

Evaluation in One Line

Test set + success rate + tool and trajectory metrics + traces = agents you can improve with evidence

What you cannot measure you cannot improve, and what you cannot trace you cannot debug.

Up Next: One agent can only do so much. In Introduction to Multi-Agent Systems you will learn why and when several specialized agents work better than one.