Agent Performance and Cost Optimization

Learn how to make agents faster and cheaper: understanding latency and token usage, choosing models per step, caching, parallel execution, tool optimization and cost controls, with a worked cost model.

28 minAdvancedCode Examples

The Core Idea: Agent costs grow quickly because every step re-sends the whole history. The main levers are fewer tokens, cheaper models for easy steps, caching, parallel tool calls and hard budgets. Measure first, then optimize the biggest cost.

  1. Beginner: Where Time and Money Go

An agent run is many model calls plus tool calls. You pay for tokens (input and output) and wait for latency (model time plus tool time). Both grow with the number of steps.

Cost driverWhy it grows
Input tokensEvery step re-sends instructions, tool descriptions and the growing history
Output tokensLong reasoning and long answers (output tokens usually cost more than input)
Number of stepsMore loops mean more calls
Model sizeLarger models cost more per token and are slower
Tool timeSlow APIs add waiting time

Rule: Measure before you optimize. Log tokens, cost, latency and step count for every run, and fix the biggest item first.

  1. Beginner: Latency and Token Usage

To cut latency, reduce the number of sequential model calls and the time spent waiting on tools. To cut tokens, send less to the model:

TechniqueEffect
Shorter, sharper instructionsFewer tokens on every call
Fewer, better toolsSmaller tool descriptions
Compact tool resultsReturn only needed fields
Trim or summarize historyStops quadratic growth
Concise output formatsFewer output tokens

Why history matters: In a 12-step run, step 12 sends the system prompt plus 11 steps of history. The total input grows roughly with the square of the number of steps, which is why long runs get expensive fast.

  1. Medium: Model Selection

Not every step needs the strongest model. A common design uses different models for different jobs:

Step typeModel choice
Planning and hard reasoningLarger, stronger model
Routine tool calls and formattingSmaller, cheaper, faster model
Classification and routingVery small model
Final answer qualityStronger model if quality matters

Be careful: Test a cheaper model with your evaluation set before switching. A saving that lowers success rate is not a saving.

In the code: Using a model that costs a tenth for half of the steps cuts the run cost to about 67% of the baseline.

  1. Medium: Caching and Parallel Execution

Caching avoids paying twice for the same work. Prompt caching discounts a repeated prefix, such as instructions and earlier history. Result caching stores tool results for identical calls.

Parallel execution runs independent tool calls at the same time. In the code, four 1.5-second calls take 6.0 seconds one after another but only 1.5 seconds in parallel.

TechniqueSavingCondition
Prompt cachingLarge (34% of baseline cost in the code, with an example 90% discount)Stable prefix at the start of the prompt
Result cachingTime and tool costSame arguments, data not stale
Parallel tool callsLatencyIndependent calls
Prompt layout
Put the stable parts first (instructions, tools, older history) and the changing parts last, so the cache can match the beginning.

  1. Advanced: Tool Optimization and Cost Control

Tools are a hidden cost. A search tool that returns a whole web page can flood the context, and each extra call adds a model round trip.

Tool optimizationWhy
Return compact, structured resultsSaves input tokens on every later step
Merge tiny tools into oneFewer round trips
Batch calls (look up ten IDs at once)Fewer steps
Truncate and say soControls size without confusing the model
Cache slow, stable lookupsSaves time and money

Finally, add cost controls so a bad run cannot become a bad bill:

ControlExample
Per-run budgetStop after a maximum number of tokens or dollars
Step limitHard cap on loops
Per-user quotasLimit usage per account
AlertsNotify on unusual spend
Cost in your logsKnow the cost per task type

Do not forget quality: Always measure success rate next to cost. Cheaper but wrong is not an optimization.

  1. Visual Explanation: What Each Optimization Saves

The figure compares the cost of one run under each optimization using the example prices in the code, then lists the biggest levers.

Part 1 · Cost of One 12-Step RunExample prices, from the code
Baseline100%
Every step re-sends the history
↓
USD 0.157 per run
Prompt caching34%
Repeated prefix at a discount
↓
USD 0.053 per run
Summarize history66%
History capped at 800 tokens
↓
USD 0.103 per run
Cheap model, half the steps67%
A tenth of the price for easy steps
↓
USD 0.106 per run
All three together17%
Combined savings
↓
USD 0.026 per run
Part 2 · Where to SaveBiggest levers first
Tokens
Send less
Trim and summarize history.
Models
Right size
Cheap model for easy steps.
Caching
Pay once
Prefix and results.
Parallel
Wait less
4 calls: 6 s becomes 1.5 s.
Click to see the cost for 10,000 runsCompare ▼
With the example prices, 10,000 baseline runs would cost about USD 1,566, while all three optimizations together cost about USD 263. Provider caching rules and prices differ, so recompute with your own numbers.

Part 1 shows each variant as a share of the baseline cost. Part 2 groups the savings by lever. Open the bar for the cost of 10,000 runs.

  1. Python Implementation: A Cost Model for a 12-Step Agent Run

This script models the input and output cost of a 12-step run where every step re-sends the history. It compares the baseline with prompt caching, history summarization, a cheaper model for half of the steps, and all three together, and shows the latency gain of parallel tool calls. The prices are made-up examples.

agent_cost_model.pyPython 3.10+ · standard library only
# Cost of one agent run: every step re-sends the growing historyIN_PRICE, OUT_PRICE = 3.0, 15.0            # example prices in USD per million tokens (not real quotes) def run_cost(steps=12, system=1000, per_step=400, output=150, cached=False, summarize_cap=None, cheap_share=0.0):    total = 0.0    for i in range(1, steps + 1):        history = (i - 1) * per_step        if summarize_cap:            history = min(history, summarize_cap)             # old steps are compressed into a short summary        prefix = system + history        if cached:            input_cost = prefix * IN_PRICE * 0.1 + per_step * IN_PRICE      # repeated prefix at an example 90% discount        else:            input_cost = (prefix + per_step) * IN_PRICE        step_cost = (input_cost + output * OUT_PRICE) / 1_000_000        if i / steps <= cheap_share:            step_cost *= 0.1                                  # easy steps use a model that costs a tenth        total += step_cost    return total baseline = run_cost()variants = [    ("baseline", baseline),    ("+ prompt caching", run_cost(cached=True)),    ("+ summarize history (cap 800)", run_cost(summarize_cap=800)),    ("+ cheap model for 50% of steps", run_cost(cheap_share=0.5)),    ("all three together", run_cost(cached=True, summarize_cap=800, cheap_share=0.5)),]for name, cost in variants:    print(name.ljust(32), "USD", round(cost, 4), "per run |", round(cost / baseline * 100), "% of baseline | USD", round(cost * 10000, 2), "per 10,000 runs") # Latency: independent tool calls can run in paralleltool_seconds = [1.5, 1.5, 1.5, 1.5]print("4 tool calls one after another:", sum(tool_seconds), "s | in parallel:", max(tool_seconds), "s")
EXPECTED OUTPUT (your numbers may differ slightly)
baseline USD 0.1566 per run | 100 % of baseline | USD 1566.0 per 10,000 runs
+ prompt caching USD 0.0529 per run | 34 % of baseline | USD 529.2 per 10,000 runs
+ summarize history (cap 800) USD 0.1026 per run | 66 % of baseline | USD 1026.0 per 10,000 runs
+ cheap model for 50% of steps USD 0.1056 per run | 67 % of baseline | USD 1055.7 per 10,000 runs
all three together USD 0.0263 per run | 17 % of baseline | USD 262.98 per 10,000 runs
4 tool calls one after another: 6.0 s | in parallel: 1.5 s

Caching gives the largest single saving in this model, and combining all three cuts the cost to about one sixth. The model is simplified, because real providers have their own caching rules, so use it to build intuition and then measure your own runs.

Knowledge Check

Question 1: Why do long agent runs get expensive so quickly?
Question 2: What should you check before switching a step to a cheaper model?

Key Points

✓Cost and latency come from tokens, steps, model size and tool time. Measure them first.
✓History is re-sent every step, so trim, summarize and cap it.
✓Use stronger models only where needed and test cheaper ones with your evaluation set.
✓Cache stable prefixes and tool results, and run independent tool calls in parallel.
✓Make tools return compact results and merge or batch calls.
✓Add per-run budgets, step limits, quotas and alerts, and always track quality with cost.

The Big Picture

Optimization in One Line

Measure → send fewer tokens → route to cheaper models → cache → parallelize → cap the budget

Cheaper agents are not just a nicety. They decide whether a product is affordable to run.

Up Next: Now push agents further. In Autonomous and Long-Horizon Agents you will see how agents run for hours, remember across steps and recover from failure.