Production Agent Architecture

Learn how to run agents for real users: the agent backend and API layer, databases and state persistence, job queues, retries and idempotency, scaling with capacity planning, and deployment practices.

30 minAdvancedCode Examples

The Core Idea: A production agent is not one script. It is a backend: an API accepts requests, a queue holds jobs, workers run the agent loop, a database stores state and results, and monitoring watches everything. The design goal is that no single failure loses work.

  1. Beginner: The Agent Backend

In a demo, you run an agent in a notebook and wait. In production, many users send requests at once, runs can take minutes, and servers restart. So the agent runs as a service made of several parts:

PartJob
API layerReceives requests, checks identity, returns a job id or a result
QueueHolds work until a worker is free
WorkersRun the agent loop: call the model, run tools, update state
DatabaseStores jobs, state, results and history
CacheSpeeds up repeated work and reduces cost
MonitoringLogs, traces, metrics and alerts

Key idea: Separate accepting work from doing work. The API answers fast, and workers take their time.

  1. Beginner: APIs for Agents

Agent runs are slow compared with normal web requests, so the API usually works asynchronously:

1
Submit: The client sends the task. The API stores a job and replies immediately with a job id.
2
Wait: The client polls for status, or receives live updates through streaming or webhooks.
3
Fetch: When the status is done, the client reads the result.
4
Cancel (optional): The client can stop a run that is no longer needed.
What the API must also do
Authenticate users, enforce rate limits and quotas, validate input, never expose secrets, and return clear errors.

  1. Medium: Databases and State Persistence

Everything the agent needs to resume must be stored outside the process: the goal, the step history, tool results and the current status. If a worker dies, another worker can load the state and continue.

StoreTypical content
Relational databaseJobs, users, status, results
Key-value storeFast session state and locks
Object storageLarge files and artifacts
Vector databaseSemantic memory and documents

In the code: Jobs live in a database table with a status column: queued, running, done or failed. A restart does not lose them.

  1. Medium: Queues, Retries and Idempotency

A queue decouples the API from the workers and smooths traffic spikes. Because work can be retried, every job must be safe to run more than once.

IdeaPurposeIn the code
Status transitionsTrack where each job isqueued, running, done, failed
Retries with a limitRecover from temporary failuresreq-2 failed once, then succeeded on attempt 2
Give-up stateStop hopeless jobsreq-3 failed after 3 attempts
Idempotency keyIgnore duplicate submissionsThe second req-1 was ignored

Why idempotency matters: Networks cause duplicate requests, and workers crash mid-job. Without idempotency a customer could be refunded twice or emailed twice.

  1. Advanced: Scaling and Deployment

Capacity planning starts with Little's law: the number of runs in progress equals the arrival rate times the time each run takes.

concurrent runs=requests per second×seconds per run\text{concurrent runs} = \text{requests per second} \times \text{seconds per run}

In the code: 5 requests per second at 20 seconds per run means 100 runs at once. With 4 runs per worker, you need 25 workers.

PracticeWhy
Stateless workersAny worker can run any job, so you can add or remove workers freely
Autoscale on queue lengthMatch capacity to demand
Respect model provider rate limitsThe model API can be the real bottleneck
Containers and a deployment pipelineRepeatable releases and fast rollbacks
Secrets from a secret managerNever in code or prompts
Graceful shutdownFinish or re-queue running jobs on a restart
Staged rollouts and monitoringCatch bad releases early

  1. Visual Explanation: The Request Path of a Production Agent

The figure shows the path of a request through the API, queue, worker and database, then summarizes what the code demonstrated about jobs and capacity.

Part 1 · The Request PathFrom user to result
APIAccept
Check the user
↓
Store the job
↓
Return a job id
QueueHold
Jobs wait their turn
↓
Smooths spikes
WorkerRun
Load state
↓
Run the agent loop
↓
Save progress
DatabaseRemember
Status and results
↓
Client fetches the result
Part 2 · What the Code ShowedFour jobs, four lessons
req-1
Done
Ran once. A duplicate was ignored.
req-2
Retried
Failed once, succeeded on attempt 2.
req-3
Gave up
Failed after 3 attempts.
Capacity
25 workers
5 req/s times 20 s over 4 per worker.
Click to see why workers should be statelessCompare ▼
If all progress lives in the database, any worker can pick up any job, and a crashed worker is replaced without losing work. State kept only in a worker's memory is lost when it dies.

Part 1 reads left to right. Part 2 explains the four jobs and the capacity result. Open the bar to see why stateless workers matter.

  1. Python Implementation: A Durable Job Queue and Capacity Planning

This script builds a small durable job queue in a database with status transitions, retries with a limit, a give-up state and an idempotency key that ignores duplicate submissions. It ends with a capacity calculation based on Little's law.

agent_backend.pyPython 3.10+ · standard library only
import json, sqlite3 # A durable job queue in a database: the heart of a production agent backenddb = sqlite3.connect(":memory:")db.execute("""CREATE TABLE jobs (id INTEGER PRIMARY KEY, idem_key TEXT UNIQUE, payload TEXT,              status TEXT, attempts INTEGER DEFAULT 0, result TEXT)""") def submit(idem_key, payload):    try:        cursor = db.execute("INSERT INTO jobs (idem_key, payload, status) VALUES (?, ?, 'queued')", (idem_key, json.dumps(payload)))        return "queued as job " + str(cursor.lastrowid)    except sqlite3.IntegrityError:        return "duplicate request ignored (same idempotency key)" def claim_next():    row = db.execute("SELECT id, payload, attempts FROM jobs WHERE status = 'queued' ORDER BY id LIMIT 1").fetchone()    if row:        db.execute("UPDATE jobs SET status = 'running', attempts = attempts + 1 WHERE id = ?", (row[0],))    return row def run_agent(payload, attempt):    if attempt <= payload["fail_times"]:        raise RuntimeError("tool timeout")    return "report for " + payload["topic"] MAX_ATTEMPTS = 3 def worker_loop():    while True:        row = claim_next()        if row is None:            return        job_id, payload, attempts = row[0], json.loads(row[1]), row[2] + 1        try:            result = run_agent(payload, attempts)            db.execute("UPDATE jobs SET status = 'done', result = ? WHERE id = ?", (result, job_id))        except RuntimeError:            status = "failed" if attempts >= MAX_ATTEMPTS else "queued"          # retry or give up            db.execute("UPDATE jobs SET status = ? WHERE id = ?", (status, job_id)) print(submit("req-1", {"topic": "solar", "fail_times": 0}))print(submit("req-2", {"topic": "wind", "fail_times": 1}))print(submit("req-1", {"topic": "solar", "fail_times": 0}))print(submit("req-3", {"topic": "coal", "fail_times": 5}))worker_loop()for row in db.execute("SELECT id, idem_key, status, attempts, result FROM jobs ORDER BY id"):    print(row) # Capacity planning with Little's law: concurrent runs = arrival rate x time per rundef workers_needed(requests_per_second, seconds_per_run, runs_per_worker):    concurrent = requests_per_second * seconds_per_run    return concurrent, -(-concurrent // runs_per_worker) concurrent, workers = workers_needed(5, 20, 4)print("5 requests/s x 20 s per run =", concurrent, "runs at once ->", workers, "workers of 4 runs each")
EXPECTED OUTPUT (your numbers may differ slightly)
queued as job 1
queued as job 2
duplicate request ignored (same idempotency key)
queued as job 3
(1, 'req-1', 'done', 1, 'report for solar')
(2, 'req-2', 'done', 2, 'report for wind')
(3, 'req-3', 'failed', 3, None)
5 requests/s x 20 s per run = 100 runs at once -> 25 workers of 4 runs each

The duplicate submission is ignored, the flaky job succeeds on its second attempt, and the hopeless job is marked failed after three attempts. A real system would use a proper queue and database server, but the status logic is the same.

Knowledge Check

Question 1: What does an idempotency key protect against?
Question 2: According to Little's law, 5 requests per second at 20 seconds per run means how many concurrent runs?

Key Points

✓A production agent is a backend of API, queue, workers, database, cache and monitoring.
✓Separate accepting work from doing work. The API returns a job id and workers do the run.
✓Store all state outside the worker so any worker can resume any job.
✓Use status transitions, retry limits, a give-up state and idempotency keys.
✓Capacity planning: concurrent runs = arrival rate × seconds per run.
✓Deploy with containers, autoscaling, secret managers, graceful shutdown and staged rollouts.

The Big Picture

Production in One Line

Accept fast → queue durably → run on stateless workers → persist everything → scale on demand

Production is about making failures boring.

Up Next: Running at scale also means running at a cost. In Agent Performance and Cost Optimization you will learn how to cut latency, tokens and spend.