The Core Idea: A production agent is not one script. It is a backend: an API accepts requests, a queue holds jobs, workers run the agent loop, a database stores state and results, and monitoring watches everything. The design goal is that no single failure loses work.
- Beginner: The Agent Backend
In a demo, you run an agent in a notebook and wait. In production, many users send requests at once, runs can take minutes, and servers restart. So the agent runs as a service made of several parts:
| Part | Job |
|---|---|
| API layer | Receives requests, checks identity, returns a job id or a result |
| Queue | Holds work until a worker is free |
| Workers | Run the agent loop: call the model, run tools, update state |
| Database | Stores jobs, state, results and history |
| Cache | Speeds up repeated work and reduces cost |
| Monitoring | Logs, traces, metrics and alerts |
Key idea: Separate accepting work from doing work. The API answers fast, and workers take their time.
- Beginner: APIs for Agents
Agent runs are slow compared with normal web requests, so the API usually works asynchronously:
- Medium: Databases and State Persistence
Everything the agent needs to resume must be stored outside the process: the goal, the step history, tool results and the current status. If a worker dies, another worker can load the state and continue.
| Store | Typical content |
|---|---|
| Relational database | Jobs, users, status, results |
| Key-value store | Fast session state and locks |
| Object storage | Large files and artifacts |
| Vector database | Semantic memory and documents |
In the code: Jobs live in a database table with a status column: queued, running, done or failed. A restart does not lose them.
- Medium: Queues, Retries and Idempotency
A queue decouples the API from the workers and smooths traffic spikes. Because work can be retried, every job must be safe to run more than once.
| Idea | Purpose | In the code |
|---|---|---|
| Status transitions | Track where each job is | queued, running, done, failed |
| Retries with a limit | Recover from temporary failures | req-2 failed once, then succeeded on attempt 2 |
| Give-up state | Stop hopeless jobs | req-3 failed after 3 attempts |
| Idempotency key | Ignore duplicate submissions | The second req-1 was ignored |
Why idempotency matters: Networks cause duplicate requests, and workers crash mid-job. Without idempotency a customer could be refunded twice or emailed twice.
- Advanced: Scaling and Deployment
Capacity planning starts with Little's law: the number of runs in progress equals the arrival rate times the time each run takes.
In the code: 5 requests per second at 20 seconds per run means 100 runs at once. With 4 runs per worker, you need 25 workers.
| Practice | Why |
|---|---|
| Stateless workers | Any worker can run any job, so you can add or remove workers freely |
| Autoscale on queue length | Match capacity to demand |
| Respect model provider rate limits | The model API can be the real bottleneck |
| Containers and a deployment pipeline | Repeatable releases and fast rollbacks |
| Secrets from a secret manager | Never in code or prompts |
| Graceful shutdown | Finish or re-queue running jobs on a restart |
| Staged rollouts and monitoring | Catch bad releases early |
- Visual Explanation: The Request Path of a Production Agent
The figure shows the path of a request through the API, queue, worker and database, then summarizes what the code demonstrated about jobs and capacity.
Click to see why workers should be statelessCompare ▼
Part 1 reads left to right. Part 2 explains the four jobs and the capacity result. Open the bar to see why stateless workers matter.
- Python Implementation: A Durable Job Queue and Capacity Planning
This script builds a small durable job queue in a database with status transitions, retries with a limit, a give-up state and an idempotency key that ignores duplicate submissions. It ends with a capacity calculation based on Little's law.
import json, sqlite3 # A durable job queue in a database: the heart of a production agent backenddb = sqlite3.connect(":memory:")db.execute("""CREATE TABLE jobs (id INTEGER PRIMARY KEY, idem_key TEXT UNIQUE, payload TEXT, status TEXT, attempts INTEGER DEFAULT 0, result TEXT)""") def submit(idem_key, payload): try: cursor = db.execute("INSERT INTO jobs (idem_key, payload, status) VALUES (?, ?, 'queued')", (idem_key, json.dumps(payload))) return "queued as job " + str(cursor.lastrowid) except sqlite3.IntegrityError: return "duplicate request ignored (same idempotency key)" def claim_next(): row = db.execute("SELECT id, payload, attempts FROM jobs WHERE status = 'queued' ORDER BY id LIMIT 1").fetchone() if row: db.execute("UPDATE jobs SET status = 'running', attempts = attempts + 1 WHERE id = ?", (row[0],)) return row def run_agent(payload, attempt): if attempt <= payload["fail_times"]: raise RuntimeError("tool timeout") return "report for " + payload["topic"] MAX_ATTEMPTS = 3 def worker_loop(): while True: row = claim_next() if row is None: return job_id, payload, attempts = row[0], json.loads(row[1]), row[2] + 1 try: result = run_agent(payload, attempts) db.execute("UPDATE jobs SET status = 'done', result = ? WHERE id = ?", (result, job_id)) except RuntimeError: status = "failed" if attempts >= MAX_ATTEMPTS else "queued" # retry or give up db.execute("UPDATE jobs SET status = ? WHERE id = ?", (status, job_id)) print(submit("req-1", {"topic": "solar", "fail_times": 0}))print(submit("req-2", {"topic": "wind", "fail_times": 1}))print(submit("req-1", {"topic": "solar", "fail_times": 0}))print(submit("req-3", {"topic": "coal", "fail_times": 5}))worker_loop()for row in db.execute("SELECT id, idem_key, status, attempts, result FROM jobs ORDER BY id"): print(row) # Capacity planning with Little's law: concurrent runs = arrival rate x time per rundef workers_needed(requests_per_second, seconds_per_run, runs_per_worker): concurrent = requests_per_second * seconds_per_run return concurrent, -(-concurrent // runs_per_worker) concurrent, workers = workers_needed(5, 20, 4)print("5 requests/s x 20 s per run =", concurrent, "runs at once ->", workers, "workers of 4 runs each")The duplicate submission is ignored, the flaky job succeeds on its second attempt, and the hopeless job is marked failed after three attempts. A real system would use a proper queue and database server, but the status logic is the same.
Knowledge Check
Key Points
The Big Picture
Accept fast → queue durably → run on stateless workers → persist everything → scale on demand
Production is about making failures boring.
Up Next: Running at scale also means running at a cost. In Agent Performance and Cost Optimization you will learn how to cut latency, tokens and spend.