The Core Idea: Serving is a memory-bandwidth game. The big wins come from sharing each weight load across many requests (continuous batching), wasting less cache memory (paging), and getting more tokens out of every big-model pass (speculative decoding).
- Beginner: Serving Is a Different Problem From Training
Training happens once. Serving (also called inference) happens millions of times, for many users at once, and users are waiting for each answer. Serving well means answering fast, serving many people per GPU and keeping the cost per token low.
| Metric | Meaning | Who feels it |
|---|---|---|
| Time to first token (TTFT) | How long until the first word appears | Every user, as the wait |
| Tokens per second per user | How fast the answer streams | Every user, as the reading speed |
| Throughput | Total tokens per second across all users | The company, as cost |
| Cost per million tokens | Hardware cost divided by tokens served | The company |
The core tension: Handling more requests at once raises throughput and lowers cost, but can slow each individual user down. Good serving finds the best balance.
- Beginner: Prefill and Decode
Every request has two phases with very different behavior:
Why this matters: Decode is slow because the GPU spends most of its time loading weights from memory to produce a single token. Anything that lets one weight load serve more tokens speeds things up a lot.
- Medium: Batching and Continuous Batching
Because loading the weights is the expensive part, serving many requests together is nearly free per request: one trip to memory feeds a whole batch of tokens. That is batching.
But requests finish at different times. With static batching, the whole batch waits for its longest request, and short requests leave slots empty. With continuous batching (also called in-flight batching), the moment a request finishes, a waiting request takes its slot, so the GPU stays busy.
| Static batching | Continuous batching | |
|---|---|---|
| Slot freed when a request ends | Stays empty until the whole batch ends | Immediately refilled from the queue |
| GPU utilization | Low and uneven | High |
| Short requests | Wait for long ones | Leave right away |
Result: In the simulation below, continuous batching keeps the GPU about 80% busy versus about 60% for static batching, finishing the same work in fewer steps.
- Medium: PagedAttention and Prefix Caching
Each request needs a KV cache, and its final length is not known in advance. A naive server reserves the maximum length for every request, wasting most of the memory.
PagedAttention (introduced by the vLLM project) stores the cache in small fixed-size pages, allocated only as needed, like virtual memory in an operating system. Little memory is wasted, so many more requests fit on the same GPU.
Prefix caching: If many requests start with the same system prompt, their first pages of the cache are identical and can be shared instead of recomputed, which cuts both time and memory.
- Advanced: Speculative Decoding
Decode produces one token per expensive pass. Speculative decoding uses a small, fast draft model to guess the next few tokens (for example 4), then asks the big model to check all of them in a single pass. The big model keeps the correct prefix and fixes the first mistake.
Here is the chance that the big model accepts a drafted token and is how many tokens are drafted. The final answer has exactly the same quality as running the big model alone, because the big model verifies everything.
| Accept rate α | Tokens per big step (k = 4) |
|---|---|
| 0.6 | about 2.3 |
| 0.8 | about 3.4 |
| 0.9 | about 4.1 |
Trade-off: It needs an extra draft model and works best when the draft often agrees with the big model, for example on predictable text such as code.
- Visual Explanation: Keeping the GPU Busy
The figure compares static and continuous batching using the numbers from the simulation below, then summarizes four ways to speed up serving.
Click to see where the numbers come fromCompare ▼
Part 1 shows why waiting for the longest request wastes slots. Part 2 pairs each trick with the bottleneck it fixes. Open the bar to see where the numbers come from.
- Python Implementation: Batching, Speculative Decoding and Paged Memory
This script simulates 32 requests of random length on a GPU with 8 slots under static and continuous batching, calculates the expected speed-up of speculative decoding, and compares reserving the maximum cache length against paging.
import numpy as np # 1. Static vs continuous batchingrng = np.random.default_rng(0)lengths = rng.integers(10, 200, size=32) # how many tokens each request will generateslots = 8 # requests the GPU can handle at once def static_batching(lengths, slots): steps = 0 for i in range(0, len(lengths), slots): steps += lengths[i:i + slots].max() # the whole batch waits for its longest request return steps def continuous_batching(lengths, slots): queue = list(lengths) running, steps = [], 0 while queue or running: while queue and len(running) < slots: # refill a slot the moment one frees up running.append(queue.pop(0)) steps += 1 running = [r - 1 for r in running if r - 1 > 0] return steps useful = int(lengths.sum())s, c = static_batching(lengths, slots), continuous_batching(lengths, slots)print("Useful tokens to generate:", useful)print("Static batching steps:", int(s), "| GPU busy", round(100 * useful / (s * slots)), "%")print("Continuous batching steps:", int(c), "| GPU busy", round(100 * useful / (c * slots)), "%") # 2. Speculative decoding: a small model drafts k tokens, the big model checks them at oncedef expected_tokens_per_step(accept_rate, k): return (1 - accept_rate ** (k + 1)) / (1 - accept_rate) for a in [0.6, 0.8, 0.9]: print("accept rate", a, "| draft 4 tokens -> about", round(expected_tokens_per_step(a, 4), 2), "tokens per big-model step") # 3. Paged KV cache: memory wasted by reserving the maximum length for every requestmax_len, page = 4096, 16used = rng.integers(50, 1500, size=100)reserved = len(used) * max_lenpaged = int(np.sum(np.ceil(used / page) * page))print("Reserved up front:", format(reserved, ","), "token slots | paged:", format(paged, ","), "token slots")Continuous batching finishes the same 3,510 tokens in about a quarter fewer steps. Speculative decoding yields between about 2 and 4 tokens per big-model step depending on how often the draft is accepted. Paging uses roughly one fifth of the memory that up-front reservation does.
Interactive Knowledge Check
Key Points
The Big Picture
Batch continuously + page the cache + reuse prefixes + draft and verify = more users per GPU
These techniques are why one GPU can serve many users and why API prices keep falling.
Up Next: You can now build and serve a model, but how do you know it is any good? In LLM Evaluation you will learn benchmarks, metrics, judges and how to measure honestly.