Efficient LLM Serving: Batching, PagedAttention and Speculative Decoding

Learn how production systems serve LLMs fast and cheaply: prefill vs. decode, latency and throughput metrics, continuous batching, PagedAttention, prefix caching, quantization and speculative decoding.

30 minAdvancedCode Examples

The Core Idea: Serving is a memory-bandwidth game. The big wins come from sharing each weight load across many requests (continuous batching), wasting less cache memory (paging), and getting more tokens out of every big-model pass (speculative decoding).

  1. Beginner: Serving Is a Different Problem From Training

Training happens once. Serving (also called inference) happens millions of times, for many users at once, and users are waiting for each answer. Serving well means answering fast, serving many people per GPU and keeping the cost per token low.

MetricMeaningWho feels it
Time to first token (TTFT)How long until the first word appearsEvery user, as the wait
Tokens per second per userHow fast the answer streamsEvery user, as the reading speed
ThroughputTotal tokens per second across all usersThe company, as cost
Cost per million tokensHardware cost divided by tokens servedThe company

The core tension: Handling more requests at once raises throughput and lowers cost, but can slow each individual user down. Good serving finds the best balance.

  1. Beginner: Prefill and Decode

Every request has two phases with very different behavior:

PrefillRead the prompt
What: Process all prompt tokens at once and fill the KV cache.
Bottleneck: Compute. Big parallel matrix math keeps the GPU busy.
User feels: Time to first token
DecodeWrite the answer
What: Generate one token at a time, reading the whole model and the cache for each token.
Bottleneck: Memory bandwidth. Little math per byte moved.
User feels: Streaming speed

Why this matters: Decode is slow because the GPU spends most of its time loading weights from memory to produce a single token. Anything that lets one weight load serve more tokens speeds things up a lot.

  1. Medium: Batching and Continuous Batching

Because loading the weights is the expensive part, serving many requests together is nearly free per request: one trip to memory feeds a whole batch of tokens. That is batching.

But requests finish at different times. With static batching, the whole batch waits for its longest request, and short requests leave slots empty. With continuous batching (also called in-flight batching), the moment a request finishes, a waiting request takes its slot, so the GPU stays busy.

Static batchingContinuous batching
Slot freed when a request endsStays empty until the whole batch endsImmediately refilled from the queue
GPU utilizationLow and unevenHigh
Short requestsWait for long onesLeave right away

Result: In the simulation below, continuous batching keeps the GPU about 80% busy versus about 60% for static batching, finishing the same work in fewer steps.

  1. Medium: PagedAttention and Prefix Caching

Each request needs a KV cache, and its final length is not known in advance. A naive server reserves the maximum length for every request, wasting most of the memory.

PagedAttention (introduced by the vLLM project) stores the cache in small fixed-size pages, allocated only as needed, like virtual memory in an operating system. Little memory is wasted, so many more requests fit on the same GPU.

Prefix caching: If many requests start with the same system prompt, their first pages of the cache are identical and can be shared instead of recomputed, which cuts both time and memory.

Quantization helps too
Smaller weights (8-bit, 4-bit) mean fewer bytes to load per token, which directly speeds up memory-bound decoding. Quantizing the KV cache lets even more requests fit.

  1. Advanced: Speculative Decoding

Decode produces one token per expensive pass. Speculative decoding uses a small, fast draft model to guess the next few tokens (for example 4), then asks the big model to check all of them in a single pass. The big model keeps the correct prefix and fixes the first mistake.

expected tokens per big-model step=1−αk+11−α\text{expected tokens per big-model step} = \frac{1 - \alpha^{k+1}}{1 - \alpha}

Here α\alpha is the chance that the big model accepts a drafted token and kk is how many tokens are drafted. The final answer has exactly the same quality as running the big model alone, because the big model verifies everything.

Accept rate αTokens per big step (k = 4)
0.6about 2.3
0.8about 3.4
0.9about 4.1

Trade-off: It needs an extra draft model and works best when the draft often agrees with the big model, for example on predictable text such as code.

  1. Visual Explanation: Keeping the GPU Busy

The figure compares static and continuous batching using the numbers from the simulation below, then summarizes four ways to speed up serving.

Part 1 · Static vs. Continuous BatchingKeep the GPU busy
Static batchingWasteful
8 requests start together
↓
Short ones finish early
↓
Their slots sit empty until the longest ends
↓
About 61% of GPU slots used
Continuous batchingEfficient
Request finishes
↓
Next request in the queue takes the slot at once
↓
No idle waiting
↓
About 82% of GPU slots used
Part 2 · Four Big Speed-UpsDifferent bottlenecks, different tricks
Batching
Share weight loads
One memory trip feeds many requests.
Paged KV cache
Waste less memory
Pages allocated only when needed.
Prefix cache
Reuse work
Shared prompts are computed once.
Speculative decoding
More tokens per step
A draft model proposes, the big model verifies.
Click to see where the numbers come fromCompare ▼
Static batching: each group of 8 waits for its longest request. Continuous: a slot is refilled the instant it frees. Paged cache: reserving 4,096 slots for 100 requests needs 409,600 slots, but paging by 16 uses only about 80,000 for the lengths in the simulation.

Part 1 shows why waiting for the longest request wastes slots. Part 2 pairs each trick with the bottleneck it fixes. Open the bar to see where the numbers come from.

  1. Python Implementation: Batching, Speculative Decoding and Paged Memory

This script simulates 32 requests of random length on a GPU with 8 slots under static and continuous batching, calculates the expected speed-up of speculative decoding, and compares reserving the maximum cache length against paging.

serving_simulation.pyPython 3.10+ · NumPy only
import numpy as np # 1. Static vs continuous batchingrng = np.random.default_rng(0)lengths = rng.integers(10, 200, size=32)                 # how many tokens each request will generateslots = 8                                                # requests the GPU can handle at once def static_batching(lengths, slots):    steps = 0    for i in range(0, len(lengths), slots):        steps += lengths[i:i + slots].max()              # the whole batch waits for its longest request    return steps def continuous_batching(lengths, slots):    queue = list(lengths)    running, steps = [], 0    while queue or running:        while queue and len(running) < slots:            # refill a slot the moment one frees up            running.append(queue.pop(0))        steps += 1        running = [r - 1 for r in running if r - 1 > 0]    return steps useful = int(lengths.sum())s, c = static_batching(lengths, slots), continuous_batching(lengths, slots)print("Useful tokens to generate:", useful)print("Static batching steps:", int(s), "| GPU busy", round(100 * useful / (s * slots)), "%")print("Continuous batching steps:", int(c), "| GPU busy", round(100 * useful / (c * slots)), "%") # 2. Speculative decoding: a small model drafts k tokens, the big model checks them at oncedef expected_tokens_per_step(accept_rate, k):    return (1 - accept_rate ** (k + 1)) / (1 - accept_rate) for a in [0.6, 0.8, 0.9]:    print("accept rate", a, "| draft 4 tokens -> about", round(expected_tokens_per_step(a, 4), 2), "tokens per big-model step") # 3. Paged KV cache: memory wasted by reserving the maximum length for every requestmax_len, page = 4096, 16used = rng.integers(50, 1500, size=100)reserved = len(used) * max_lenpaged = int(np.sum(np.ceil(used / page) * page))print("Reserved up front:", format(reserved, ","), "token slots | paged:", format(paged, ","), "token slots")
EXPECTED OUTPUT (your numbers may differ slightly)
Useful tokens to generate: 3510
Static batching steps: 724 | GPU busy 61 %
Continuous batching steps: 533 | GPU busy 82 %
accept rate 0.6 | draft 4 tokens -> about 2.31 tokens per big-model step
accept rate 0.8 | draft 4 tokens -> about 3.36 tokens per big-model step
accept rate 0.9 | draft 4 tokens -> about 4.1 tokens per big-model step
Reserved up front: 409,600 token slots | paged: 80,048 token slots

Continuous batching finishes the same 3,510 tokens in about a quarter fewer steps. Speculative decoding yields between about 2 and 4 tokens per big-model step depending on how often the draft is accepted. Paging uses roughly one fifth of the memory that up-front reservation does.

Interactive Knowledge Check

Question 1: Why is decoding usually limited by memory bandwidth?
Question 2: What is the benefit of continuous batching?

Key Points

✓Serving is judged by time to first token, streaming speed, throughput and cost per token.
✓Prefill is compute-bound. Decode is memory-bandwidth-bound.
✓Continuous batching refills freed slots immediately and keeps the GPU busy.
✓PagedAttention stores the KV cache in small pages so little memory is wasted.
✓Prefix caching shares the computed cache of common prompts across requests.
✓Speculative decoding drafts several tokens cheaply and verifies them in one pass without changing quality.

The Big Picture

Serving in One Line

Batch continuously + page the cache + reuse prefixes + draft and verify = more users per GPU

These techniques are why one GPU can serve many users and why API prices keep falling.

Up Next: You can now build and serve a model, but how do you know it is any good? In LLM Evaluation you will learn benchmarks, metrics, judges and how to measure honestly.