Attention in LLMs: Causal Masking and KV Caching

Understand how causal masking keeps a language model from seeing the future, and how the KV cache lets it generate text without redoing old work.

30 minIntermediateCode Examples

The Core Idea: When a language model writes a sentence, every new word is chosen by looking back at the words that came before it. Causal masking is the rule that stops the model from looking at future words. The KV cache is the memory trick that stops it from redoing old work at every step. Together they make training fair and generation fast.

1. Beginner: Attention as a Smart Lookup

A token is a small piece of text, often a word or part of a word. Attention is the step where each token looks at other tokens and decides which ones matter for understanding itself. Think of a student writing an answer with a stack of notes on the desk. For every new sentence, the student scans the notes, picks the useful ones, and blends them into the answer.
Attention formalizes that with three roles. Every token produces all three of them from its own vector:
Query
The question on a sticky note: what am I looking for right now?
Key
The label on a note: what do I contain, so others can match me?
Value
The actual content of the note: what do I hand over if I am chosen?
The recipe is simple. Compare the Query of the current token with the Key of every available token to get scores. Turn the scores into percentages with softmax (a function that makes numbers positive and makes them add up to 1). Then take a weighted mix of the Values using those percentages. A token whose Key matches well contributes a lot; a token that matches poorly contributes almost nothing.

Key question for this lesson: which tokens are the Keys and Values allowed to come from? The answer is the whole point of causal masking.

2. Beginner: The No-Peeking Rule

Imagine an exam where the question is: The cat sat on the ___. If the answer sheet is printed right next to the question, you can simply copy it. A language model is trained to predict the next token, so during training it must never be allowed to see the answer sheet, which here means the tokens that come later in the sentence.
A model that generates one token after another is called autoregressive: each new token is predicted from all earlier tokens, then added to the text, and the loop repeats. GPT-style chat models work this way.
During training
The whole sentence is fed in at once so the GPU can work on every position in parallel. Without protection, position 3 could read position 4 and cheat. A causal mask blocks every connection that points to the future.
During generation
Future tokens do not exist yet, so nothing can be peeked at. The model must behave exactly as it did in training, which is why training uses the mask: the two situations must match.

Analogy: reading a book with a sliding card that covers everything to the right of your current word. You may re-read anything on the left as often as you like, but the right side stays hidden.

3. Intermediate: The Math of Causal Masking

Let XX hold one vector per token, one row each. Three learned weight matrices turn it into queries, keys and values: Q=XWQQ = XW_Q, K=XWKK = XW_K, V=XWVV = XW_V. Causal attention then computes:
Attention(Q,K,V)=softmax(QKTdk+M)V\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\dfrac{QK^{T}}{\sqrt{d_k}} + M\right)V
Here dkd_k is the size of each key vector. Dividing by dk\sqrt{d_k} keeps the scores from growing too large when vectors are long, which would make softmax too extreme. The matrix MM is the mask: Mij=0M_{ij} = 0 when j≤ij \leq i, and Mij=−∞M_{ij} = -\infty when j>ij \gt i. Because e−∞=0e^{-\infty} = 0, softmax gives every future position a weight of exactly zero.
Step 1. Project every token into a query, key and value vector.
Step 2. Score every pair: S=QKT/dkS = QK^{T} / \sqrt{d_k}. Entry SijS_{ij} says how much token ii matches token jj.
Step 3. Add the mask so every score above the diagonal becomes −∞-\infty.
Step 4. Apply softmax to each row so the weights of a row add up to 1.
Step 5. Multiply by VV to get one blended output vector per token.
For four tokens, the mask pattern looks like this. A check mark means the row token may read the column token:
Row reads →Token 1Token 2Token 3Token 4
Token 1✓−∞−∞−∞
Token 2✓✓−∞−∞
Token 3✓✓✓−∞
Token 4✓✓✓✓

Worked example. Suppose token 3 has raw scores 2.0, 1.0, 0.5 against tokens 1, 2, 3, and −∞ against token 4 after masking. Softmax uses e2.0≈7.39e^{2.0} \approx 7.39, e1.0≈2.72e^{1.0} \approx 2.72, e0.5≈1.65e^{0.5} \approx 1.65 and e−∞=0e^{-\infty} = 0. The sum is about 11.76, so the weights are roughly 0.629, 0.231, 0.140 and 0. They add up to 1, and the future token gets no say.

4. Intermediate: Why Generation Gets Slow

Now look at generation. Suppose the prompt is The cat sat and the model must produce the next token. To choose it, the model only needs the output at the last position, but that position needs the Keys and Values of every earlier token. A naive loop would rebuild everything from scratch after each new token: embed all tokens, project all of them into K and V in every layer, run attention, and keep only the last row of the result.
That is wasteful. If the model generates nn tokens, step tt reprocesses tt positions, so the total number of token positions processed is 1+2+⋯+n=n(n+1)/21 + 2 + \dots + n = n(n+1)/2 instead of just nn.
Tokens generatedPositions processed without cachePositions processed with cache
105510
1005,050100
1,000500,5001,000

The key observation. Because of the causal mask, the output of token jj depends only on tokens 11 to jj. When a new token arrives at the end, the Keys and Values of all older tokens do not change at all, since the model weights are frozen during generation. Old work can be saved and reused. This only works because of causal masking: in a model where old tokens could read new ones, every old vector would change at each step and nothing could be reused.

5. Advanced: The KV Cache

The KV cache stores the Key and Value vectors of every token already processed, in every layer, in GPU memory. Inference then has two phases:
Prefill
The whole prompt is processed in one parallel pass, using the causal mask. All its Keys and Values are written into the cache. This phase is heavy on computation.
Decode
One token at a time. The new token computes its own query, key and value. Its key and value are appended to the cache, and its query reads all cached Keys and Values. No mask is needed, because the cache holds only the past.
Why cache K and V but not Q? A query is used once, by its own token, to read the past. After that token has produced its output, its query is never needed again. Keys and Values, however, are read by every future token, so they are worth keeping.
The price is memory. The cache grows by one entry per token, per layer. Its size in bytes is:
bytes per token=2×L×Hkv×dhead×b\text{bytes per token} = 2 \times L \times H_{kv} \times d_{head} \times b
The factor 2 counts both K and V, LL is the number of layers, HkvH_{kv} the number of key/value heads, dheadd_{head} the size of each head, and bb the bytes per stored number (2 for 16-bit values). For a classic 7B-class setup with 32 layers, 32 heads, head size 128 and 16-bit values, that is 2×32×32×128×2=524,2882 \times 32 \times 32 \times 128 \times 2 = 524{,}288 bytes, or 512 KiB per token.
Context lengthCache size for ONE sequence
1,024 tokens0.5 GiB
4,096 tokens2 GiB
32,768 tokens16 GiB
Servers handle many users at once, and each user has a separate cache, so memory quickly becomes the limit. Every decode step also has to read the whole cache from memory, which is why long-context generation is often limited by memory bandwidth rather than raw compute. Common ways to reduce the cost:
Fewer K/V heads
Multi-Query Attention and Grouped-Query Attention let many query heads share a small number of key/value heads, shrinking HkvH_{kv}.
Smarter memory layout
Paged cache management (as in the vLLM server) stores the cache in small blocks so memory is not wasted on empty space.
Smaller numbers
Cache quantization stores K and V with fewer bits, trading a little precision for a smaller footprint.

6. Visual Explanation: Masking and Caching Side by Side

The figure below has two parts. Part 1 shows who is allowed to read whom, and how the same rule applies in training and in generation. Part 2 shows the three moves of the KV cache.
Part 1 · Causal MaskingRow = token asking, column = token being read
Mask · 4 TokensLower triangle
✓✗✗✗✓✓✗✗✓✓✓✗✓✓✓✓
Row i sees only columns 1 to i
Same Rule EverywhereNo peeking
Training: all rows at once, mask applied
↓
Generation: one new row per step
↓
The model never sees the future
Part 2 · The KV Cache LoopOld K and V are stored, only the new token is computed
Step 1 · Prefill
Whole Prompt
Process every prompt token in parallel and store all K and V in GPU memory.
Step 2 · Decode
One New Token
Compute q, k and v for the new token only, then read the cached K and V.
Step 3 · Append
Cache Grows
Add the new k and v to the cache, output the token, and repeat from Step 2.
Result: n positions processed instead of n(n+1)/2
Click to see why only K and V are cachedExplain ▼
A query is used once, by the token that created it, to read the past. Keys and Values are read again by every later token, so they are the only parts worth storing. The price is memory that grows with every token.
How to read it: start with the mask grid, then follow the three cache steps from left to right. The loop of Step 2 and Step 3 repeats for every generated token.

7. Python Implementation: Causal Attention with a KV Cache

The program below builds a tiny single-head attention layer with NumPy. It computes causal attention for a whole sequence at once (training style), then recomputes the same outputs one token at a time using a KV cache (generation style), and checks that both give identical results. It ends by computing the cache size for a real-sized model.
causal_attention_kv_cache.pyPython 3.11+ · NumPy only
import numpy as np np.random.seed(0) # Tiny sizes so the output stays readabled_model = 8d_k = 4seq_len = 5 # Random projection weights (frozen, like a trained model)W_q = np.random.randn(d_model, d_k)W_k = np.random.randn(d_model, d_k)W_v = np.random.randn(d_model, d_k)X = np.random.randn(seq_len, d_model) def softmax(scores):    scores = scores - scores.max(axis=-1, keepdims=True)    e = np.exp(scores)    return e / e.sum(axis=-1, keepdims=True) # Training style: all tokens at once, with a causal maskdef full_causal_attention(X):    n = X.shape[0]    Q = X @ W_q    K = X @ W_k    V = X @ W_v    scores = Q @ K.T / np.sqrt(d_k)    mask = np.triu(np.ones((n, n)), k=1) == 1    scores = np.where(mask, -np.inf, scores)    weights = softmax(scores)    return weights @ V, weights # Generation style: a cache holding all past keys and valuesdef new_cache():    return {"K": np.zeros((0, d_k)), "V": np.zeros((0, d_k))} def decode_step(x_new, cache):    q = x_new @ W_q    k = x_new @ W_k    v = x_new @ W_v    cache["K"] = np.vstack([cache["K"], k])    cache["V"] = np.vstack([cache["V"], v])    scores = cache["K"] @ q / np.sqrt(d_k)    weights = softmax(scores)    return weights @ cache["V"] full_out, full_w = full_causal_attention(X) cache = new_cache()cached_out = np.array([decode_step(X[t], cache) for t in range(seq_len)]) print("Outputs match:", np.allclose(full_out, cached_out))print("Future weights are zero:", np.allclose(np.triu(full_w, k=1), 0))print("Rows sum to 1:", np.allclose(full_w.sum(axis=1), 1))print("First token attends only to itself:", np.round(full_w[0], 3))print("Cache shape after 5 tokens:", cache["K"].shape) # Memory cost of a real-sized cache (7B-class settings, 16-bit values)layers = 32kv_heads = 32head_dim = 128bytes_per_value = 2per_token = 2 * layers * kv_heads * head_dim * bytes_per_valueprint("KV cache per token (KiB):", per_token / 1024)print("KV cache for 4096 tokens (GiB):", per_token * 4096 / (1024 * 1024 * 1024))
Expected output
Outputs match: TrueFuture weights are zero: TrueRows sum to 1: TrueFirst token attends only to itself: [1. 0. 0. 0. 0.]Cache shape after 5 tokens: (5, 4)KV cache per token (KiB): 512.0KV cache for 4096 tokens (GiB): 2.0
Notice that decode_step never builds a mask. The cache contains only past tokens plus the current one, so the mask is built into the data itself. This is why the two approaches agree.

Knowledge Check

Q1. What does the causal mask do to the attention score of a future token?
Q2. Which vectors does the KV cache store?
Q3. Why does the KV cache become a problem for very long conversations?

Key Points

✓
Attention compares a Query with Keys to get scores, turns them into weights with softmax, and blends the Values.
✓
Causal masking adds −∞ to every score above the diagonal, so each token reads only itself and earlier tokens.
✓
The mask lets training run on a full sequence in parallel without letting the model cheat.
✓
Without a cache, generating n tokens processes about n(n+1)/2 positions; with a cache it processes n.
✓
The cache works because of causality: the Keys and Values of old tokens never change once computed.
✓
Inference has two phases: prefill (whole prompt in parallel) and decode (one token at a time using the cache).
✓
The cache trades memory for speed. Its size is 2 × layers × KV heads × head size × bytes, per token, per sequence.

The Big Picture

Causal masking and the KV cache are two sides of the same property. The mask guarantees that each token depends only on the past, and that guarantee is exactly what makes it safe to store the past and reuse it. Training uses the mask to learn fairly in parallel; generation uses the cache to stay fast one token at a time.
Mask the future → Attend to the past → Cache K and V → Generate fast

Up Next: Decoding Strategies: Temperature, Top-k and Top-p Sampling. Once the model has computed its output for the next position, how do we actually pick the token? That choice shapes how creative or predictable the text feels.