The Core Idea: Attention treats text like a bag of words, so a model needs a way to know where each token sits. Modern LLMs use RoPE: they rotate each query and key by an amount that depends on position, so the attention score only depends on the distance between two tokens.
- Beginner: Attention Cannot Tell Word Order
Self-attention compares every token with every other token, but it does it like a bag of words. If you shuffle the words, the raw attention math gives the same pairs of scores. To a plain attention layer, dog bites man and man bites dog look almost identical, yet they mean very different things.
So we must inject position information into the model. This is called positional encoding. The question is how, and the answer changed a lot between the original Transformer (2017) and today's LLMs.
Simple analogy: Imagine a class photo where nobody is standing in order. You know who is in the photo, but not who stands next to whom. Positional encoding is like putting a numbered sticker on every person.
- Beginner: Three Ways to Add Position
Researchers have tried several families of ideas. Here is the whole story in one table:
| Method | How it works | Used by | Main weakness |
|---|---|---|---|
| Sinusoidal (absolute) | Adds a fixed wave pattern for each position to the token embedding | Original Transformer (2017) | Struggles on text longer than seen in training |
| Learned (absolute) | Learns one vector per position, like a lookup table | GPT-2, BERT | Hard limit: no vector exists for unseen positions |
| Relative (RoPE, ALiBi) | Encodes the distance between two tokens inside the attention score | Most modern LLMs (RoPE is the most popular) | Needs tricks to stretch to much longer text |
The key insight: What usually matters in language is how far apart two words are, not their exact seat numbers. Relative methods build that directly into attention, and that is why they won.
- Medium: RoPE (Rotary Position Embedding) in Plain English
RoPE gives each token's query and key vector a rotation that depends on its position. Token at position 0 is not rotated, position 1 is rotated a little, position 2 a bit more, and so on, like turning a dial one notch per word.
The magic happens in the dot product. When query and key are both rotated, the angle between them depends only on the difference of their positions. The score between word 5 and word 2 is therefore the same as between word 105 and word 102, because both pairs are exactly 3 apart.
Why it is elegant: RoPE adds no extra parameters and does not touch the values. It only rotates queries and keys, and rotation never changes a vector's length, so the signal stays stable.
- Medium: A Tiny Worked Example With One Rotating Arrow
To see the idea with numbers, use a single pair of dimensions, a 2D arrow starting at , and a rotation of 30° per position. Each position turns the arrow a further 30°:
| Position | Total rotation | Arrow after rotation |
|---|---|---|
| 0 | 0° | (1.000, 0.000) |
| 1 | 30° | (0.866, 0.500) |
| 2 | 60° | (0.500, 0.866) |
| 3 | 90° | (0.000, 1.000) |
Now take the dot product of an arrow at position with an identical arrow at position . The result is . Only the gap matters:
| Pair of positions | Gap | Dot product |
|---|---|---|
| (3, 3) | 0 | 1.000 |
| (3, 2) | 1 | 0.866 |
| (3, 1) | 2 | 0.500 |
| (3, 0) | 3 | 0.000 |
| (13, 10) | 3 | 0.000 |
See it? The pairs (3, 0) and (13, 10) are far apart in absolute position, yet they get the same score because both gaps are 3. That is the relative-position property, and nobody had to teach it to the model.
- Advanced: Stretching the Context Window
A model trained with positions up to 4,096 has never seen a rotation angle for position 20,000. If you simply feed it longer text, the angles go outside the range it learned, and quality drops sharply. Several techniques stretch the usable context:
| Technique | Idea | Trade-off |
|---|---|---|
| Position interpolation | Squeeze new positions into the old range, so position 8,000 is treated as 4,000 | Needs a short fine-tune; nearby words become slightly harder to tell apart |
| NTK-aware scaling | Change the rotation speeds (the base number) instead of all positions equally, keeping fast pairs sharp | Works with little or no fine-tuning |
| YaRN | Mixes the two ideas and adjusts attention strength | Strong results with a small amount of extra training |
| ALiBi | Skips rotations and subtracts a penalty that grows with distance | Simple, but a different family from RoPE |
- Visual Explanation: The Rotating Arrows and the Gap Rule
Each token gets turned a little more than the one before it. Pick a position to see where its arrow points (using 30° per position) and how similar it is to the arrow at position 0.
- Python Implementation: RoPE From Scratch and the Relative-Position Test
This function rotates every pair of dimensions by the angle position × speed. Then we test the key promise of RoPE: the score depends only on the distance between tokens, not on where they sit in the text.
import numpy as np def rope(x, position, base=10000.0): # Rotate each pair of dimensions (x[2i], x[2i+1]) by an angle that grows with position d = len(x) i = np.arange(d // 2) theta = base ** (-2 * i / d) # one frequency per pair: fast to slow angle = position * theta cos, sin = np.cos(angle), np.sin(angle) out = np.empty_like(x) out[0::2] = x[0::2] * cos - x[1::2] * sin out[1::2] = x[0::2] * sin + x[1::2] * cos return out rng = np.random.default_rng(0)q = rng.normal(size=8)k = rng.normal(size=8) # 1. Same distance (3 apart) at very different absolute positionsa = np.dot(rope(q, 5), rope(k, 2))b = np.dot(rope(q, 105), rope(k, 102))print("score at positions (5, 2): ", round(float(a), 4))print("score at positions (105, 102): ", round(float(b), 4))print("Same score for the same distance:", np.isclose(a, b)) # 2. A different distance gives a different scorec = np.dot(rope(q, 5), rope(k, 4))print("score at distance 1:", round(float(c), 4), "| at distance 3:", round(float(a), 4)) # 3. Rotation never changes the length of a vectorprint("Length before and after rotation:", round(float(np.linalg.norm(q)), 4), round(float(np.linalg.norm(rope(q, 50))), 4)) # 4. Position interpolation: squeeze 8192 positions into the 4096 range seen in trainingscale = 4096 / 8192print("Position 8000 is treated as position", 8000 * scale)Look at the first three lines of output: positions (5, 2) and (105, 102) are 100 steps apart in absolute terms but both have a gap of 3, and the scores are identical. Distance 1 gives a different score, so the model can tell near from far.
Interactive Knowledge Check
Key Points
The Big Picture
Token meaning + rotation by position → attention sees distance, not seat numbers
Order is what turns a pile of words into a sentence. RoPE gives the model that sense of order in a way that is cheap, stable and works at many text lengths.
Up Next: You now know the two most important parts inside an LLM block: attention and positions. In Modern LLM Architectures you will see how they are combined with RMSNorm, SwiGLU, GQA and Mixture of Experts into today's models.