Modern LLM Architectures: Decoder-Only, RMSNorm, SwiGLU, GQA and MoE

Learn what a modern large language model is built from: the decoder-only design, pre-norm RMSNorm, SwiGLU feed-forward layers, RoPE, grouped-query attention and Mixture of Experts, and count the parameters of a 7B model in code.

30 minIntermediateCode Examples

The Core Idea: A modern LLM is just one decoder block repeated many times. Compared with the 2017 Transformer, each part got a practical upgrade: RMSNorm before sublayers, a SwiGLU feed-forward, RoPE for position, GQA for a smaller cache, and sometimes Mixture of Experts to run only part of the model.

  1. Beginner: What Is a Modern LLM Made Of?

The original Transformer from 2017 had two halves: an encoder that reads and a decoder that writes. Today's chatbots such as GPT-style, Llama-style and Mistral-style models keep only the decoder half and stack it many times. This design is called a decoder-only Transformer.

A decoder-only model is just one block repeated, for example 32 times for a 7B model or 80 times for a 70B model. Each block has two jobs: an attention part that mixes information between tokens, and a feed-forward part that processes each token on its own.

EmbeddingStart
Turns each token id into a vector of numbers.
N × Decoder BlockThe repeated part
Attention + feed-forward, each with a residual shortcut and a normalization step.
Output HeadEnd
Turns the final vector into a score for every word in the vocabulary.

Good news: Once you understand one block, you understand the whole model. The rest of this lesson is about what changed inside that block since 2017.

  1. Beginner: What Changed Since 2017

The overall shape stayed the same, but almost every small part was upgraded. Most modern open LLMs share these upgrades:

PartOriginal Transformer (2017)Typical modern LLMWhy it changed
Overall designEncoder + decoderDecoder-onlySimpler, and next-token training scales very well
NormalizationLayerNorm after each sublayerRMSNorm before each sublayer (pre-norm)Cheaper and trains more stably in deep stacks
Feed-forward activationReLUSwiGLUGating gives better quality for the same size
PositionSinusoidal, added onceRoPE inside attentionRelative positions and longer context
Attention headsMulti-head (MHA)Grouped-query (GQA)Smaller KV cache, faster generation
Bias termsUsed everywhereMostly removedFewer parameters, no loss in quality

Remember: None of these changes is a new magic idea. Each one is a practical improvement that makes models cheaper to train, more stable, or faster to run.

  1. Medium: RMSNorm and SwiGLU, the Two Quiet Upgrades

Normalization keeps the numbers flowing through a deep network at a healthy size. LayerNorm subtracts the average and divides by the spread. RMSNorm skips the subtraction and only rescales by the root-mean-square (RMS), which is simpler and works just as well in practice.

RMSNorm(x)=x1d∑ixi2+ϵ⋅g\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}} \cdot g

The feed-forward part of each block used to be two matrices with a ReLU in the middle. SwiGLU uses three matrices: a gate decides how much of an up signal is allowed through, then a down matrix returns to the model size.

SwiGLU(x)=(SiLU(xWgate)⊙xWup) Wdown\text{SwiGLU}(x) = \big(\text{SiLU}(xW_{gate}) \odot xW_{up}\big)\, W_{down}

Plain English: Think of a water pipe with a tap. The up path is the water, the gate is the tap that decides how much flows, and down collects it into the right size pipe. Because the tap is learned, the network can switch information on and off per token.

  1. Medium: Pre-Norm Blocks and Residual Shortcuts

Each decoder block follows the same recipe. The input is normalized first, passed through a sublayer, and then added back to the original input. This add-back is the residual connection, a shortcut that lets information and gradients flow straight through the whole stack.

1
RMSNorm: Rescale the incoming vector.
2
Attention with GQA and RoPE: Tokens look back at earlier tokens. RoPE is applied to Q and K.
3
Add the shortcut: Result = original input + attention output.
4
RMSNorm, then SwiGLU feed-forward: Each token is processed on its own.
5
Add the shortcut again: Result = previous result + feed-forward output.

Why pre-norm? Putting the normalization before the sublayer keeps the shortcut path clean. That makes it much easier to train networks with dozens of layers without the signal blowing up or fading away.

  1. Advanced: Mixture of Experts (MoE)

Making a model bigger usually makes it slower, because every token passes through every parameter. A Mixture of Experts model breaks the feed-forward part into several smaller experts and adds a tiny router. For each token the router picks only a few experts, such as 2 out of 8, and the rest stay idle.

Dense modelEvery parameter works
Per token: All parameters are used
Cost: Grows with total size
Example idea: A 7B model uses 7B parameters every step
Mixture of ExpertsOnly some experts work
Per token: Router picks the top few experts
Cost: Grows with the active size, not the total size
Example idea: A model with 8 experts can hold far more knowledge than it uses on any one token

Trade-off: MoE models are cheap to run per token but still need memory for all experts, and they are trickier to train because the router must spread work evenly. Several well-known recent open and closed models use this design.

A quick parameter count
A Llama-style 7B model has about 6.74 billion parameters: embeddings (131 million), then 32 blocks of attention (about 67 million each with full multi-head) and SwiGLU feed-forward (about 135 million each), plus an output head. The code below counts them exactly.

  1. Visual Explanation: One Modern Decoder Block

This is one decoder block, from input to output. Click each part to see what changed compared with the 2017 Transformer.

🏗️ WHAT CHANGED? · CLICK A PART2017 vs. today

  1. Python Implementation: RMSNorm, SwiGLU and a Parameter Counter

This script builds the two quiet upgrades from scratch, RMSNorm and a SwiGLU feed-forward layer, and then counts the parameters of a Llama-style 7B model. The counter reproduces the well-known figure of about 6.74 billion and shows how GQA makes the model smaller.

modern_llm_block.pyPython 3.10+ · NumPy only
import numpy as np rng = np.random.default_rng(0) def layer_norm(x, eps=1e-5):    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps) def rms_norm(x, weight, eps=1e-5):    rms = np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps)    return x / rms * weight def silu(x):    return x / (1 + np.exp(-x)) def swiglu_ffn(x, W_gate, W_up, W_down):    return (silu(x @ W_gate) * (x @ W_up)) @ W_down      # gate decides how much of "up" passes d_model, d_ff = 8, 16x = rng.normal(size=(3, d_model)) * 5 + 2                 # 3 tokens with large, off-center valuesweight = np.ones(d_model) normed = rms_norm(x, weight)print("RMS of each token after RMSNorm:", np.round(np.sqrt(np.mean(normed ** 2, axis=-1)), 3))print("Largest mean left after LayerNorm:", round(float(np.abs(layer_norm(x).mean(-1)).max()), 6), "(LayerNorm also re-centers to zero)") W_gate = rng.normal(size=(d_model, d_ff)) * 0.3W_up = rng.normal(size=(d_model, d_ff)) * 0.3W_down = rng.normal(size=(d_ff, d_model)) * 0.3out = swiglu_ffn(normed, W_gate, W_up, W_down)print("SwiGLU output shape:", out.shape) # Count the parameters of a Llama-style decoderdef count_params(layers, d_model, d_ff, vocab, heads, kv_heads, tied_embeddings=False):    head_dim = d_model // heads    attention = 2 * d_model * d_model + 2 * d_model * kv_heads * head_dim   # Q and O + K and V    ffn = 3 * d_model * d_ff                                                # gate, up, down    norms = 2 * d_model    embeddings = vocab * d_model    output_head = 0 if tied_embeddings else embeddings    return layers * (attention + ffn + norms) + embeddings + output_head + d_model mha = count_params(32, 4096, 11008, 32000, heads=32, kv_heads=32)gqa = count_params(32, 4096, 11008, 32000, heads=32, kv_heads=8)print("7B-style with multi-head attention:", format(mha, ","), "parameters")print("Same model with GQA (8 KV heads):  ", format(gqa, ","), "parameters")
EXPECTED OUTPUT (your numbers may differ slightly)
RMS of each token after RMSNorm: [1. 1. 1.]
Largest mean left after LayerNorm: 0.0 (LayerNorm also re-centers to zero)
SwiGLU output shape: (3, 8)
7B-style with multi-head attention: 6,738,415,616 parameters
Same model with GQA (8 KV heads): 5,933,109,248 parameters

The parameter count for the multi-head version is exactly 6,738,415,616, which matches the published size of a 7B Llama-style model. Switching to 8 KV heads removes about 0.8 billion parameters and, as you learned in the attention lesson, also shrinks the KV cache by 4×.

Interactive Knowledge Check

Question 1: What does pre-norm mean in a modern LLM block?
Question 2: What is the main benefit of a Mixture of Experts model?

Key Points

✓Most modern chat models are decoder-only: one block repeated many times.
✓A block = pre-RMSNorm + attention (with RoPE and GQA) + shortcut, then pre-RMSNorm + SwiGLU feed-forward + shortcut.
✓RMSNorm skips re-centering and only rescales, so it is simpler and cheaper than LayerNorm.
✓SwiGLU uses a learned gate to decide how much information passes through the feed-forward layer.
✓GQA shares keys and values between heads, saving parameters and KV-cache memory.
✓Mixture of Experts activates only a few experts per token, so cost follows the active size, not the total size.

The Big Picture

A Modern LLM in One Line

Embedding → N × (RMSNorm → GQA + RoPE → RMSNorm → SwiGLU) → Output head

You can now read the architecture description of almost any open LLM. Differences between models are mostly the number of layers, the width, the number of KV heads, and whether the feed-forward part is dense or Mixture of Experts.

Up Next: You know how an LLM is built. In LLM Pre-Training you will see how that empty network learns language from trillions of tokens, and how much compute that costs.