The Core Idea: Pre-training teaches an LLM language with one simple game: guess the next token. Repeat it over trillions of tokens of text and the model absorbs grammar, facts and reasoning. The price is enormous compute, which follows a simple rule: 6 × parameters × tokens.
- Beginner: What Is Pre-Training?
A brand-new LLM is just a huge pile of random numbers. It cannot answer questions, write code or even finish a sentence. Pre-training is the long first stage where it learns the general patterns of language by reading an enormous amount of text.
The task it practices is incredibly simple: guess the next token. Show it The capital of France is and it should give a high score to Paris. Repeat this trillions of times on text from books, websites, articles and code, and the model slowly picks up grammar, facts, reasoning patterns and even coding style, all as a side effect.
Why this works: The text itself is the teacher. Every sentence on the internet is already a free exercise with its own answer, so no human has to label anything. This is called self-supervised learning.
- Beginner: The Data Recipe
Quality and variety of data matter as much as the model itself. A typical pre-training dataset is built in several steps:
Rule of thumb: Better data beats more data. Careful cleaning and de-duplication often improve a model more than simply adding another pile of raw text.
- Medium: The Loss, Perplexity and the Training Loop
The model gives a probability to every possible next token. The loss measures how surprised it was by the real one. This measure is called cross-entropy:
If the model gave the right word probability 1.0, the loss is 0. If it gave only 0.01, the loss is large (about 4.6). Perplexity is just and has a friendly meaning: the model is as confused as if it had to choose between that many equally likely words.
| Probability given to the correct word | Loss | Perplexity |
|---|---|---|
| 0.90 | 0.105 | 1.11 |
| 0.50 | 0.693 | 2.00 |
| 0.10 | 2.303 | 10.0 |
| 0.01 | 4.605 | 100 |
Each training step is the loop you already know from gradient descent: run a batch of text through the model, compute the loss, compute gradients with backpropagation, and update the weights with an optimizer (usually AdamW with a warm-up and a gradual cosine decay of the learning rate).
Good news: This is the same loop you coded in the Gradient Descent lesson. Pre-training simply runs it at a gigantic scale with a gigantic dataset.
- Medium: Scaling Laws, Compute and Cost
One of the most important discoveries about LLMs is that their loss improves in a smooth, predictable way as you increase three things together: the number of parameters (), the amount of training tokens () and the compute. Researchers can therefore predict how good a big model will be by training small ones first.
Training compute is estimated with a famous rule of thumb: about 6 floating-point operations per parameter per token.
Chinchilla rule of thumb: For a given compute budget, a model trains best with roughly 20 tokens per parameter. A 7B model should see about 140 billion tokens. Many recent models deliberately train on far more than that, because a smaller model trained longer is cheaper to run for millions of users afterwards.
- Advanced: Training at Scale
One GPU cannot hold or train a large model, so real runs split the work across thousands of GPUs. You will hear four main strategies:
| Strategy | What is split | Simple picture |
|---|---|---|
| Data parallel | The batch of text | Every GPU holds a full model copy and trains on different text, then they average their gradients |
| Tensor parallel | Single big matrices | Each GPU computes a slice of every matrix multiplication |
| Pipeline parallel | The layers | GPU 1 holds layers 1 to 10, GPU 2 holds layers 11 to 20, and so on, like an assembly line |
| Sharding (ZeRO / FSDP) | Weights, gradients, optimizer states | Each GPU stores only a slice and fetches what it needs |
After pre-training: The result is a base model. It is excellent at continuing text, but it does not follow instructions yet. It will happily continue your question with more questions. Turning it into a helpful assistant is the job of fine-tuning, the next lesson.
- Visual Explanation: From Raw Text to a Base Model
Follow the journey from raw text to a trained model, then pick a model size to see the training budget suggested by the rules of thumb in this lesson.
- Python Implementation: A Tiny Pre-Training Run and a Compute Calculator
This script pre-trains the world's smallest language model on a 11-word text: a table of scores for which word follows which. It uses exactly the real recipe: predict the next token, measure cross-entropy, take a gradient descent step. Then it uses the 6ND rule to estimate the cost of real models.
import numpy as np text = "the cat sat on the mat the cat ate the rat"words = text.split()vocab = sorted(set(words))idx = {w: i for i, w in enumerate(vocab)}ids = np.array([idx[w] for w in words])V = len(vocab) inputs, targets = ids[:-1], ids[1:] # next-token prediction: from token t, predict token t+1rows = np.arange(len(targets)) rng = np.random.default_rng(0)W = rng.normal(scale=0.1, size=(V, V)) # logits table: row = current token, column = next token def softmax(z): z = z - z.max(axis=1, keepdims=True) e = np.exp(z) return e / e.sum(axis=1, keepdims=True) for step in range(201): probs = softmax(W[inputs]) # predicted distribution at every position loss = -np.mean(np.log(probs[rows, targets])) # cross-entropy on the true next token if step % 50 == 0: print("step", step, "| loss", round(float(loss), 3), "| perplexity", round(float(np.exp(loss)), 2)) grad = probs.copy() grad[rows, targets] -= 1 # gradient of the loss = probs - one_hot grad /= len(targets) np.add.at(W, inputs, -1.0 * grad) # gradient descent step, learning rate 1.0 probs = softmax(W[[idx["the"]]])[0]best = np.argsort(probs)[::-1][:2]print("After the word 'the', the model predicts:", [(vocab[i], round(float(probs[i]), 2)) for i in best]) # How much compute does real pre-training need?def training_cost(params, tokens, gpu_tflops=312, utilization=0.4): flops = 6 * params * tokens # rule of thumb: 6 FLOPs per parameter per token gpu_hours = flops / (gpu_tflops * 1e12 * utilization) / 3600 return flops, gpu_hours for name, n in [("1B", 1e9), ("7B", 7e9), ("70B", 70e9)]: tokens = 20 * n # Chinchilla rule of thumb: ~20 tokens per parameter flops, hours = training_cost(n, tokens) print(name, "| tokens", format(tokens, ".1e"), "| FLOPs", format(flops, ".2e"), "| A100 GPU-hours", format(round(hours), ","))The loss falls from about 1.95 (confused, perplexity of 7) to about 0.59, and after the word the the model correctly prefers cat, which followed the most often. Real pre-training is this same idea with trillions of tokens and billions of parameters. The last three lines show why it takes a cluster of GPUs.
Interactive Knowledge Check
Key Points
The Big Picture
Raw text → clean tokens → predict the next token → measure loss → update weights → repeat trillions of times
A base model is not yet an assistant, but it holds the knowledge that every later stage builds on. That is why pre-training uses the largest share of the compute budget.
Up Next: Your base model can continue text but does not follow instructions. In Fine-Tuning you will learn how SFT, LoRA and QLoRA turn it into a helpful assistant using a tiny fraction of the pre-training cost.