LLM Pre-Training: Next-Token Prediction, Data, Scaling Laws and Compute

Learn how a random neural network becomes a language model: the next-token objective, how training data is collected and cleaned, loss and perplexity, scaling laws, the 6ND compute rule and how training is spread over thousands of GPUs.

30 minIntermediateCode Examples

The Core Idea: Pre-training teaches an LLM language with one simple game: guess the next token. Repeat it over trillions of tokens of text and the model absorbs grammar, facts and reasoning. The price is enormous compute, which follows a simple rule: 6 × parameters × tokens.

  1. Beginner: What Is Pre-Training?

A brand-new LLM is just a huge pile of random numbers. It cannot answer questions, write code or even finish a sentence. Pre-training is the long first stage where it learns the general patterns of language by reading an enormous amount of text.

The task it practices is incredibly simple: guess the next token. Show it The capital of France is and it should give a high score to Paris. Repeat this trillions of times on text from books, websites, articles and code, and the model slowly picks up grammar, facts, reasoning patterns and even coding style, all as a side effect.

InputA piece of text
The cat sat on the
TaskPredict the next token
Give every word in the vocabulary a probability.
FeedbackCompare with the real next token
Real next word: mat. Lower the loss if it gave mat a high probability.

Why this works: The text itself is the teacher. Every sentence on the internet is already a free exercise with its own answer, so no human has to label anything. This is called self-supervised learning.

  1. Beginner: The Data Recipe

Quality and variety of data matter as much as the model itself. A typical pre-training dataset is built in several steps:

1
Collect: Web pages, books, scientific papers, code repositories, encyclopedias and forums.
2
Clean: Remove menus, ads, broken text, unsafe content and personal data where possible.
3
Deduplicate: Remove repeated documents so the model does not just memorize them.
4
Filter for quality: Keep well-written text and drop spam or gibberish, often using a small classifier.
5
Tokenize and mix: Convert text to token ids with the tokenizer and blend sources in carefully chosen proportions.

Rule of thumb: Better data beats more data. Careful cleaning and de-duplication often improve a model more than simply adding another pile of raw text.

  1. Medium: The Loss, Perplexity and the Training Loop

The model gives a probability to every possible next token. The loss measures how surprised it was by the real one. This measure is called cross-entropy:

loss=−log⁡P(correct next token)\text{loss} = -\log P(\text{correct next token})

If the model gave the right word probability 1.0, the loss is 0. If it gave only 0.01, the loss is large (about 4.6). Perplexity is just elosse^{\text{loss}} and has a friendly meaning: the model is as confused as if it had to choose between that many equally likely words.

Probability given to the correct wordLossPerplexity
0.900.1051.11
0.500.6932.00
0.102.30310.0
0.014.605100

Each training step is the loop you already know from gradient descent: run a batch of text through the model, compute the loss, compute gradients with backpropagation, and update the weights with an optimizer (usually AdamW with a warm-up and a gradual cosine decay of the learning rate).

Good news: This is the same loop you coded in the Gradient Descent lesson. Pre-training simply runs it at a gigantic scale with a gigantic dataset.

  1. Medium: Scaling Laws, Compute and Cost

One of the most important discoveries about LLMs is that their loss improves in a smooth, predictable way as you increase three things together: the number of parameters (NN), the amount of training tokens (DD) and the compute. Researchers can therefore predict how good a big model will be by training small ones first.

Training compute is estimated with a famous rule of thumb: about 6 floating-point operations per parameter per token.

Compute (FLOPs)≈6×N×D\text{Compute (FLOPs)} \approx 6 \times N \times D

Chinchilla rule of thumb: For a given compute budget, a model trains best with roughly 20 tokens per parameter. A 7B model should see about 140 billion tokens. Many recent models deliberately train on far more than that, because a smaller model trained longer is cheaper to run for millions of users afterwards.

  1. Advanced: Training at Scale

One GPU cannot hold or train a large model, so real runs split the work across thousands of GPUs. You will hear four main strategies:

StrategyWhat is splitSimple picture
Data parallelThe batch of textEvery GPU holds a full model copy and trains on different text, then they average their gradients
Tensor parallelSingle big matricesEach GPU computes a slice of every matrix multiplication
Pipeline parallelThe layersGPU 1 holds layers 1 to 10, GPU 2 holds layers 11 to 20, and so on, like an assembly line
Sharding (ZeRO / FSDP)Weights, gradients, optimizer statesEach GPU stores only a slice and fetches what it needs
Stability tricks
Large runs also use mixed precision (16-bit numbers for speed, 32-bit for safety), gradient clipping (cap huge updates), checkpointing (save progress regularly) and constant monitoring of the loss curve, because a sudden loss spike can ruin days of work.

After pre-training: The result is a base model. It is excellent at continuing text, but it does not follow instructions yet. It will happily continue your question with more questions. Turning it into a helpful assistant is the job of fine-tuning, the next lesson.

  1. Visual Explanation: From Raw Text to a Base Model

Follow the journey from raw text to a trained model, then pick a model size to see the training budget suggested by the rules of thumb in this lesson.

💸 TRAINING BUDGET EXPLORER · CLICK A SIZEChinchilla rule of thumb, A100 GPUs

  1. Python Implementation: A Tiny Pre-Training Run and a Compute Calculator

This script pre-trains the world's smallest language model on a 11-word text: a table of scores for which word follows which. It uses exactly the real recipe: predict the next token, measure cross-entropy, take a gradient descent step. Then it uses the 6ND rule to estimate the cost of real models.

tiny_pretraining.pyPython 3.10+ · NumPy only
import numpy as np text = "the cat sat on the mat the cat ate the rat"words = text.split()vocab = sorted(set(words))idx = {w: i for i, w in enumerate(vocab)}ids = np.array([idx[w] for w in words])V = len(vocab) inputs, targets = ids[:-1], ids[1:]       # next-token prediction: from token t, predict token t+1rows = np.arange(len(targets)) rng = np.random.default_rng(0)W = rng.normal(scale=0.1, size=(V, V))    # logits table: row = current token, column = next token def softmax(z):    z = z - z.max(axis=1, keepdims=True)    e = np.exp(z)    return e / e.sum(axis=1, keepdims=True) for step in range(201):    probs = softmax(W[inputs])                         # predicted distribution at every position    loss = -np.mean(np.log(probs[rows, targets]))      # cross-entropy on the true next token    if step % 50 == 0:        print("step", step, "| loss", round(float(loss), 3), "| perplexity", round(float(np.exp(loss)), 2))    grad = probs.copy()    grad[rows, targets] -= 1                           # gradient of the loss = probs - one_hot    grad /= len(targets)    np.add.at(W, inputs, -1.0 * grad)                  # gradient descent step, learning rate 1.0 probs = softmax(W[[idx["the"]]])[0]best = np.argsort(probs)[::-1][:2]print("After the word 'the', the model predicts:", [(vocab[i], round(float(probs[i]), 2)) for i in best]) # How much compute does real pre-training need?def training_cost(params, tokens, gpu_tflops=312, utilization=0.4):    flops = 6 * params * tokens                        # rule of thumb: 6 FLOPs per parameter per token    gpu_hours = flops / (gpu_tflops * 1e12 * utilization) / 3600    return flops, gpu_hours for name, n in [("1B", 1e9), ("7B", 7e9), ("70B", 70e9)]:    tokens = 20 * n                                    # Chinchilla rule of thumb: ~20 tokens per parameter    flops, hours = training_cost(n, tokens)    print(name, "| tokens", format(tokens, ".1e"), "| FLOPs", format(flops, ".2e"), "| A100 GPU-hours", format(round(hours), ","))
EXPECTED OUTPUT (your numbers may differ slightly)
step 0 | loss 1.954 | perplexity 7.05
step 50 | loss 0.739 | perplexity 2.09
step 100 | loss 0.637 | perplexity 1.89
step 150 | loss 0.606 | perplexity 1.83
step 200 | loss 0.592 | perplexity 1.81
After the word 'the', the model predicts: [('cat', 0.49), ('rat', 0.24)]
1B | tokens 2.0e+10 | FLOPs 1.20e+20 | A100 GPU-hours 267
7B | tokens 1.4e+11 | FLOPs 5.88e+21 | A100 GPU-hours 13,088
70B | tokens 1.4e+12 | FLOPs 5.88e+23 | A100 GPU-hours 1,308,761

The loss falls from about 1.95 (confused, perplexity of 7) to about 0.59, and after the word the the model correctly prefers cat, which followed the most often. Real pre-training is this same idea with trillions of tokens and billions of parameters. The last three lines show why it takes a cluster of GPUs.

Interactive Knowledge Check

Question 1: What is the training task during LLM pre-training?
Question 2: According to the 6ND rule, what happens to training compute if you double the number of tokens?

Key Points

✓Pre-training = predicting the next token on huge amounts of raw text (self-supervised, no manual labels).
✓Data quality, cleaning and de-duplication matter as much as data quantity.
✓The loss is cross-entropy: it is large when the model gave the correct next token a low probability. Perplexity = e^loss.
✓Training is the same loop as gradient descent: forward pass, loss, backpropagation, AdamW update.
✓Scaling laws let researchers predict quality from model size, data and compute. Compute ≈ 6 × N × D, and about 20 tokens per parameter is compute-optimal.
✓Big runs use data, tensor and pipeline parallelism plus sharding. The result is a base model that still needs fine-tuning to follow instructions.

The Big Picture

Pre-Training in One Line

Raw text → clean tokens → predict the next token → measure loss → update weights → repeat trillions of times

A base model is not yet an assistant, but it holds the knowledge that every later stage builds on. That is why pre-training uses the largest share of the compute budget.

Up Next: Your base model can continue text but does not follow instructions. In Fine-Tuning you will learn how SFT, LoRA and QLoRA turn it into a helpful assistant using a tiny fraction of the pre-training cost.