Text Generation and Sampling: Temperature, Top-k and Top-p

Learn how an LLM turns probabilities into text: greedy decoding, temperature, top-k and top-p sampling, repetition penalties, seeds and stop sequences, with every method built from scratch in code.

25 minBeginnerCode Examples

The Core Idea: An LLM only produces probabilities for the next token. A sampling method chooses one. Temperature, top-k and top-p let you trade predictability for variety without retraining anything.

  1. Beginner: The Model Only Gives Probabilities

An LLM does not produce a sentence directly. At every step it produces a score for every word in its vocabulary, which is turned into a probability distribution. Something then has to pick one word. That choice rule is called a decoding strategy or sampling method.

The same model can sound careful, boring, creative or chaotic depending only on this rule. No retraining is needed.

Step 1Scores (logits)
The model scores every possible next token.
Step 2Probabilities
Softmax turns the scores into numbers that add up to 1.
Step 3Pick one
A rule chooses a token. It is added to the text and the loop repeats.

  1. Beginner: Greedy Decoding and Why It Is Not Enough

The simplest rule is greedy decoding: always take the most likely token. It is fast and gives the same answer every time, which is great for facts and code.

But greedy has two problems. It can get stuck in repetitive loops, and it makes writing feel flat because it never takes a less obvious but still good word. Sampling fixes this by choosing randomly according to the probabilities, so likely words win most of the time but not always.

Beam search: Another classic method keeps the best few partial sentences at once. It works well for translation, but for open-ended chat it tends to produce dull text, so chat models usually sample instead.

  1. Medium: Temperature, the Creativity Dial

Temperature TT divides the scores before the softmax. It reshapes the probabilities without changing their order:

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

TemperatureEffectGood for
Close to 0The top token gets almost all the probability. Behaves like greedy.Facts, extraction, code, math
Around 0.7Mostly the top choices, with some varietyGeneral chat and writing
1.0The model's natural distributionBrainstorming
Above 1.5Rare tokens become likely. Text may turn into nonsense.Rarely useful

Remember: Low temperature does not make the model smarter. It only makes it more predictable.

  1. Medium: Top-k and Top-p, Cutting Off the Bad Tail

Even at a sensible temperature, a vocabulary of 100,000 tokens has a long tail of silly words. Each one has a tiny probability, but together they can add up and occasionally derail a sentence. Two filters remove the tail:

Top-kFixed size list
Rule: Keep only the kk most likely tokens (for example 40), then renormalize.
Weakness: Same kk even when the model is very sure or very unsure.
Top-p (nucleus)Adaptive size
Rule: Keep the smallest group of tokens whose probabilities add up to pp (for example 0.9).
Strength: Few tokens when the model is confident, more when it is unsure.

Practical default: Many applications use temperature around 0.7 with top-p around 0.9 to 0.95. Change one knob at a time so you can see what it does.

  1. Advanced: Repetition, Seeds and Stopping

Real systems add a few more controls around the main sampler:

ControlWhat it does
Repetition / frequency penaltyLowers the score of tokens already used, to avoid loops
SeedFixes the random choices so a sampled answer can be repeated (not always guaranteed across hardware)
Stop sequencesEnds generation when a chosen string appears, such as a closing tag
Max tokensA hard limit on the length of the answer
Min-p and similarNewer filters that keep tokens above a fraction of the top probability
Why answers still differ at temperature 0
Tiny numerical differences on the GPU, or ties between tokens, can occasionally change the winner. Temperature 0 is nearly deterministic, not a guarantee.

  1. Visual Explanation: How a Word Gets Chosen

The figure follows one next-word decision using the numbers from the code below: five candidate words with scores that become probabilities.

Part 1 · From Scores to a Chosen WordSame model, different rules
GreedyTemperature ≈ 0
Probabilities: the 0.47, a 0.29, cat 0.17
↓
Always take the highest
↓
Picks: the (every time)
Temperature 1Sampling
Probabilities: the 0.47, a 0.29, cat 0.17
↓
Draw a random token by probability
↓
Mostly the, sometimes a or cat
Top-p 0.9Sampling + cut
Drop the unlikely tail (dog, banana)
↓
Renormalize what is left, then draw
↓
Safe variety only
Part 2 · What Each Dial ChangesTune one at a time
Temperature
Sharper or flatter
Changes how spread out the probabilities are.
Top-k
Fixed cut
Keeps exactly the k best tokens.
Top-p
Adaptive cut
Keeps the smallest group reaching probability p.
Click to see a quick rule of thumbCompare ▼
Need exact, repeatable output (code, JSON, facts)? Use temperature near 0. Need natural variety (writing, ideas)? Use temperature 0.7 to 1 with top-p 0.9 to 0.95. Seeing weird rare words? Lower top-p or temperature.

In Part 1, every card starts from the same probabilities and applies a different rule. Part 2 summarizes what each dial changes. Open the bar for a rule of thumb.

  1. Python Implementation: Temperature, Top-k and Top-p

This script builds each sampling method from scratch on a five-word example, then draws 1,000 samples to show how the choices spread out.

sampling_methods.pyPython 3.10+ · NumPy only
import numpy as np words = ["the", "a", "cat", "dog", "banana"]logits = np.array([3.0, 2.5, 2.0, 1.0, -1.0])    # raw scores from a model def softmax(z, temperature=1.0):    z = z / temperature    z = z - z.max()    e = np.exp(z)    return e / e.sum() def show(title, p):    print(title.ljust(22), {w: round(float(x), 3) for w, x in zip(words, p)}) # 1. Temperature reshapes the probabilitiesfor T in [0.5, 1.0, 2.0]:    show("temperature " + str(T), softmax(logits, T)) # 2. Top-k: keep only the k most likely words, then renormalizedef top_k(p, k):    keep = np.argsort(p)[::-1][:k]    out = np.zeros_like(p)    out[keep] = p[keep]    return out / out.sum() show("top-k (k=2)", top_k(softmax(logits), 2)) # 3. Top-p (nucleus): keep the smallest set whose probabilities add up to pdef top_p(p, threshold):    order = np.argsort(p)[::-1]    cumulative = np.cumsum(p[order])    cutoff = np.searchsorted(cumulative, threshold) + 1    out = np.zeros_like(p)    out[order[:cutoff]] = p[order[:cutoff]]    return out / out.sum() show("top-p (p=0.9)", top_p(softmax(logits), 0.9)) # 4. Greedy vs sampling over 1,000 drawsrng = np.random.default_rng(0)p = softmax(logits)draws = rng.choice(words, size=1000, p=p)print("Greedy always picks:", words[int(np.argmax(p))])print("Sampling 1,000 times:", {w: int((draws == w).sum()) for w in words})
EXPECTED OUTPUT (your numbers may differ slightly)
temperature 0.5 {'the': 0.657, 'a': 0.242, 'cat': 0.089, 'dog': 0.012, 'banana': 0.0}
temperature 1.0 {'the': 0.47, 'a': 0.285, 'cat': 0.173, 'dog': 0.064, 'banana': 0.009}
temperature 2.0 {'the': 0.346, 'a': 0.27, 'cat': 0.21, 'dog': 0.127, 'banana': 0.047}
top-k (k=2) {'the': 0.622, 'a': 0.378, 'cat': 0.0, 'dog': 0.0, 'banana': 0.0}
top-p (p=0.9) {'the': 0.506, 'a': 0.307, 'cat': 0.186, 'dog': 0.0, 'banana': 0.0}
Greedy always picks: the
Sampling 1,000 times: {'the': 440, 'a': 297, 'cat': 192, 'dog': 61, 'banana': 10}

Notice how a low temperature piles probability on the, while a high temperature spreads it out. Top-k with k=2k=2 keeps just two words, while top-p keeps three because it needs three to reach 0.9. In the 1,000 draws, the wins about 44% of the time but not always.

Interactive Knowledge Check

Question 1: What does lowering the temperature do?
Question 2: What is the main advantage of top-p over top-k?

Key Points

✓The model outputs a probability for every token. A decoding rule picks one.
✓Greedy always takes the top token: repeatable but can loop and sounds flat.
✓Temperature reshapes probabilities: near 0 is predictable, higher is more varied.
✓Top-k keeps a fixed number of best tokens. Top-p keeps the smallest set reaching probability p.
✓Repetition penalties, stop sequences and max tokens give extra control.
✓Use low temperature for exact tasks and moderate temperature with top-p for writing.

The Big Picture

Sampling in One Line

Scores → probabilities → (temperature, top-k, top-p) → pick one token → repeat

Two users can run the same model and get very different feelings simply by changing these dials. Knowing them lets you debug strange or repetitive output quickly.

Up Next: Every token you generate, and every token in your prompt, has to fit inside a limit. In Context Windows and the KV Cache you will learn how to budget that space and keep long conversations affordable.