VRAM, Precision and Quantization

Learn how much GPU memory an LLM needs, what float16, bfloat16, int8 and int4 mean, why training needs far more memory than inference, and how quantization shrinks a model with only a small loss in quality.

28 minIntermediateCode Examples

The Core Idea: A model must fit in GPU memory. Memory = parameters × bytes per number, so using fewer bits (quantization) lets the same model fit on a much smaller GPU with only a small quality loss.

  1. Beginner: Why Memory Is the First Question

Before asking how smart is this model?, practitioners ask will it even fit on my GPU? A model is just billions of numbers (parameters), and every number must live in the GPU's fast memory, called VRAM, while the model runs.

memory for weights=number of parameters×bytes per parameter\text{memory for weights} = \text{number of parameters} \times \text{bytes per parameter}

So a model with 7 billion parameters needs 7 billion times the size of one number. How big one number is depends on its precision.

Analogy: Think of a photo album. Each photo is a parameter. A high-resolution print takes more shelf space than a thumbnail. Lowering precision is like storing thumbnails: you can fit many more photos, but each loses some detail.

  1. Beginner: Number Formats (Precision)

FormatBitsBytes per number7B model weightsTypical use
float32 (FP32)32428 GBOlder training, scientific work
float16 / bfloat1616214 GBStandard for training and inference
int8817 GBQuantized inference
int440.53.5 GBAggressive quantization, local models

bfloat16 keeps the same range as float32 (so training rarely overflows) but with less precision, which is why it became the favorite for deep learning. float16 has more precision but a smaller range.

Easy mental math: Weights in gigabytes ≈ parameters in billions × bytes per number. A 13B model in 16-bit is about 26 GB.

  1. Medium: What Else Uses Memory?

Weights are only part of the story. Running a model also needs:

ItemWhat it isGrows with
WeightsThe learned parametersModel size and precision
KV cacheStored keys and values of earlier tokensContext length and number of users
ActivationsTemporary results inside the layersBatch size
OverheadFramework buffers and fragmentationRoughly 10 to 20 percent extra

Training needs far more than inference. With the common Adam optimizer in mixed precision, each parameter needs about 16 bytes: 2 for the weights, 2 for the gradient, 4 for a 32-bit master copy and 8 for the two optimizer states. A 7B model therefore needs about 112 GB to train fully, versus 14 GB to run.

Rule of thumb for inference: Plan for weights × 1.2, plus the KV cache for your longest conversations.

  1. Medium: Quantization, Shrinking the Numbers

Quantization stores weights with fewer bits. Instead of a precise decimal like 0.7312, we keep a small integer and one shared scale number, then multiply to recover an approximation:

w≈q×sq=round(w/s)w \approx q \times s \qquad q = \text{round}(w / s)

1
Find the scale: Divide the largest absolute weight by the biggest integer the format can hold (127 for int8, 7 for int4).
2
Round: Convert every weight to the nearest integer.
3
Store: Save the small integers and the scale. This is the shrunken model.
4
Dequantize on the fly: During the calculation, multiply back by the scale.

Why it works: Neural network weights are noisy and redundant. A small rounding error in each weight usually changes the output very little, so quality drops only slightly.

  1. Advanced: Group-Wise Methods and the Quality Trade-Off

A single scale for a whole matrix is crude: one huge weight forces a coarse scale on everyone. Group-wise quantization uses a separate scale for each small group (for example 64 or 128 weights), which fits the values much better and is used by most 4-bit methods.

ApproachIdeaWhere you will see it
Post-training quantizationQuantize an already trained model with no retrainingGPTQ, AWQ, GGUF files
4-bit NormalFloat (NF4)A 4-bit format designed for bell-shaped weightsQLoRA fine-tuning
Quantization-aware trainingTrain while simulating the roundingBest quality, higher cost
KV-cache quantizationCompress the cache, not the weightsLong-context serving
The trade-off
8-bit is usually almost lossless. 4-bit is the common sweet spot: about 4 times smaller than 16-bit with a small quality drop. Below 4 bits quality falls faster, and tasks like math and code are often hit first. Always test on your task.

  1. Visual Explanation: Bytes, Bits and Rounding

The figure first compares one 7-billion-parameter model at three precisions, then shows the quantize and dequantize round trip for a single weight.

Part 1 · Same Model, Different PrecisionBytes per number decide the size
16-bit2 bytes each
7 billion parameters
↓
× 2 bytes
↓
About 14 GB
8-bit1 byte each
7 billion parameters
↓
× 1 byte
↓
About 7 GB
4-bit0.5 byte each
7 billion parameters
↓
× 0.5 byte
↓
About 3.5 GB
Part 2 · How Quantization WorksIntegers plus one scale
QuantizeSave time
Original weight: 0.73
↓
scale = 0.10
↓
q = round(0.73 / 0.10) = 7
↓
Stored: tiny integer 7 + scale
DequantizeUse time
Stored integer 7
↓
× scale 0.10
↓
Approximately 0.70 (small error)
Click to see why group-wise is betterCompare ▼
With one scale for everything, a single large weight forces a coarse step size on all weights. With one scale per 64 weights, each small group gets a step size that suits it, so the error drops. In the code below it falls by roughly a factor of 3.

Part 1 reads left to right: fewer bytes per number means a smaller model. Part 2 shows that quantizing stores a small integer and a scale, and using it multiplies back to an approximation. Open the bar to see why groups help.

  1. Python Implementation: Memory Math and Quantization From Scratch

This script calculates how much memory 7B and 70B models need at four precisions, quantizes a random weight matrix to int8 and int4, and compares one global scale with group-wise scales.

quantization_demo.pyPython 3.10+ · NumPy only
import numpy as np # 1. Memory needed just to store the weightsdef weights_gb(params_billion, bits):    return params_billion * 1e9 * bits / 8 / 1e9 for bits, name in [(32, "float32"), (16, "float16 / bfloat16"), (8, "int8"), (4, "int4")]:    print(name.ljust(18), "7B model:", round(weights_gb(7, bits), 1), "GB | 70B model:", round(weights_gb(70, bits), 1), "GB") # 2. Symmetric int8 quantization of a weight matrixrng = np.random.default_rng(0)W = rng.normal(size=(256, 256)).astype(np.float32) def quantize(W, bits):    qmax = 2 ** (bits - 1) - 1                          # 127 for int8, 7 for int4    scale = np.abs(W).max() / qmax    q = np.clip(np.round(W / scale), -qmax, qmax)    return q, scale def error_after(W, bits):    q, scale = quantize(W, bits)    return float(np.mean((W - q * scale) ** 2)) print("Mean squared error int8:", format(error_after(W, 8), ".2e"))print("Mean squared error int4:", format(error_after(W, 4), ".2e")) # 3. Group-wise quantization: one scale per small group of 64 weights fits betterdef error_groupwise(W, bits, group=64):    flat = W.reshape(-1, group)    qmax = 2 ** (bits - 1) - 1    scales = np.abs(flat).max(axis=1, keepdims=True) / qmax    q = np.clip(np.round(flat / scales), -qmax, qmax)    return float(np.mean((flat - q * scales) ** 2)) print("int4 with one scale for everything:", format(error_after(W, 4), ".2e"))print("int4 with one scale per 64 weights:", format(error_groupwise(W, 4), ".2e"))
EXPECTED OUTPUT (your numbers may differ slightly)
float32 7B model: 28.0 GB | 70B model: 280.0 GB
float16 / bfloat16 7B model: 14.0 GB | 70B model: 140.0 GB
int8 7B model: 7.0 GB | 70B model: 70.0 GB
int4 7B model: 3.5 GB | 70B model: 35.0 GB
Mean squared error int8: 1.16e-04
Mean squared error int4: 3.81e-02
int4 with one scale for everything: 3.81e-02
int4 with one scale per 64 weights: 1.17e-02

The first four lines are the memory table. Then the error numbers show int8 is about 300 times more accurate than int4 in this test, and that group-wise scales cut the int4 error to about a third.

Interactive Knowledge Check

Question 1: About how much memory do the weights of a 7B model need in 16-bit?
Question 2: Why is group-wise quantization usually more accurate than one scale for the whole matrix?

Key Points

✓Weight memory = parameters × bytes per parameter. A 7B model is about 14 GB in 16-bit.
✓bfloat16 keeps float32 range with less precision and is the deep-learning favorite.
✓Inference also needs the KV cache, activations and about 10 to 20 percent overhead.
✓Full training with Adam needs about 16 bytes per parameter, far more than inference.
✓Quantization stores small integers plus a scale. int8 is almost lossless, int4 is the common sweet spot.
✓Group-wise scales reduce quantization error. Always test the quantized model on your own task.

The Big Picture

Quantization in One Line

Fewer bits per number → smaller model → fits on smaller hardware → small quality cost

Knowing the memory math lets you choose a model and a GPU in minutes instead of by trial and error.

Up Next: Now that you can size a model, you need to talk to one. In Prompt Engineering you will learn how to write instructions that reliably get the answers you want.