Fine-Tuning LLMs: SFT, LoRA and QLoRA

Learn how a pretrained base model becomes a helpful specialist: supervised fine-tuning with instruction data, why full fine-tuning is costly, how LoRA and QLoRA train tiny adapters, and when to fine-tune instead of prompting or using RAG.

30 minIntermediateCode Examples

The Core Idea: A base model knows a lot but does not behave like an assistant. Fine-tuning teaches it a behavior with a small, high-quality dataset. LoRA makes this cheap by freezing the model and training only tiny adapter matrices.

  1. Beginner: Why Fine-Tune a Model That Already Knows So Much?

After pre-training, a base model knows a huge amount about language and the world, but it behaves like a very well-read autocomplete. Ask it What is the capital of France? and it might continue with What is the capital of Germany? because that is what often follows a question in its training text.

Fine-tuning continues training the model on a much smaller, carefully chosen dataset so that it learns a behavior: follow instructions, answer politely, use a company's writing style, speak medical language, or output clean JSON.

Pre-trainingLearn the world
Data: Trillions of raw tokens
Goal: General knowledge and language skill
Cost: Millions of GPU-hours
Fine-tuningLearn a behavior
Data: Thousands to hundreds of thousands of good examples
Goal: A specific style, task or format
Cost: A few GPU-hours to a few days

Analogy: Pre-training is like going through school: broad general knowledge. Fine-tuning is the job training: learning how to apply that knowledge in one specific role.

  1. Beginner: Supervised Fine-Tuning (SFT) and Instruction Data

The most common first step is supervised fine-tuning (SFT), also called instruction tuning. The training data is a set of examples written as instruction and ideal answer pairs:

Instruction (prompt)Ideal response (target)
Explain gradient descent to a 10-year-old.Imagine walking down a foggy hill by feeling the slope under your feet...
Convert this to JSON: name Ana, age 31a JSON object with the name and age fields
Summarize this email in one sentence.A one-sentence summary written in a clear and polite tone.

The training loop is the same next-token loss as pre-training, but usually computed only on the response part. The model is not rewarded for repeating the question, only for producing a good answer.

Quality over quantity: A small set of high-quality, varied examples often beats a huge noisy set. Many successful assistants were tuned with only thousands of carefully written examples.

  1. Medium: Full Fine-Tuning and Its Problem

The simplest approach is full fine-tuning: update every weight of the model. It is powerful, but it has serious costs:

ProblemWhat it means for a 7B model
MemoryWeights, gradients and optimizer states together need roughly 16 bytes per parameter, so about 112 GB for 7 billion parameters. Far beyond a normal GPU.
StorageEvery fine-tuned version is a full copy of about 14 GB (16-bit weights).
ForgettingChanging all weights can erase some of the original skills. This is called catastrophic forgetting.

The big question: Do we really need to change all 7 billion numbers to teach a new style? Research says no. The needed change is usually simple, and that is what LoRA exploits.

  1. Medium: LoRA (Low-Rank Adaptation)

LoRA keeps the original weight matrix WW frozen and learns only a small correction next to it. The correction is the product of two thin matrices, BB and AA:

Wnew=W+αr BAW_{new} = W + \frac{\alpha}{r} \, B A

If WW is a 4096×40964096 \times 4096 table, then AA is r×4096r \times 4096 and BB is 4096×r4096 \times r, where the rank rr is tiny, such as 8. So instead of training 16,777,21616{,}777{,}216 numbers we train only 2×4096×8=65,5362 \times 4096 \times 8 = 65{,}536, about 0.39%.

1
Freeze the base model: No original weight changes, so the original knowledge is protected.
2
Add small adapters: Insert AA and BB next to chosen layers, usually the attention matrices.
3
Start safe: BB starts as all zeros, so the model begins exactly as the pretrained one.
4
Train only AA and BB: Gradients flow into the adapters only, so memory and time drop sharply.
5
Ship a tiny file: The adapter is a few megabytes. You can load, swap or merge it into WW whenever you like.

Why it works: Fine-tuning changes are usually low-rank: they live in a small number of directions. A thin pair of matrices is enough to capture them, so quality stays close to full fine-tuning.

  1. Advanced: QLoRA, Forgetting and Choosing the Right Tool

QLoRA goes one step further. It stores the frozen base model in 4-bit numbers (about 0.5 byte per parameter, so a 7B model shrinks from about 14 GB to about 3.5 GB) and trains 16-bit LoRA adapters on top. This lets one consumer GPU fine-tune models that would otherwise need a data-center.

MethodTrainsRough memory for a 7B modelBest when
Prompting / RAGNothingJust the model for inferenceKnowledge changes often, or you only need a better prompt
Full fine-tuningAll weightsAbout 112 GBYou have a big budget and need maximum change
LoRASmall adaptersAbout 14 GB for the 16-bit base, plus a littleStyle, format or domain adaptation on one good GPU
QLoRASmall adapters on a 4-bit baseAbout 4 to 6 GB in practiceLimited hardware, such as a single consumer GPU
Fine-tune, prompt or retrieve?
Use prompting first, it is free. Use RAG when the model needs fresh or private facts. Use fine-tuning to change behavior: tone, format, domain language or a skill. Fine-tuning is poor at teaching many new facts.

Next stage: alignment. After SFT, many assistants are refined with preference data, where humans or AI judges say which of two answers is better. Methods such as RLHF and DPO use this to make answers more helpful and safe. That is a topic of its own.

  1. Visual Explanation: The Fine-Tuning Pipeline and the Method Explorer

This is the path from a base model to a specialist assistant, followed by a quick explorer that compares the four tools from this lesson for a 7B model.

🧰 METHOD EXPLORER · CLICK A METHOD7B model, rough numbers

  1. Python Implementation: Training a LoRA Adapter From Scratch

This script shows LoRA working on a small scale. The big weight matrix W is frozen. The new skill is a low-rank change, and we recover it by training only the thin matrices A and B, which hold just a quarter of the parameters of W in this tiny example and less than half a percent in a real layer.

lora_from_scratch.pyPython 3.10+ · NumPy only
import numpy as np rng = np.random.default_rng(0)d, r, n = 16, 2, 200 W = rng.normal(size=(d, d))                              # frozen pretrained weightsdelta = rng.normal(size=(d, r)) @ rng.normal(size=(r, d)) / d    # the new skill is a low-rank changeX = rng.normal(size=(n, d))Y = X @ (W + delta).T                                    # what the fine-tuned model should output A = rng.normal(size=(r, d)) * 0.1                        # LoRA matrices: the ONLY things we trainB = np.zeros((d, r))                                     # B starts at zero, so we begin exactly at the pretrained modellr = 1.0 def current_loss():    return np.mean((X @ (W + B @ A).T - Y) ** 2) print("Trainable parameters: LoRA =", A.size + B.size, "| full fine-tuning =", W.size)print("Loss before training:", round(float(current_loss()), 5)) for step in range(601):    E = X @ (W + B @ A).T - Y    G = (2 / (n * d)) * E.T @ X                          # gradient with respect to the combined update B @ A    dB, dA = G @ A.T, B.T @ G    B -= lr * dB    A -= lr * dA    if step in (100, 300, 600):        print("step", step, "| loss", format(current_loss(), ".2e")) # Same idea at real scale: one 4096 x 4096 layer with rank 8full, lora = 4096 * 4096, 2 * 4096 * 8print("One real layer: full =", format(full, ","), "| LoRA =", format(lora, ","), "|", round(100 * lora / full, 2), "% of the size")
EXPECTED OUTPUT (your numbers may differ slightly)
Trainable parameters: LoRA = 64 | full fine-tuning = 256
Loss before training: 0.15784
step 100 | loss 4.47e-03
step 300 | loss 3.50e-11
step 600 | loss 1.05e-22
One real layer: full = 16,777,216 | LoRA = 65,536 | 0.39 % of the size

The loss before training is about 0.16 (the model lacks the new skill). It collapses toward zero within a few hundred steps even though only 64 of 256 numbers were trained. In a real 4096 × 4096 layer the same trick trains only 0.39% of the numbers.

Interactive Knowledge Check

Question 1: In LoRA, which weights are trained?
Question 2: You need the model to know your company's latest prices, which change weekly. What is the best first choice?

Key Points

✓Fine-tuning teaches behavior (style, format, skill). It is not an efficient way to teach many new facts.
✓SFT trains on instruction and ideal-answer pairs, with the same next-token loss, usually only on the answer.
✓Full fine-tuning needs about 16 bytes per parameter, so about 112 GB for a 7B model, and risks catastrophic forgetting.
✓LoRA freezes W and learns W + (α/r)·B·A with tiny matrices, training well under 1% of the parameters.
✓QLoRA stores the frozen base in 4-bit so fine-tuning fits on a single consumer GPU.
✓Try prompting first, use RAG for fresh or private facts, and fine-tune to change behavior.

The Big Picture

Fine-Tuning in One Line

Base model + a few good examples + small adapters (LoRA) = a specialist assistant at a fraction of the cost

You now understand the full journey of an LLM: how attention and positions work, how a modern block is built, how pre-training creates a base model, and how fine-tuning turns it into something useful. The same ideas appear in every model you will use.

Up Next: Alignment methods such as RLHF and DPO, plus prompting techniques, build on everything in this series. They are the next topics in Prompting and Alignment.