← All tutorials
Knowledge Distillation

Knowledge distillation

A big model is accurate but expensive — too slow, too heavy, too costly to run at scale or on a phone. A small model is cheap but, trained on its own, not as smart. Knowledge distillation bridges the two: it trains the small model to imitate the big one, transferring not just the answers but the rich, graded judgement behind them. The student ends up far better than an identical model trained on the raw labels alone.

The one-line idea. Train a small student to match the full output distribution of a large teacher, not just the ground-truth label. The teacher's "I'm 80% sure it's a dog, 15% a wolf, 5% a cat" carries more information per example than a bare "dog" — and that extra signal is what a small network needs to punch above its weight.

Foundations  The core idea

Introduced by Hinton, Vinyals & Dean in 2015 (Distilling the Knowledge in a Neural Network), distillation reframes model compression as a teaching problem. You already have a large, accurate model — an ensemble, or a heavily over-parameterised network. It's a great predictor but impractical to deploy. Rather than throw its knowledge away, you use it as a supervisor for a compact model you can deploy.

Distilling a large teacher model into a small student model Input data flows into a large frozen teacher network and a small student network. The teacher emits soft probabilities; the student is trained to match both the soft teacher targets and the hard ground-truth labels via a combined loss. transfer set(inputs) Teacher (large)frozen · pre-trained Studentsmall · trainable soft targets student probs Loss α · distill (soft) + (1−α) · CE (hard) KL( teacher ‖ student ) at temperature T back-prop updates student only hard labels

The magic is in what Hinton called dark knowledge. A hard label says only "this is a 7." The teacher's soft output says "this is a 7, but it's a little like a 1 and nothing like an 8." Those relative probabilities across the wrong classes encode how the teacher sees the structure of the problem — which things resemble which. Training the student to reproduce that whole distribution hands it a far richer target than a one-hot label ever could.

Why the student can win. A small model trained from scratch on hard labels has to rediscover class similarities from limited data. Distillation gives it those similarities for free, on every example — so the same architecture reaches accuracy it could not reach alone. Distillation is one of three ways to adapt a model; here the lever is compression without collapse.

The supervisor  The teacher model

The teacher is a large, already-trained, high-accuracy model whose job is to generate supervision — not to be deployed. It's usually frozen: you run it in inference mode over your data and read off its outputs.

🎓 What makes a good teacher

Wants High accuracy on the task, and well-calibrated probabilities — its confidence should mean something. A teacher that outputs richer, better-structured distributions transfers more knowledge.

Watch A teacher far larger than the student can transfer worse, not better — the capacity gap. The student can't represent everything the teacher knows, and the too-hard target hurts. A mid-size teacher assistant in between can bridge the gap.

Teachers take many forms: a single big network, an ensemble of models whose averaged predictions you distill into one (Hinton's original motivation), or — in the LLM world — a frontier model like GPT-4 or Claude used to label or generate data for a smaller open model.

The teacher sets the ceiling — and the floor. The student inherits the teacher's biases and blind spots along with its skill; distillation can't exceed what the teacher knows on a given example. A miscalibrated or overconfident teacher passes bad targets straight through. Evaluate the teacher on your real distribution before you trust it as a supervisor — see LLM evaluation.

The apprentice  The student model

The student is the compact model you actually ship. It's smaller and faster by design — fewer layers, narrower width, sometimes a different (cheaper) architecture entirely. Its goal is to recover as much of the teacher's behaviour as possible within a fraction of the compute.

Fewer parameters

Smaller memory

Fits in less RAM/VRAM — on a phone, a browser, an edge device, or a cheaper server instance.

Lower latency

Faster inference

Fewer FLOPs per prediction → lower response time and higher throughput at the same cost.

Cheaper to run

Less energy & $

The recurring win: inference is paid on every request, forever. A 2× smaller student halves that bill.

The student needs enough capacity to absorb the useful signal — too tiny and it underfits no matter how good the teacher. The classic result: a distilled student consistently beats an identical-size model trained directly on the labels, because the soft targets are a better teacher than one-hot vectors.

Distillation vs the other compression tools. Distillation is one of several ways to shrink a model, and they compose: quantization (lower-precision weights), pruning (remove unimportant weights), and distillation stack together. Distillation is unique in changing what the small model learns, not just how its existing weights are stored.

The mechanism  Soft targets & temperature

A network's final layer produces logits — raw scores per class. A normal softmax turns them into probabilities, but a confident model's softmax is nearly one-hot: 0.99 for the winner, ~0 for everyone else. That flattens away the dark knowledge. The fix is a temperature T in the softmax:

# Softmax with temperature T. Higher T → softer, more spread-out distribution.
q_i = exp(z_i / T) / Σ_j exp(z_j / T)

# T = 1  → normal softmax        e.g. [0.95, 0.04, 0.01]  (spiky, little signal)
# T = 4  → softened distribution e.g. [0.60, 0.28, 0.12]  (reveals class structure)
# T → ∞  → uniform                     [0.33, 0.33, 0.33]  (all signal gone)

Raising T (typically 2–10) inflates the small probabilities so the student can actually see and learn the ratios between wrong classes. Crucially you apply the same T to both teacher and student when computing the distillation loss, and drop back to T = 1 at inference.

Temperature is a real knob. Too low and the targets are nearly one-hot — you lose the dark knowledge distillation exists to capture. Too high and everything blurs toward uniform, drowning the true class in noise. It's a hyperparameter to tune per task; 3–5 is a common starting range. And because soft-target gradients scale by 1/T², the loss carries a compensating T² factor (next section).

The objective  The distillation loss

The student is usually trained on two signals at once, blended by a weight α:

TermWhat it matchesWhy it's there
Distillation loss (soft)Student soft probs ↔ teacher soft probs, at temperature TTransfers the dark knowledge — the whole distribution. Usually KL divergence.
Student loss (hard)Student probs ↔ ground-truth labels, at T = 1Keeps the student anchored to the true answer where labels exist. Standard cross-entropy.
# Combined objective (Hinton et al., 2015)
L = α · T² · KL( softmax(teacher/T) ‖ softmax(student/T) )   # soft target
  + (1 − α) · CrossEntropy( labels, softmax(student) )        # hard target

# α  — weight on the teacher's soft targets (often 0.5–0.9)
# T² — rescales the soft-target gradients, which otherwise shrink by 1/T²

The T² multiplier keeps the two gradient magnitudes comparable so one term doesn't vanish as you raise the temperature. When you have no ground-truth labels for the transfer set (common when the teacher generates data), you can set α = 1 and rely on the soft targets alone.

KL vs MSE. Response-based distillation typically minimises KL divergence between the two softened distributions (equivalently, cross-entropy against soft targets). Some variants regress directly on the raw logits with mean-squared error — simpler, and it skips the temperature entirely by keeping the pre-softmax scores.

What gets transferred  Kinds of knowledge

Matching final probabilities is only the most common form. The survey literature (Gou et al., 2021) groups distillation by which part of the teacher the student imitates:

🎯 Response-based — match the outputs

The student mimics the teacher's final layer (soft logits/probabilities). This is Hinton's original recipe: simple, architecture-agnostic, and still the default. It only needs the teacher's outputs, so it works even when you can't see inside the teacher.

🧱 Feature-based — match the hidden layers

The student also matches the teacher's intermediate activations ("hints"), as in FitNets. Richer signal — the student learns how the teacher represents things, not just its conclusions — but it needs access to internals and a way (a projection) to align mismatched layer shapes.

🔗 Relation-based — match the relationships

Instead of individual values, the student reproduces relationships — between layers (e.g. Gram matrices of feature flow) or between examples (which inputs the teacher considers similar). Captures structure the other two miss.

Rule of thumb. Start response-based — it's the simplest and needs the least access. Reach for feature- or relation-based knowledge when there's a big capacity gap and the soft labels alone aren't enough, and you can open up the teacher. They add complexity (layer alignment, extra loss terms) for extra accuracy.

Who trains when  Training schemes

Distillation also varies by when teacher and student are trained relative to each other:

SchemeHowWhen to use
OfflineTeacher is pre-trained & frozen; only the student trains against its outputsThe default — simplest, and you already have a strong teacher
OnlineTeacher & student train together, learning from each other (e.g. deep mutual learning)No powerful pre-trained teacher available; several peers co-train
Self-distillationA model teaches itself — deeper layers → shallower, or a past checkpoint → current (born-again nets)Squeeze extra accuracy from one architecture with no separate teacher
Self-distillation surprises people. Training a network, then training a fresh copy of the same architecture to imitate it, often yields a better model than the original — "born-again networks." The soft targets regularise learning even without any size reduction.

Today  Distillation in the LLM era

Distillation is how many small, fast, open models get their capability. The mechanics shift a bit for generative models:

🔤 Sequence-level distillation

For text generation there's a token distribution at every step. Sequence-level KD (Kim & Rush) sidesteps the per-token bookkeeping: let the teacher generate outputs, then train the student on those teacher-written sequences. Simple, effective, and it only needs the teacher's text.

📦 White-box vs black-box

White-box You have the teacher's weights/logits — full response- and feature-based distillation (how DistilBERT was made: 40% smaller, ~60% faster, ~97% of BERT's performance).

Black-box You only see the teacher's text (an API). "Distillation" here means generating a synthetic training set from the teacher and fine-tuning the student on it — how many instruction-tuned open models were bootstrapped.

🧠 Reasoning (chain-of-thought) distillation

Have a strong teacher produce step-by-step reasoning traces, then train a small student to reproduce the reasoning, not just the final answer. A compact model can inherit a surprising amount of the teacher's problem-solving — the idea powers many small "reasoning" models.

Black-box distillation has strings attached. Generating training data from a commercial model may violate its terms of service (several forbid using outputs to train competing models), and the student inherits the teacher's errors and hallucinations as ground truth. Check the licence, and evaluate the student independently — a fluent imitator can be confidently wrong. See how these models work for what's actually being copied.

The trade-off  When it pays off — and its limits

Distillation earns its keep when inference cost is the constraint and you already have (or can access) a strong teacher:

  • Deployment targets — a phone, browser, or edge device that can't run the big model at all.
  • Latency & cost at scale — inference is paid per request forever; a smaller student compounds the savings.
  • Ensemble collapse — fold an accurate but unwieldy ensemble into one deployable network.
  • Capability transfer — give a small open model a slice of a frontier model's skill via generated data.
It is not free, and not magic. You still need the teacher (compute to train or API budget to query) and a good transfer set — distillation moves knowledge, it doesn't create it, and coverage gaps in the transfer data become gaps in the student. Expect some accuracy loss versus the teacher; the win is the accuracy-per-parameter ratio, not parity. And a huge teacher/tiny student gap can backfire (§2). Measure the student against the teacher on your distribution before shipping.

End to end  Worked example: distilling an image classifier

You have an accurate but heavy classifier (the teacher) and need something that runs on-device. Train a small CNN (the student) with a combined soft + hard loss. Here's the heart of it in PyTorch — see the PyTorch tutorial for the surrounding training loop.

import torch, torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.9):
    # Soft targets: match the teacher's softened distribution (KL divergence).
    soft_teacher = F.softmax(teacher_logits / T, dim=1)
    soft_student = F.log_softmax(student_logits / T, dim=1)
    soft_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (T * T)

    # Hard target: match the true labels at T = 1.
    hard_loss = F.cross_entropy(student_logits, labels)

    return alpha * soft_loss + (1 - alpha) * hard_loss

# --- training step ---
teacher.eval()                                   # frozen supervisor
with torch.no_grad():
    teacher_logits = teacher(x)                  # no gradients through the teacher

student_logits = student(x)                      # the model we ship
loss = distillation_loss(student_logits, teacher_logits, y, T=4.0, alpha=0.9)
loss.backward()                                  # updates the student only
optimizer.step()
DecisionChoiceWhy
TeacherFrozen, eval(), no_gradIt only supervises; no gradients flow into it
Temperature T4 (tune 2–10)Softens outputs enough to expose class structure
α (soft weight)0.9Lean on the teacher; keep a little hard-label anchoring
Soft lossKL × T²The T² rescales gradients shrunk by 1/T²
InferenceT = 1, student onlyTemperature is a training-time device; ship the small net alone
Then stack the other tools. After distillation, quantize and optionally prune the student for another cut in size and latency — the three techniques compose. Distillation gave the small model good weights; quantization and pruning make those weights cheaper to store and run.

Cheat sheet

QuestionShort answer
What is knowledge distillation?Training a small student to imitate a large teacher's outputs, not just the labels.
Teacher vs student?Teacher = big, accurate, frozen supervisor. Student = small, fast, the model you ship.
What's "dark knowledge"?The relative probabilities over wrong classes — how the teacher sees similarity.
What's temperature for?Softens the softmax so small probabilities (the dark knowledge) become learnable.
The loss?α·T²·KL(soft teacher ‖ soft student) + (1−α)·CE(hard labels).
Why the T²?Soft-target gradients scale by 1/T²; T² keeps the terms balanced.
Kinds of knowledge?Response (outputs), feature (hidden layers), relation (relationships).
Training schemes?Offline (frozen teacher), online (co-train), self (teach itself).
In LLMs?Sequence-level KD, white-box vs black-box (synthetic data), reasoning distillation.
When NOT to?No good teacher, no transfer data, or the teacher↔student gap is too large.

Knowledge check

Six questions. Pick an answer for each, then check your score.

1. What is the core idea of knowledge distillation?

2. What is "dark knowledge"?

3. Why raise the softmax temperature during distillation?

4. In the standard loss, what are the two terms?

5. Which is a genuine risk when distilling?

6. What is black-box distillation of an LLM?

Next steps

🎚️ Fine-tuning vs RAG vs Prompting

The adaptation levers — where distillation sits among ways to shape a model.

🧠 How LLMs & Transformers Work

What's being copied — logits, softmax and the forward pass distillation matches.

🔥 PyTorch

Build the loop — autograd and the training step around the distillation loss.

🧪 LLM Evaluation

Prove the student — measure it against the teacher before you ship.