knowledge distillation
/ NOL-ij dis-tih-LAY-shun /
A seasoned professor doesn't just tell you the right answer — they tell you which wrong answers are tempting and why, and that nuance teaches you more than the bare fact. Knowledge distillation borrows this idea: a large, accurate but heavy model (the "teacher") trains a smaller, faster model (the "student") not just on the right answers but on its full, nuanced opinions.
Here is the mechanism. A classifier doesn't only pick one answer; internally it assigns a probability to every option — maybe 80% "husky," 15% "wolf," 5% "cat." Those numbers, called soft targets, encode rich information: the teacher is saying huskies look a lot like wolves and nothing like cats. The student trains to match the teacher's full probability spread, not just the single top label. That extra signal lets a much smaller network learn to imitate the big one's behavior far better than training the small network on the original hard labels alone.
Distillation matters because the best models are often too big and slow to run on a phone or in real time. It compresses much of a large model's skill into something cheap to deploy. The honest limits: the student usually doesn't fully match the teacher — there is some accuracy loss — and it can only learn what the teacher knew, faithfully inheriting the teacher's blind spots and biases. It is a compression technique, not a way to create new capability; you can't distill a student smarter than its teacher.
A huge image classifier runs accurately but too slowly for a phone. Engineers train a small student network to copy the big model's soft probabilities — not just "dog" but "82% dog, 11% wolf, 5% fox." Those near-misses teach the student which classes resemble which, so it ends up far more accurate than a small network trained on plain labels, and small enough to run on-device.
The teacher's "soft" probabilities carry more than the bare right answer.
Distillation compresses existing skill, it doesn't create new ability: a student can't surpass its teacher, and it inherits the teacher's blind spots and biases along with its strengths.