model distillation
/ MOD-ul dis-tih-LAY-shun /
Model distillation is a way to compress a big, expensive model into a small, cheap one that behaves almost as well — by having the small model learn to imitate the big one. The large model is the "teacher" and the small one the "student": you feed both the same inputs, and you train the student not just to match the final answers but to mimic the teacher's full pattern of responses. The student ends up a compact echo of its teacher, running far faster and cheaper.
The reason it works better than just training a small model from scratch is subtle and rather elegant. The teacher's outputs carry more information than a plain right-or-wrong label: when shown a photo of a dog, the teacher might say 90% dog, 7% wolf, 3% cat — and those "soft" relative scores quietly teach the student that dogs resemble wolves more than cats. Learning from this richer signal lets a small student capture much of a big teacher's nuance with a fraction of the size. A distilled model can be small enough to run on a phone.
Be clear about the ceiling, though. A student cannot exceed its teacher's knowledge — it can only approximate it, and it inevitably drops some of the teacher's edge cases and rare skills. "Almost as good" usually means a real but modest quality loss in exchange for a large speed and cost win. And distilling from someone else's model raises its own questions, since some providers' terms forbid using their model's outputs to train a competitor.
A huge model that needs a rack of servers answers slowly and costs a lot per query. A team distills it into a model one-twentieth the size that runs on a single chip, answering in a fraction of the time. On everyday questions the small model matches it; on rare, tricky ones it slips a little.
A small student imitates a big teacher — much cheaper, nearly as good.
A distilled model can approach its teacher but never surpass it, and it quietly loses some rare or edge-case skills. "Nearly as good" is a genuine trade, not a free copy — and distilling from a rival's model outputs may violate that provider's terms of use.