distillation to a smaller LLM

Distillation is teaching a small, cheap model to imitate a big, expensive one. The large teacher shows the small student how it would respond, and the student learns to match it — ending up far faster and lighter while keeping much of the teacher's skill.

In the classic form, the student is trained to match the teacher's full probability distribution over the next token (the soft labels), which carries richer information than a single right answer. A simpler and now-common variant just fine-tunes the student on text the teacher generated. Either way the student captures behaviour the teacher learned at great cost. The ceiling is the teacher: a student rarely surpasses what it imitates, and it can inherit the teacher's flaws.

Distillation is how many small, deployable models get made — turning a frontier model's quality into something that runs on a phone, or cheaply at scale.