Knowledge distillation is a model compression technique where a smaller student model learns to replicate the behavior of a larger teacher model by training on the teacher's output probability distributions rather than just hard labels. ' The near-miss classifications, the confidence levels, the relationships between classes, all of this transfers to the student.
The student learns not just what the right answer is, but how the teacher reasons about uncertainty. This produces smaller models that punch above their weight. A distilled model with 10% of the parameters might achieve 95% of the teacher's accuracy. The technique was formalized by Geoffrey Hinton in 2015 and has become required for deploying AI in resource-constrained environments.
Mobile applications, edge devices, and real-time systems all benefit from distilled models that run faster and use less memory. Modern applications include distilling large language models like GPT-4 into smaller models, distilling vision transformers into efficient CNNs, and multi-teacher distillation where students learn from ensembles.
The temperature parameter in distillation controls how much the soft labels are softened, with higher temperatures revealing more about the teacher's learned structure.
Interactive Visualizer
Knowledge Distillation
Watch how a student model learns from soft probability distributions instead of hard labels. The teacher's uncertainty and near-miss predictions contain valuable knowledge.