Knowledge distillation is the technique of training a small model, the student, to imitate a larger, already-capable model, the teacher. Picture a bright pupil who learns not just the textbook answers, but the reasoning of their mentor: they become nearly as skilled while fitting in your pocket.
The core idea: learning from "soft labels"
Instead of training the student only on hard labels (a "cat" is 1, everything else 0), we train it to reproduce the teacher's full probability distribution — the soft labels. These carry valuable information: the teacher knows a husky looks somewhat like a wolf. This dark knowledge is what gets transferred.
Outputs are softened through a temperature $T$ in the softmax:
$$ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} $$
A higher $T$ smooths the distribution, exposing inter-class similarities. The loss then blends teacher imitation with the ground truth:
$$ \mathcal{L} = \alpha \, \mathcal{L}{\text{soft}}(\text{student}, \text{teacher}) + (1-\alpha)\, \mathcal{L}, y) $$}}(\text{student
Teacher versus student
| Criterion | Teacher | Student |
|---|---|---|
| Size | Very large | Compact |
| Inference cost | High | Low |
| Accuracy | Reference | Close, slightly lower |
| Deployment | Server/cloud | Mobile, embedded |
Why it matters
- Fast, lightweight models for phones and connected devices.
- Lower inference costs and energy use.
- The foundation of many recent compact models (the "distilled" families).
To distill is to keep the essence of a giant's knowledge in a form anyone can use.