Modern AI Engineering

Lesson 8.4 · 24 min · video lesson

Knowledge Distillation: Compressing Large Models into Small Ones

How can a small model learn more from a big model’s “wrong” answers than from the right answer alone?

In short: Knowledge distillation trains a small student model to imitate a large teacher model. Instead of learning only from hard labels (the one correct class), the student also matches the teacher’s full probability distribution, softened with a temperature, which carries “dark knowledge” about how classes relate. The result is a smaller, faster model that keeps much of the teacher’s quality.

What you will learn

  • What is Knowledge Distillation?
  • Why we need Knowledge Distillation
  • Hard labels vs soft labels
  • Dark knowledge
  • Temperature in the softmax
  • The distillation loss
  • A step-by-step training walkthrough
  • Types of Knowledge Distillation
  • Real examples of Knowledge Distillation
  • Wrapping up Knowledge Distillation

In this lesson

  1. What is knowledge distillation?
  2. Why we need knowledge distillation
  3. Hard labels vs soft labels, and dark knowledge
  4. Temperature in the softmax
  5. The distillation loss
  6. A step-by-step training walkthrough
  7. Types of knowledge distillation
  8. Worked example, step by step
  9. Practice: try it yourself
  10. Real examples, pitfalls and wrapping up

Key terms in this lesson

Teacher
The large, accurate model whose behaviour is being transferred.
Student
The smaller model trained to imitate the teacher.
Soft label
A full probability distribution over classes, as opposed to a single correct class.
Dark knowledge
The similarity information hidden in a model’s probabilities for incorrect classes.
Temperature
A divisor applied to logits before softmax; higher values flatten the distribution.
KL divergence
A measure of how one probability distribution differs from another.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.3 Prefix Tuning: Learnable Context Prepended to the Input · 8.5 Continual Learning: Training Without Forgetting the Past →