What is knowledge distillation?

Knowledge distillation is a model compression technique where a small "student" model is trained to mimic the behavior of a larger, pre-trained "teacher" model. This is often done by transferring the soft probabilities (output probabilities before applying argmax) from the teacher to the student, along with the hard labels. It is commonly used to create smaller, faster models for deployment on resource-constrained devices or to improve the generalization ability of the student model.