PDF-Distil: including Prediction Disagreements in Feature-based Distillation for object detection

Heng Zhang (1) , Elisa Fromont (2) , Sébastien Lefèvre (3) , Bruno Avignon (1)

1 ATERMES [Montigny-le-Bretonneux]
2 LACODAM - Large Scale Collaborative Data Mining
3 OBELIX - Observation de l’environnement par imagerie complexe

Knowledge distillation aims at compressing deep models by transferring the learned knowledge from precise but cumbersome teacher models to compact student models. Due to the extreme imbalance between the foreground and the background of images, when traditional knowledge distillation methods are directly applied to the object detection task, there is a large performance gap between the teacher model and the student model. We tackle this imbalance problem from a sampling perspective, and we propose to include the teacher-student prediction disagreements into a feature-based detection distillation framework. This is done with PDF-Distil by dynamically generating a weighting mask applied to the knowledge distillation loss, based on the disagreements between the predictions of both models. Extensive experiments on PASCAL VOC and MS COCO datasets demonstrate that, compared to state-of-the-art methods, PDF-Distil is able to better reduce the performance gap between the teacher and student models.

Domaines: Apprentissage [cs.LG]

Heng Zhang, Elisa Fromont, Sébastien Lefèvre, Bruno Avignon. PDF-Distil: including Prediction Disagreements in Feature-based Distillation for object detection. BMVC 2021 - 32nd British Machine Vision Conference, Nov 2021, Virtual, United Kingdom. pp.1-13. ⟨hal-03487128⟩