Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection

Résumé

In video understanding, most cross-modal knowledge distillation (KD) methods are tailored for classification tasks, focusing on the discriminative representation of the trimmed videos. However, action detection requires not only categorizing actions, but also localizing them in untrimmed videos. Therefore, transferring knowledge pertaining to temporal relations is critical for this task which is missing in the previous cross-modal KD frameworks. To this end, we aim at learning an augmented RGB representation for action detection, taking advantage of additional modalities at training time through KD. We propose a KD framework consisting of two levels of distillation. On one hand, atomic-level distillation encourages the RGB student to learn the sub-representation of the actions from the teacher in a contrastive manner. On the other hand, sequence-level distillation encourages the student to learn the temporal knowledge from the teacher, which consists of transferring the Global Contextual Relations and the Action Boundary Saliency. The result is an Augmented-RGB stream that can achieve competitive performance as the two-stream network while using only RGB at inference time. Extensive experimental analysis shows that our proposed distillation framework is generic and outperforms other popular cross-modal distillation methods in action detection task.
Fichier principal
Vignette du fichier
Dai_ICCV21.pdf (1.75 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03314575 , version 1 (05-08-2021)

Identifiants

  • HAL Id : hal-03314575 , version 1

Citer

Rui Dai, Srijan Das, Francois F Bremond. Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection. ICCV 2021 - IEEE/CVF International Conference on Computer Vision, Oct 2021, Montreal, Canada. ⟨hal-03314575⟩
144 Consultations
118 Téléchargements

Partager

Gmail Facebook X LinkedIn More