Skip to Main content Skip to Navigation
New interface
Conference papers

Masking Modalities for Cross-modal Video Retrieval

Valentin Gabeur 1, 2 Arsha Nagrani 2 Chen Sun 2 Karteek Alahari 1 Cordelia Schmid 2 
1 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : Pre-training on large scale unlabelled datasets has shown impressive performance improvements in the fields of computer vision and natural language processing. Given the advent of large-scale instructional video datasets, a common strategy for pre-training video encoders is to use the accompanying speech as weak supervision. However, as speech is used to supervise the pre-training, it is never seen by the video encoder, which does not learn to process that modality. We address this drawback of current pre-training methods, which fail to exploit the rich cues in spoken language. Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech. We mask an entire modality in the input and predict it using the other two modalities. This encourages each modality to collaborate with the others, and our video encoder learns to process appearance and audio as well as speech. We show the superior performance of our 'modality masking' pre-training approach for video retrieval on the How2R, YouCook2 and Condensed Movies datasets.
Document type :
Conference papers
Complete list of metadata
Contributor : THOTH Team Connect in order to contact the contributor
Submitted on : Tuesday, November 9, 2021 - 7:54:02 AM
Last modification on : Friday, November 18, 2022 - 9:28:12 AM
Long-term archiving on: : Thursday, February 10, 2022 - 6:11:35 PM


Files produced by the author(s)


  • HAL Id : hal-03420133, version 1
  • ARXIV : 2111.01300



Valentin Gabeur, Arsha Nagrani, Chen Sun, Karteek Alahari, Cordelia Schmid. Masking Modalities for Cross-modal Video Retrieval. WACV 2022 - Winter Conference on Applications of Computer Vision, Jan 2022, Waikoloa, United States. pp.1-10. ⟨hal-03420133⟩



Record views


Files downloads