AVATAR: Unconstrained Audiovisual Speech Recognition

Valentin Gabeur (1, 2) , Paul Hongsuck Seo (3) , Arsha Nagrani (3) , Chen Sun (3) , Karteek Alahari (1) , Cordelia Schmid (3)

1 Thoth - Apprentissage de modèles à partir de données massives
2 Google Inc
3 Google Inc.

Audiovisual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth. Unlike works that simply focus on the lip motion, we investigate the contribution of entire visual frames (visual actions, objects, background etc.). This is particularly useful for unconstrained videos, where the speaker is not necessarily visible. To solve this task, we propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) which is trained end-to-end from spectrograms and full-frame RGB. To prevent the audio stream from dominating training, we propose different word-masking strategies, thereby encouraging our model to pay attention to the visual stream. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise, and show that our model outperforms all other prior work by a large margin. Finally, we also create a new, realworld test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.

Mots clés: video; audiovisual; multi-modal; speech recognition

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Valentin Gabeur, Paul Hongsuck Seo, Arsha Nagrani, Chen Sun, Karteek Alahari, et al.. AVATAR: Unconstrained Audiovisual Speech Recognition. INTERSPEECH 2022 - Conference of the International Speech Communication Association, Sep 2022, Incheon, South Korea. pp.1-6. ⟨hal-03717330⟩