Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events

Sanjeel Parekh (1) , Slim Essid (2) , Alexey Ozerov (1) , Ngoc Q K Duong (1) , Patrick Pérez (1) , Gael Richard (3, 4)

1 Technicolor R & I [Cesson Sévigné]
2 LTCI - Laboratoire Traitement et Communication de l'Information
3 S2A - Signal, Statistique et Apprentissage
4 IDS - Département Images, Données, Signal

Audiovisual representation learning is an important task from the perspective of designing machines with the ability to understand complex events. To this end, we propose a novel multimodal framework that instantiates multiple instance learning. We show that the learnt representations are useful for classifying events and localizing their characteristic audiovisual elements. The system is trained using only video-level event labels without any timing information. An important feature of our method is its capacity to learn from unsynchronized audiovisual events. We achieve state-of-the-art results on a large-scale dataset of weakly-labeled audio event videos. Visualizations of localized visual regions and audio segments substantiate our system's efficacy, especially when dealing with noisy situations where modality-specific cues appear asynchronously.

Mots clés: Audio-visual fusion; multimodal deep learning; multiple in- stance learning; event classification; audio-visual localization

Domaines: Traitement du signal et de l'image [eess.SP]

Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Q K Duong, Patrick Pérez, et al.. Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events. CVPR Workshop, 2018, Salt Lake city, United States. ⟨hal-02713307⟩