Identify, locate and separate: Audio-visual object extraction in large video collections using weak supervision

Sanjeel Parekh (1) , Alexey Ozerov (1) , Slim Essid (2) , Ngoc Q K Duong (1) , Patrick Pérez (3) , Gael Richard (2)

1 Technicolor R & I [Cesson Sévigné]
2 LTCI - Laboratoire Traitement et Communication de l'Information
3 Valeo.ai

We tackle the problem of audiovisual scene analysis for weakly-labeled data. To this end, we build upon our previous audiovisual representation learning framework to perform object classification in noisy acoustic environments and integrate audio source enhancement capability. This is made possible by a novel use of non-negative matrix factorization for the audio modality. Our approach is founded on the multiple instance learning paradigm. Its effectiveness is established through experiments over a challenging dataset of music instrument performance videos. We also show encouraging visual object localization results.

Mots clés: multiple instance learning; Index Terms-Audio-visual event detection; non-negative matrix factorization; audio-visual event detection; source separation

Domaines: Traitement du signal et de l'image [eess.SP]; Acoustique [physics.class-ph]; Vision par ordinateur et reconnaissance de formes [cs.CV]; Réseau de neurones [cs.NE]

Sanjeel Parekh, Alexey Ozerov, Slim Essid, Ngoc Q K Duong, Patrick Pérez, et al.. Identify, locate and separate: Audio-visual object extraction in large video collections using weak supervision. 2018. ⟨hal-01626389v2⟩