Visual voice activity detection as a help for speech source separation from convolutive mixtures

Bertrand Rivet (1) , Laurent Girin (1) , Christian Jutten (2)

1 GIPSA-MPACIF - GIPSA - Machines Parlantes, Agents Communicants & Interaction Face-à-face
2 GIPSA-SIGMAPHY - GIPSA - Signal Images Physique

Audio–visual speech source separation consists in mixing visual speech processing techniques (e.g., lip parameters tracking) with source separation methods to improve the extraction of a speech source of interest from a mixture of acoustic signals. In this paper, we present a new approach that combines visual information with separation methods based on the sparseness of speech: visual information is used as a voice activity detector (VAD) which is combined with a new geometric method of separation. The proposed audio–visual method is shown to be efficient to extract a real spontaneous speech utterance in the difficult case of convolutive mixtures even if the competing sources are highly non-stationary. Typical gains of 18–20 dB in signal to interference ratios are obtained for a wide range of (2 × 2) and (3 × 3) mixtures. Moreover, the overall process is computationally quite simpler than previously proposed audio–visual separation schemes.

Mots clés: Physical Sciences; Convolutive mixtures; Highly non-stationary environment; Speech enhancement; Speech source separation; Visual speech processing

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Bertrand Rivet, Laurent Girin, Christian Jutten. Visual voice activity detection as a help for speech source separation from convolutive mixtures. Speech Communication, 2007, 49 (7-8), pp.667-677. ⟨10.1016/j.specom.2007.04.008⟩. ⟨hal-00499184⟩