Skip to Main content Skip to Navigation
Conference papers

DNN Driven Speaker Independent Audio-Visual Mask Estimation for Speech Separation

Abstract : Human auditory cortex excels at selectively suppressing background noise to focus on a target speaker. The process of selective attention in the brain is known to contextually exploit the available audio and visual cues to better focus on target speaker while filtering out other noises. In this study, we propose a novel deep neural network (DNN) based audiovisual (AV) mask estimation model. The proposed AV mask estimation model contextually integrates the temporal dynamics of both audio and noise-immune visual features for improved mask estimation and speech separation. For optimal AV features extraction and ideal binary mask (IBM) estimation, a hybrid DNN architecture is exploited that leverages the complementary strengths of a stacked long short term memory (LSTM) and convolution LSTM network. The comparative simulation results in terms of speech quality and intelligibility demonstrate the significant performance improvement of our proposed AV mask estimation model as compared to audio-only and visual-only mask estimation approaches for both speaker dependent and independent scenarios.
Complete list of metadatas

Cited literature [21 references]  Display  Hide  Download
Contributor : Ricard Marxer <>
Submitted on : Wednesday, September 5, 2018 - 4:04:35 PM
Last modification on : Wednesday, December 5, 2018 - 2:01:45 PM
Document(s) archivé(s) le : Thursday, December 6, 2018 - 5:49:09 PM


AVMaskInterspeech18 (3).pdf
Files produced by the author(s)




Mandar Gogate, Ahsan Adeel, Ricard Marxer, Jon Barker, Amir Hussain. DNN Driven Speaker Independent Audio-Visual Mask Estimation for Speech Separation. Interspeech 2018, Sep 2018, Hybderabad, India. pp.2723-2727, ⟨10.21437/Interspeech.2018-2516⟩. ⟨hal-01868604⟩



Record views


Files downloads