Exploiting visual information for NAM recognition

Panikos Heracleous (1) , Denis Beautemps (1) , Viet-Anh Tran (2) , Hélène Loevenbruck (3) , Gérard Bailly (1)

1 GIPSA-MPACIF - GIPSA - Machines Parlantes, Agents Communicants & Interaction Face-à-face
2 GIPSA-lab - Grenoble Images Parole Signal Automatique
3 GIPSA-PMD - GIPSA - Parole, Multimodalité, Développement

Non-audible murmur (NAM) is an unvoiced speech received through body tissue using special acoustic sensors (i.e., NAM microphones) attached behind the talkers ear. Although NAM has different frequency characteristics compared to normal speech, it is possible to perform automatic speech recognition (ASR) using conventional methods. In using a NAM microphone, body transmission and the loss of lip radiation act as a low-pass filter; as a result, higher frequency components are attenuated in NAM signal. A decrease in NAM recognition performance is attributed to spectral reduction. To address the problem of loss of lip radiation, visual information extracted from the talker's facial movements is fused with NAM speech. Experimental results revealed a relative improvement of 39% when fused NAM speech and facial information were used as compared to using only NAM speech. Results also showed that improvements in the recognition rate depend on the place of articulation.

Domaines: Sciences de l'information et de la communication

Panikos Heracleous, Denis Beautemps, Viet-Anh Tran, Hélène Loevenbruck, Gérard Bailly. Exploiting visual information for NAM recognition. IEICE Electronics Express, 2009, 6 (2), pp.77-82. ⟨10.1587/elex.6.77⟩. ⟨hal-00357985⟩