Recognizing prosody from the lips

Marion Dohen (1) , Hélène Loevenbruck (1) , Harold Hill (2, 3)

1 GIPSA-PMD - GIPSA - Parole, Multimodalité, Développement
2 ATR - ATR Cognitive Information Science Labs
3 School of Psychology [Wollongong]

The aim of this chapter is to examine the possibility of extracting prosodic information from lip features. We used two measurement techniques enabling automatic lip feature extraction to evaluate the "lip pattern" of prosodic focus in French. Two corpora with Subject-Verb-Object (SVO) sentences were designed. Four focus conditions (S, V, O or neutral) were elicited in a natural dialogue situation. In a first set of experiments, we recorded two speakers of French with front and profile video cameras. The speakers wore blue make-up and facial markers. In a second set we recorded five speakers with a 3D optical tracker. An analysis of the lip features showed that visible articulatory lip correlates of focus exist for all speakers. Two types of patterns were observed: absolute and differential. A potential outcome of this study is to provide criteria for automatic visual detection of prosodic focus from lip data.

Mots clés: Video; Lip motion capture; Speech Production; Articulation; Optotrak; Prosody; Focus; Visual features

Domaines: Traitement du signal et de l'image [eess.SP]; Linguistique; Linguistique; Traitement du signal et de l'image [eess.SP]

Marion Dohen, Hélène Loevenbruck, Harold Hill. Recognizing prosody from the lips: is it possible to extract prosodic focus from lip features?. Alan Wee-Chung Liew & Shilin Wang. Visual Speech Recognition: Lip Segmentation and Mapping, Medical Information Science Reference, pp.416-438, 2009, 978-1-60566-186-5. ⟨10.4018/978-1-60566-186-5.ch014⟩. ⟨hal-00360765⟩