Intelligibility of natural and 3D-cloned German speech

Sascha Fagel (1) , Gérard Bailly (2) , Frédéric Elisei (3)

1 ISK - Institut für Sprache und Kommunikation
2 GIPSA-MPACIF - GIPSA - Machines Parlantes, Agents Communicants & Interaction Face-à-face
3 GIPSA-Services - GIPSA-Services

We investigate the intelligibility of natural visual and audiovisual speech compared to re-synthesized speech movements rendered by a talking head. This talking head is created using the speaker cloning methodology of the Institut de la Communication Parlée in Grenoble (now department for speech and cognition in GIPSA- Lab). A German speaker with colored markers on the face was recorded audiovisually using multiple cameras. The three-dimensional coordinates of the markers were extracted and parameterized. Spoken VCV sequences were then visually re-synthesized. A perception experiment was carried out to measure the visual and audiovisual intelligibility of natural and synthesized video, using the original audio with and without added noise. Identification scores show that the clone is capable of recovering almost 70% of the intelligibility gain provided by the original face. Part of this loss is due to missing visual cues in the present synthesis, due notably to the lack of a tongue.

Mots clés: facial animation; visual speech synthesis; talking head; motion capture; speech intelligibility

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Sascha Fagel, Gérard Bailly, Frédéric Elisei. Intelligibility of natural and 3D-cloned German speech. AVSP 2007 - 6th International Conference on Auditory-Visual Speech Processing, Aug 2007, Hilvarenbeek, Netherlands. pp.56-61. ⟨hal-00169563⟩