Lip-synching using speaker-specific articulation, shape and appearance models

Gérard Bailly (1) , Oxana Govokhina (1, 2) , Frédéric Elisei (3) , Gaspard Breton (2)

1 GIPSA-MPACIF - GIPSA - Machines Parlantes, Agents Communicants & Interaction Face-à-face
2 Orange Labs R&D [Rennes]
3 GIPSA-Services - GIPSA-Services

We describe here the control, shape and appearance models that are built using an original photogrammetric method to capture characteristics of speaker-specific facial articulation, anatomy, and texture. Two original contributions are put forward here: the trainable trajectory formation model that predicts articulatory trajectories of a talking face from phonetic input and the texture model that computes a texture for each 3D facial shape according to articulation.Usingmotion capture data from different speakers and module-specific evaluation procedures, we show here that this cloning system restores detailed idiosyncrasies and the global coherence of visible articulation. Results of a subjective evaluation of the global system with competing trajectory formation models are further presented and commented.

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Gérard Bailly, Oxana Govokhina, Frédéric Elisei, Gaspard Breton. Lip-synching using speaker-specific articulation, shape and appearance models. EURASIP Journal on Audio, Speech, and Music Processing, 2009, Special issue on animating virtual speakers or singers from audio: Lip-synching facial animation, pp.ID 769494. ⟨10.1155/2009/769494⟩. ⟨hal-00447061⟩