Lip shape and hand position fusion for automatic vowel recognition in Cued Speech for French

Panikos Heracleous (1) , Noureddine Aboutabit (1) , Denis Beautemps (1)

1 GIPSA-MPACIF - GIPSA - Machines Parlantes, Agents Communicants & Interaction Face-à-face

Cued Speech is a sound-based system, which uses handshapes in different positions and in combination with lippatterns of speech, and makes all the sounds of spoken language clearly understandable to deaf and hearing-impaired people. The aim of Cued Speech is to overcome the problems of lip reading and thus enable deaf children and adults to wholly understand spoken language. Cued Speech recognition requires gesture recognition and lip shape recognition. In addition, the integration of the two components is of great importance. This article presents hidden Markov model (HMM)-based vowel recognition as used in Cued Speech for French. Based on concatenative feature fusion and multi-stream HMM decision fusion, lip shape and hand position components were integrated into a single component, and automatic vowel recognition was realized. In the case of multi-stream HMM decision fusion, the obtained vowel classication accuracy using lip shape and hand position information was 87.6%, showing absolute improvement of 19.6% in comparison with a use restricted only to lip parameters. The results achieved show the effectiveness of the proposed approaches to Cued Speech recognition.

Mots clés: multi-stream HMM fusion; concatenative fusion; Cued Speech; HMM; vowel recognition

Domaines: Sciences de l'information et de la communication

Panikos Heracleous, Noureddine Aboutabit, Denis Beautemps. Lip shape and hand position fusion for automatic vowel recognition in Cued Speech for French. IEEE Signal Processing Letters, 2009, 16 (5), pp.339-342. ⟨10.1109/LSP.2009.2016011⟩. ⟨hal-00346166v2⟩