A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition

Li Liu (1) , Gang Feng (2) , Denis Beautemps (3) , Xiao-Ping Zhang (1)

1 Ryerson University [Toronto]
2 GIPSA-VSLD - GIPSA - Voix Systèmes Linguistiques et Dialectologie
3 GIPSA-CRISSP - GIPSA - Cognitive Robotics, Interactive Systems, & Speech Processing

Cued Speech (CS) is an augmented lip reading with the help of hand coding. Due to lips and hand movements are asynchronous and a direct fusion of these asynchronous features may reduce the efficiency of the recognition, the fusion of them in automatic CS recognition is a challenging problem. In our previous work, we built a hand preceding model for hand positions (vowels) by investigating the temporal organization of hand movements in French CS. In this work, we investigate a suitable value of the hand preceding time for consonants by analyzing the temporal movements of hand shapes in French CS. Then, based on these two results, we propose an efficient resynchronization procedure for the fusion of multi-stream features in CS. This procedure is applied to the continuous CS phoneme recognition based on the multi-stream CNN-HMMs architecture. The result shows that using this procedure brings an improvement of about 4.6% in the phoneme recognition correctness, compared with the state-of-the-art, which does not take into account the asynchrony of multi-modalities.

Mots clés: CNN-HMMs; resynchronization procedure; hand preced- ing time; multi-modal fusion; Index Terms-Cued Speech

Domaines: Sciences de l'information et de la communication

Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang. A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition. EUSIPCO 2019 - 27th European Signal Processing Conference, Sep 2019, A Coruna, Spain. ⟨10.23919/EUSIPCO.2019.8903053⟩. ⟨hal-02352153⟩