A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition

Li Liu 1 Gang Feng 2 Denis Beautemps 3 Xiao-Ping Zhang 1
2 GIPSA-VSLD - VSLD
GIPSA-DPC - Département Parole et Cognition
3 GIPSA-CRISSP - CRISSP
GIPSA-DPC - Département Parole et Cognition
Abstract : Cued Speech (CS) is an augmented lip reading with the help of hand coding. Due to lips and hand movements are asynchronous and a direct fusion of these asynchronous features may reduce the efficiency of the recognition, the fusion of them in automatic CS recognition is a challenging problem. In our previous work, we built a hand preceding model for hand positions (vowels) by investigating the temporal organization of hand movements in French CS. In this work, we investigate a suitable value of the hand preceding time for consonants by analyzing the temporal movements of hand shapes in French CS. Then, based on these two results, we propose an efficient resynchronization procedure for the fusion of multi-stream features in CS. This procedure is applied to the continuous CS phoneme recognition based on the multi-stream CNN-HMMs architecture. The result shows that using this procedure brings an improvement of about 4.6% in the phoneme recognition correctness, compared with the state-of-the-art, which does not take into account the asynchrony of multi-modalities.
Document type :
Conference papers
Complete list of metadatas

Cited literature [22 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02352153
Contributor : Denis Beautemps <>
Submitted on : Wednesday, November 6, 2019 - 5:16:35 PM
Last modification on : Saturday, November 9, 2019 - 1:25:40 AM

File

submitted vesion Eusipco 2019....
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02352153, version 1

Collections

Citation

Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang. A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition. EUSIPCO, Sep 2019, Coruna, Spain. ⟨hal-02352153⟩

Share

Metrics

Record views

9

Files downloads

8