Skip to Main content Skip to Navigation
Conference papers

A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition

Abstract : Cued Speech (CS) is an augmented lip reading with the help of hand coding. Due to lips and hand movements are asynchronous and a direct fusion of these asynchronous features may reduce the efficiency of the recognition, the fusion of them in automatic CS recognition is a challenging problem. In our previous work, we built a hand preceding model for hand positions (vowels) by investigating the temporal organization of hand movements in French CS. In this work, we investigate a suitable value of the hand preceding time for consonants by analyzing the temporal movements of hand shapes in French CS. Then, based on these two results, we propose an efficient resynchronization procedure for the fusion of multi-stream features in CS. This procedure is applied to the continuous CS phoneme recognition based on the multi-stream CNN-HMMs architecture. The result shows that using this procedure brings an improvement of about 4.6% in the phoneme recognition correctness, compared with the state-of-the-art, which does not take into account the asynchrony of multi-modalities.
Document type :
Conference papers
Complete list of metadata

Cited literature [22 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02352153
Contributor : Denis Beautemps Connect in order to contact the contributor
Submitted on : Wednesday, November 6, 2019 - 5:16:35 PM
Last modification on : Wednesday, November 3, 2021 - 5:13:49 AM
Long-term archiving on: : Saturday, February 8, 2020 - 2:06:20 AM

File

submitted vesion Eusipco 2019....
Files produced by the author(s)

Identifiers

Collections

Citation

Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang. A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition. EUSIPCO 2019 - 27th European Signal Processing Conference, Sep 2019, A Coruna, Spain. ⟨10.23919/EUSIPCO.2019.8903053⟩. ⟨hal-02352153⟩

Share

Metrics

Record views

240

Files downloads

437