Lip Reading with Hahn Convolutional Neural Networks moments

Abderrahim Mesbah (1) , Hicham Hammouchi (1, 2) , Aissam Berrahou (2) , Hassan Berbia (2) , Hassan Qjidaa (1) , Mohamed Daoudi (3, 4)

1 USMBA - Université Sidi Mohamed Ben Abdellah
2 UM5 - Université Mohammed V de Rabat [Agdal]
3 CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
4 IMT Lille Douai - Ecole nationale supérieure Mines-Télécom Lille Douai

Lipreading or Visual speech recognition is the process of decoding speech from speakers mouth movements. It is used for people with hearing impairment , to understand patients attained with laryngeal cancer, people with vocal cord paralysis and in noisy environment. In this paper we aim to develop a visual-only speech recognition system based only on video. Our main targeted application is in the medical field for the assistance to la-ryngectomized persons. To that end, we propose Hahn Convolutional Neu-ral Network (HCNN), a novel architecture based on Hahn moments as first layer in the Convolutional neural network (CNN) architecture. We show that HCNN helps in reducing the dimensionality of video images, in gaining training time. HCNN model is trained to classify letters, digits or words given as video images. We evaluated the proposed method on three datasets, AVLetters, OuluVS2 and BBC LRW, and we show that it achieves significant results in comparison with other works in the literature.

Mots clés: Visual speech recognition; Lipreading; Laryngectomy; Deep learning

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Abderrahim Mesbah, Hicham Hammouchi, Aissam Berrahou, Hassan Berbia, Hassan Qjidaa, et al.. Lip Reading with Hahn Convolutional Neural Networks moments. Image and Vision Computing, In press. ⟨hal-02109397⟩