Multimodal Mathematical Expressions Recognition: Case of Speech and Handwriting

Sofiane Medjkoune (1) , Harold Mouchère (1) , Simon Petitrenaud (2) , Christian Viard-Gaudin (1)

1 irccyn-ivc
2 LIUM - Laboratoire d'Informatique de l'Université du Mans

In this work, we propose to combine two modalities, handwriting and speech, to build a mathematical expression recognition system. Based on two sub-systems which process each modality, we explore various fusion methods to resolve ambiguities which naturally occur independently. The results that are reported on the HAMEX bimodal database show an improvement with respect to a mono-modal based system.

Mots clés: Multimodality; graphical languages; data fusion; handwriting; speech

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Sofiane Medjkoune, Harold Mouchère, Simon Petitrenaud, Christian Viard-Gaudin. Multimodal Mathematical Expressions Recognition: Case of Speech and Handwriting. 15th International Conference on Human-Computer Interaction, Jul 2013, Las Vegas, United States. pp.77-86, ⟨10.1007/978-3-642-39330-3_9⟩. ⟨hal-00852846⟩