Skip to Main content Skip to Navigation
Conference papers

Text Alignment from Bimodal Mathematical Expression Sources

Sofiane Medjkoune 1 Harold Mouchère 1 Christian Viard-Gaudin 1 Simon Petitrenaud 2
1 irccyn-ivc
IRCCyN - Institut de Recherche en Communications et en Cybernétique de Nantes
Abstract : In this paper we propose a new approach tomerge mathematical expression recognition results coming fromhandwriting and speech modalities. Using a bimodal descriptionof mathematical expressions allows taking advantage of thecomplementarities between both signals, and can disambiguatesituations were a single modality would not be clear enough. Tocombine the signals coming from both modalities, we propose torepresent them in the same space as a textual description. First,from the handwriting signal, we generate the Nbest mathematicalexpressions; each of them is next translated as different possiblestrings. From the audio signal, an automatic speech recognitionsystem provides a transcript, which is also available as a string.A string comparison algorithm is achieved to select the bestmathematical expressions. This bimodal system is evaluated onreal bimodal data from the HAMEX dataset and the results arecompared to a single modality (handwriting) based system.
Document type :
Conference papers
Complete list of metadatas
Contributor : Harold Mouchère <>
Submitted on : Wednesday, December 17, 2014 - 4:06:33 PM
Last modification on : Wednesday, December 19, 2018 - 3:02:08 PM
Long-term archiving on: : Monday, March 23, 2015 - 3:36:25 PM


Publisher files allowed on an open archive



Sofiane Medjkoune, Harold Mouchère, Christian Viard-Gaudin, Simon Petitrenaud. Text Alignment from Bimodal Mathematical Expression Sources. 14th International Conference on Frontiers in Handwriting Recognition, Sep 2014, Crete, Greece. pp.205--209, ⟨10.1109/ICFHR.2014.42⟩. ⟨hal-01096510⟩



Record views


Files downloads