Using BLSTM for interpretation of 2-D languages: Case of handwritten mathematical expressions

Abstract : In this work, we study how to extend the capability of BLSTM networks with CTC to process data which are not only text strings but graphical two-dimensional languages such as handwritten mathematical expressions. An online math expression is a sequence of strokes which is later labeled by BLSTM network. Besides normal math symbols, we introduce 6 additional specific labels assigned to each of the different possible spatial relationships that exist between sub-expressions. The output of BLSTM network with CTC is a sequence of labels. Our aim is to build a two-dimensional (2-D) expression from this sequence of labels. CTC technology is a good choice for sequence transcription tasks but does not provide the alignment between the inputs and the target labels. In our case, we need the labels of strokes in the building process. A local CTC is proposed to solve this problem. As a result, BLSTM network is able to perform at the same time the symbol recognition task, the segmentation task and the relationship recognition task, which is a new perspective for the mathematical expression domain.
Complete list of metadatas
Contributor : Harold Mouchère <>
Submitted on : Tuesday, August 22, 2017 - 6:31:33 PM
Last modification on : Wednesday, December 19, 2018 - 3:02:05 PM




Ting Zhang, Harold Mouchère, Christian Viard-Gaudin. Using BLSTM for interpretation of 2-D languages: Case of handwritten mathematical expressions. Document Numérique, 2016, De l’analyse du manuscrit à la recherche d’information dans les réseaux sociaux, 19 (2), ⟨⟩. ⟨10.3166/DN.19.2-3.135-157⟩. ⟨hal-01576302⟩



Record views