A tree-BLSTM-based recognition system for online handwritten mathematical expressions

Ting Zhang (1) , Harold Mouchère (2, 3) , Christian Viard-Gaudin (2)

1 National Engineering Research Center for E-learning
2 LS2N - Laboratoire des Sciences du Numérique de Nantes
3 LS2N - équipe IPI - Image Perception Interaction

Long short-term memory networks (LSTM) achieves great success in temporal dependency modelling for chain-structured data, such as texts and speeches. An extension towards more complex data structures as encountered in 2D graphic languages is proposed in this work. Specifically, we address the problem of handwritten mathematical expression recognition, using a Tree-based BLSTM architecture allowing the direct labelling of nodes (symbol) and edges (relationship) from a graph modelling the input strokes. One major difference with the traditional approaches is that there is no explicit segmentation, recognition and layout extraction steps but a unique trainable system that produces directly a Stroke Label Graph describing a mathematical expression. The proposed system, considering no grammar, achieves competitive results in online math expression recognition domain.

Mots clés: Mathematical expression recognition; tree-based BLSTM; local CTC; online handwriting

Domaines: Réseau de neurones [cs.NE]; Apprentissage [cs.LG]; Traitement des images [eess.IV]; Intelligence artificielle [cs.AI]

Ting Zhang, Harold Mouchère, Christian Viard-Gaudin. A tree-BLSTM-based recognition system for online handwritten mathematical expressions. Neural Computing and Applications, 2018, ⟨10.1007/s00521-018-3817-2⟩. ⟨hal-01900412⟩