SpottingNet: Learning the Similarity of Word Images with Convolutional Neural Network for Word Spotting in Handwritten Historical Documents

Abstract : Word spotting is a content-based retrieval process that obtains a ranked list of word image candidates similar to the query word in digital document images. In this paper, we propose a similarity score fusion method integrated with hybrid deep-learning classification and regression models to enhance performance for Query-by-Example (QBE) word spotting. Based on the convolutional neural network end-to-end framework, the presented models enable conjointly learning of the representative word image descriptors and evaluation of the similarity measure between word descriptors directly from the word image, which are the two crucial factors in this task. In addition, we present a sample generation method using location jitter to balance similar and dissimilar image pairs and enlarge the dataset. Experiments are conducted on the classical George Washington (GW) dataset without involving any recognition methods or prior word category information. Our experiments show that the proposed model yields state-of-the-art mean average precision (mAP) of 80.03%, significantly outperforming previous results.