Bootstrapped OCR error detection for a less-resourced language variant

Adrien Barbaresi (1, 2)

1 BBAW - Berlin-Brandenburgische Akademie der Wissenschaften
2 OeAW - Austrian Academy of Sciences

This study focuses on isolated error detection in a retro-digitized newspaper corpus published from 1946 to 1990 in the former German Democratic Republic. As there are OCR errors throughout the corpus but no clean reference for this variant of German, automatic OCR correction implies to overcome data sparseness and non-standard spelling, including compounds and inflected forms. The contributions of this paper are (1) a method to bootstrap detection of potential misspellings, (2) an assessment of several types of training data, and (3) an evaluation of several off-the-shelf candidate selection techniques. The chosen solution based on statistical affix analysis reaches an accuracy 10 points higher than existing morphological analysis systems on error detection, while a combination of fuzzy and approximate string search performs best for error correction. The criteria are met since it is possible to correct erroneous tokens without introducing too much noise.

Mots clés: OCR error correction; Affix trees; Cultural Heritage; Morphological Analysis

Domaines: Linguistique; Informatique et langage [cs.CL]; Héritage culturel et muséologie

Adrien Barbaresi. Bootstrapped OCR error detection for a less-resourced language variant. 13th Conference on Natural Language Processing (KONVENS 2016), Sep 2016, Bochum, Germany. pp.21-26. ⟨hal-01371689⟩