Elephant: Sequence Labeling for Word and Sentence Segmentation

Kilian Evang (1) , Valerio Basile (1) , Grzegorz Chrupała (2) , Johan Bos (1)

1 University of Groningen [Groningen]
2 Tilburg University [Netherlands]

Tokenization is widely regarded as a solved problem due to the high accuracy that rule-based tokenizers achieve. But rule-based tokenizers are hard to maintain and their rules language specific. We show that high-accuracy word and sentence segmentation can be achieved by using supervised sequence labeling on the character level combined with unsupervised feature learning. We evaluated our method on three languages and obtained error rates of 0.27 ‰ (English), 0.35 ‰ (Dutch) and 0.76 ‰ (Italian) for our best models .

Mots clés: Tokenization; Sequence labeling; Machine learning; Deep learning; segmentation

Domaines: Linguistique; Informatique et langage [cs.CL]; Traitement du texte et du document

Kilian Evang, Valerio Basile, Grzegorz Chrupała, Johan Bos. Elephant: Sequence Labeling for Word and Sentence Segmentation. EMNLP 2013, Oct 2013, Seattle, United States. ⟨hal-01344500⟩