Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis

Karen Fort (1, 2) , Adeline Nazarenko (2) , Sophie Rosset (3)

1 SEMAGRAMME - Semantic Analysis of Natural Language
2 LIPN - Laboratoire d'Informatique de Paris-Nord
3 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

Manual corpus annotation is getting widely used in Natural Language Processing (NLP). While being recognized as a difficult task, no in-depth analysis of its complexity has been performed yet. We provide in this article a grid of analysis of the different complexity dimensions of an annotation task, which helps estimating beforehand the difficulties and cost of annotation campaigns. We observe the applicability of this grid on existing annotation campaigns and detail its application on a real-world example.

Keywords: manual corpus annotation; annotation campaign management; annotation cam- paign cost estimate

Domains: Document and Text Processing

Karen Fort, Adeline Nazarenko, Sophie Rosset. Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis. International Conference on Computational Linguistics, Dec 2012, Mumbaï, India. pp.895--910. ⟨hal-00769631⟩