Evaluer des annotations manuelles dispersées : les coefficients sont-ils suffisants pour estimer l'accord inter-annotateurs ?

Karen Fort (1, 2) , Claire François (1) , Maha Ghribi (1)

1 INIST - Institut de l'information scientifique et technique
2 LIPN - Laboratoire d'Informatique de Paris-Nord

This article details work aiming at evaluating the quality of the manual annotation of gene renaming relations in scientific abstracts, which generates dispersed annotations. To evaluate these annotations, we computed and compared the results obtained using the commonly advocated inter-annotator agreement coefficients like κ (Cohen, 1960) or π (Scott, 1955) and analyzed to which extent they are relevant for our data. We also studied the different weighting computations applicable to κω (Cohen, 1968) and α (Krippendorff, 1980, 2004) and defined a way to compute distances between categories based on the produced annotations. We then propose a first approach to estimate the bias introduced by prevalence.

Mots clés: inter-annotator agreement; Manual annotation; evaluation; inter-annotator agreement.

Domaines: Traitement du texte et du document

Karen Fort, Claire François, Maha Ghribi. Evaluer des annotations manuelles dispersées : les coefficients sont-ils suffisants pour estimer l'accord inter-annotateurs ?. Traitement Automatique des Langues Naturelles (TALN), Jul 2010, Montréal, France. pp.0. ⟨hal-00484265v1⟩