Analyzing the Impact of Prevalence on the Evaluation of a Manual Annotation Campaign

Abstract : This article details work aiming at evaluating the quality of the manual annotation of gene renaming couples in scientific abstracts, which generates sparse annotations. To evaluate these annotations, we compare the results obtained using the commonly advocated inter-annotator agreement coefficients such as S, κ and π, the less known R, the weighted coefficients κω and α as well as the F-measure and the SER. We analyze to which extent they are relevant for our data. We then study the bias introduced by prevalence by changing the way the contingency table is built. We finally propose an original way to synthesize the results by computing distances between categories, based on the produced annotations.
Type de document :
Communication dans un congrès
International Conference on Language Resources and Evaluation (LREC), May 2012, Istanbul, Turkey. 2012
Liste complète des métadonnées

Littérature citée [21 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-00709174
Contributeur : Karën Fort <>
Soumis le : lundi 18 juin 2012 - 10:01:12
Dernière modification le : mardi 15 janvier 2019 - 14:54:16
Document(s) archivé(s) le : mercredi 19 septembre 2012 - 02:31:18

Fichier

lrec-iaa2012_VFinale_Submitted...
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00709174, version 1

Collections

Citation

Karën Fort, Claire François, Olivier Galibert, Maha Ghribi. Analyzing the Impact of Prevalence on the Evaluation of a Manual Annotation Campaign. International Conference on Language Resources and Evaluation (LREC), May 2012, Istanbul, Turkey. 2012. 〈hal-00709174〉

Partager

Métriques

Consultations de la notice

281

Téléchargements de fichiers

270