 |
|
|
|
| HAL : hal-00484265, version 2 |
| Fiche détaillée |  Récupérer au format |
|
|
| Traitement Automatique des Langues Naturelles (TALN), Montréal : France (2010) |
|
|
| Versions disponibles : | v1 (18-05-2010) | v2 (27-05-2010) |
|
|
|
|
| Evaluer des annotations manuelles dispersées : les coefficients sont-ils suffisants pour estimer l'accord inter-annotateurs ? |
|
|
| Karen Fort 1, 2Claire François 1 |
|
|
| Quaero Collaboration(s) |
|
|
| (2010) |
|
|
|
| This article details work aiming at evaluating the quality of the manual annotation of gene renaming relations in scientific abstracts, which generates dispersed annotations. To evaluate these annotations, we computed and compared the results obtained using the commonly advocated inter-annotator agreement coefficients like κ (Cohen, 1960) or π (Scott, 1955) and analyzed to which extent they are relevant for our data. We also studied the different weighting computations applicable to κω (Cohen, 1968) and α (Krippendorff, 1980, 2004) and defined a way to compute distances between categories based on the produced annotations. We then propose a first approach to estimate the bias introduced by prevalence. |
|
|
|
|
|
|
|
|
|
|
|
| 1 : | Institut de l'information scientifique et technique (INIST) |
|
|
|
CNRS : UPS76 |
|
|
| 2 : | Laboratoire d'informatique de Paris-nord (LIPN) |
|
|
|
CNRS : UMR7030 – Université Paris XIII - Paris Nord |
|
|
|
|
|
|
|
|
|
| SRDI / INIST-CNRS |
|
|
|
|
| Domaine | : | Informatique/Traitement du texte et du document |
|
|
| Manual annotation – evaluation – inter-annotator agreement. |
|
|
|
| Liste des fichiers attachés à ce document : | |||||
|
|||||
|
|
|
| hal-00484265, version 2 | |
| http://hal.archives-ouvertes.fr/hal-00484265 | |
| oai:hal.archives-ouvertes.fr:hal-00484265 | |
| Contributeur : Karën Fort | |
| Soumis le : Jeudi 27 Mai 2010, 10:18:26 | |
| Dernière modification le : Jeudi 27 Mai 2010, 12:17:02 | |
