Weighted Krippendorff's alpha is a more reliable metrics for multi- coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation.

Abstract : The question of data reliability is of first importance to assess the quality of manually annotated corpora. Although Cohen ' s κ is the prevailing reliability measure used in NLP, alternative statistics have been proposed. This paper presents an experimental study with four measures (Cohen's κ, Scott's π, binary and weighted Krippendorff ' s α) on three tasks: emotion, opinion and coreference annotation. The reported studies investigate the factors of influence (annotator bias, category prevalence, number of coders, number of categories) that should affect reliability estimation. Results show that the use of a weighted measure re- stricts this influence on ordinal annotations. They suggest that weighted α is the most reliable metrics for such an annotation scheme.
Type de document :
Communication dans un congrès
EACL 2014, Apr 2014, Gotenborg, Sweden. 10 p., 2014
Liste complète des métadonnées

Littérature citée [40 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-00998303
Contributeur : Jeanne Villaneau <>
Soumis le : dimanche 1 juin 2014 - 09:04:05
Dernière modification le : mercredi 16 mai 2018 - 11:24:04
Document(s) archivé(s) le : lundi 1 septembre 2014 - 10:40:34

Fichier

EACL_2014_Data_reliability_Ant...
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00998303, version 1

Citation

Jean-Yves Antoine, Jeanne Villaneau, Anaïs Lefeuvre. Weighted Krippendorff's alpha is a more reliable metrics for multi- coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation.. EACL 2014, Apr 2014, Gotenborg, Sweden. 10 p., 2014. 〈hal-00998303〉

Partager

Métriques

Consultations de la notice

564

Téléchargements de fichiers

953