Human Annotation of ASR Error Regions: is "gravity" a Sharable Concept for Human Annotators?

Daniel Luzzati 1 Cyril Grouin 2 Ioana Vasilescu 2 Martine Adda-Decker 3, 2 Eric Bilinski 2 Nathalie Camelin 1 Juliette Kahn 4 Carole Lailler 1 Lori Lamel 2 Sophie Rosset 2
1 LIUM - Laboratoire d'Informatique de l'Université du Mans
2 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur
3 LPP - LPP - Laboratoire de Phonétique et Phonologie - UMR 7018
4 LNE
Abstract : This paper is concerned with human assessments of the severity of errors in ASR outputs. We did not design any guidelines so that each annotator involved in the study could consider the " seriousness " of an ASR error using their own scientific background. Eight human annotators were involved in an annotation task on three distinct corpora, one of the corpora being annotated twice, hiding this annotation in duplicate to the annotators. None of the computed results (inter-annotator agreement, edit distance, majority annotation) allow any strong correlation between the considered criteria and the level of seriousness to be shown, which underlines the difficulty for a human to determine whether a ASR error is serious or not.
Keywords : ASR Seriousness Errors Speech Recognition Annotation
Type de document :
Poster
Ninth International Conference on Language Resources and Evaluation (LREC'14), May 2014, Reykjavik, Iceland. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pp.3050-3056, 2014, 〈http://lrec2014.lrec-conf.org/en/〉
Liste complète des métadonnées
https://hal.archives-ouvertes.fr/hal-01134802
Contributeur : Gwénaëlle Lo Bue <>
Soumis le : mardi 24 mars 2015 - 12:01:33
Dernière modification le : mardi 20 novembre 2018 - 14:04:02

Identifiants

  • HAL Id : hal-01134802, version 1

Collections

UNIV-LEMANS | UNIV-PARIS3 | USPC | LIUM | LIUM-LST | LPP | LIMSI

Citation

Daniel Luzzati, Cyril Grouin, Ioana Vasilescu, Martine Adda-Decker, Eric Bilinski, et al.. Human Annotation of ASR Error Regions: is "gravity" a Sharable Concept for Human Annotators?. Ninth International Conference on Language Resources and Evaluation (LREC'14), May 2014, Reykjavik, Iceland. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pp.3050-3056, 2014, 〈http://lrec2014.lrec-conf.org/en/〉. 〈hal-01134802〉

Exporter

BibTeX TEI DC DCterms EndNote

Partager

Métriques

Consultations de la notice

228

Contact

support.ccsd.cnrs.fr
hal.support@ccsd.cnrs.fr
Logo CCSD