Inter-annotator agreement for a speech corpus pronounced by French and German language learners

Odile Mella 1 Dominique Fohr 1 Anne Bonneau 1
1 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : This paper presents the results of an investigation of inter-annotator agreement for the non-native and native French part of the IFCASL corpus. This large bilingual speech corpus for French and German language learners was manually annotated by several annotators. This manual annotation is the starting point which will be used both to improve the automatic segmentation algorithms and derive diagnosis and feedback. The agreement is evaluated by comparing the manual alignments of seven annotators to the manual alignment of an expert, for 18 sentences. Whereas results for the presence of the devoicing diacritic show a certain degree of disagreement between the annotators and the expert, there is a very good consistency between annotators and the expert for temporal boundaries as well as insertions and deletions. We find a good overall agreement for boundaries between annotators and expert with a mean deviation of 7.6 ms and 93% of boundaries within 20 ms.
Document type :
Conference papers
Liste complète des métadonnées

Cited literature [10 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01185194
Contributor : Odile Mella <>
Submitted on : Wednesday, August 19, 2015 - 12:13:36 PM
Last modification on : Saturday, December 22, 2018 - 5:26:06 PM
Document(s) archivé(s) le : Friday, November 20, 2015 - 10:44:12 AM

File

slate_agreement_v7.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01185194, version 1

Collections

Citation

Odile Mella, Dominique Fohr, Anne Bonneau. Inter-annotator agreement for a speech corpus pronounced by French and German language learners. Workshop on Speech and Language Technology in Education, ISCA Special Interest Group (SIG) on Speech and Language Technology in Education, Sep 2015, Leipzig, Germany. ⟨hal-01185194⟩

Share

Metrics

Record views

382

Files downloads

187