Better Evaluation of ASR in Speech Translation Context Using Word Embeddings

Abstract : This paper investigates the evaluation of ASR in spoken language translation context. More precisely, we propose a simple extension of WER metric in order to penalize differently substitution errors according to their context using word embeddings. For instance, the proposed metric should catch near matches (mainly morphological variants) and penalize less this kind of error which has a more limited impact on translation performance. Our experiments show that the correlation of the new proposed metric with SLT performance is better than the one of WER. Oracle experiments are also conducted and show the ability of our metric to find better hypotheses (to be translated) in the ASR N-best. Finally, a preliminary experiment where ASR tuning is based on our new metric shows encouraging results. For reproductible experiments, the code allowing to call our modified WER and the corpora used are made available to the research community.
Document type :
Conference papers
Liste complète des métadonnées

Cited literature [23 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01350102
Contributor : Laurent Besacier <>
Submitted on : Friday, July 29, 2016 - 4:09:34 PM
Last modification on : Thursday, April 4, 2019 - 10:18:05 AM
Document(s) archivé(s) le : Sunday, October 30, 2016 - 11:40:24 AM

File

metrics_correlation_asr-smt.pd...
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01350102, version 1

Collections

Citation

Ngoc-Tien Le, Christophe Servan, Benjamin Lecouteux, Laurent Besacier. Better Evaluation of ASR in Speech Translation Context Using Word Embeddings. Interspeech 2016, Sep 2016, San-Francisco, United States. ⟨hal-01350102⟩

Share

Metrics

Record views

492

Files downloads

389