Document-level Neural MT: A Systematic Comparison

António V Lopes (1) , M Amin Farajian (1) , Rachel Bawden (2) , Michael Zhang (3) , André F T Martins (1)

1 Unbabel
2 School of Informatics [Edimbourg]
3 University of Washington [Seattle]

In this paper we provide a systematic comparison of existing and new document-level neural machine translation solutions. As part of this comparison, we introduce and evaluate a document-level variant of the recently proposed Star Transformer architecture. In addition to using the traditional metric BLEU, we report the accuracy of the models in handling anaphoric pronoun translation as well as coherence and cohesion using contrastive test sets. Finally, we report the results of human evaluation in terms of Multidimensional Quality Metrics (MQM) and analyse the correlation of the results obtained by the automatic metrics with human judgments.

Mots clés: machine translation; document-level; context; evaluation

Domaines: Informatique et langage [cs.CL]

António V Lopes, M Amin Farajian, Rachel Bawden, Michael Zhang, André F T Martins. Document-level Neural MT: A Systematic Comparison. 22nd Annual Conference of the European Association for Machine Translation, 2020, Lisboa, Portugal. pp.225-234. ⟨hal-02900686⟩