On "Human Parity" and "Super Human Performance" in Machine Translation Evaluation

Thierry Poibeau (1)

1 Lattice - Lattice - Langues, Textes, Traitements informatiques, Cognition - UMR 8094

In this paper, we reassess claims of human parity and super human performance in machine translation. These terms have already been discussed, as well as the evaluation protocols used to achieved these conclusions (human-parity is achieved i) only for a very reduced number of languages, ii) on very specific types of documents and iii) with very literal translations). However, we think it is necessary to consider these questions again. We show that the terms used are themselves problematic, and that human translation involves much more than what is embedded in automatic systems. We also discuss ethical issues related to the way results are presented and advertised. Finally, we claim that a better assessment of human capacities should be put forward and that the goal of replacing humans by machines is not a desirable one.

Mots clés: Neural machine translation; Evaluation; Human parity; Human factors; Ethics

Domaines: Traitement du texte et du document; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]; Linguistique; Informatique; Linguistique; Méthodes et statistiques

Thierry Poibeau. On "Human Parity" and "Super Human Performance" in Machine Translation Evaluation. Language Resource and Evaluation Conference, Jun 2022, Marseille, France. ⟨hal-03738720⟩