Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem

Erwan Moreau (1) , Carl Vogel (1)

1 School of Computer Science and Statistics [Dublin]

In this paper we address the question of the reliability of the predictions made by MT Quality Estimation (QE) systems. In particular, we show that standard supervised QE systems, usually trained to minimize MAE, make serious mistakes at predicting the quality of the sentences in the tails of the quality range. We describe the problem and propose several experiments to clarify their causes and effects. We use the WMT12 and WMT13 QE Shared Task datasets to prove that our claims hold in general and are not specific to a dataset or a system.

Mots clés: Quality estimation; Machine Translation

Domaines: Traitement du texte et du document

Erwan Moreau, Carl Vogel. Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem. Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics, Aug 2014, Dublin, Ireland. pp.2205--2216. ⟨hal-01066394⟩