Survey on evaluation methods for dialogue systems

Jan Deriu , Alvaro Rodrigo , Arantxa Otegi , Guillermo Echegoyen , Sophie Rosset (1) , Eneko Agirre , Mark Cieliebak

1 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.

Mots clés: Dialogue systems · Evaluation metrics · Discourse model · Conversational AI · Chatbots

Domaines: Informatique [cs]; Informatique et langage [cs.CL]

Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, et al.. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 2020, 54 (1), pp.755-810. ⟨10.1007/s10462-020-09866-x⟩. ⟨hal-03006231⟩