Fine-grained Linguistic Evaluation of Question Answering Systems

Sarra El Ayari (1, 2) , Brigitte Grau (2, 3) , Anne-Laure Ligozat (2, 3)

1 UP11 - Université Paris-Sud - Paris 11
2 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur
3 ENSIIE - Ecole Nationale Supérieure d'Informatique pour l'Industrie et l'Entreprise

Question answering systems are complex systems using natural language processing. Some evaluation campaigns are organized to evaluate such systems in order to propose a classification of systems based on final results (number of correct answers). Nevertheless, teams need to evaluate more precisely the results obtained by their systems if they want to do a diagnostic evaluation. There are no tools or methods to do these evaluations systematically. We present REVISE, a tool for glass box evaluation based on diagnostic of question answering system results.

Domaines: Linguistique; Informatique

Sarra El Ayari, Brigitte Grau, Anne-Laure Ligozat. Fine-grained Linguistic Evaluation of Question Answering Systems. Language Resources and Evaluation Conference (LREC), May 2010, Valletta, Malta. pp.2354-2360. ⟨hal-01070043⟩