Modular resource development and diagnostic evaluation framework for fast NLP system improvement

Gaël de Chalendar (1) , Damien Nouvel (2)

1 LIST (CEA) - Laboratoire d'Intégration des Systèmes et des Technologies
2 LIFAT - Laboratoire d'Informatique Fondamentale et Appliquée de Tours

Natural Language Processing systems are large-scale softwares, whose development involves many man-years of work, in terms of both coding and resource development. Given a dictionary of 110k lemmas, a few hundred syntactic analysis rules, 20k ngrams matrices and other resources, what will be the impact on a syntactic analyzer of adding a new possible category to a given verb? What will be the consequences of a new syntactic rules addition? Any modification may imply, besides what was expected, unforeseeable side-effects and the complexity of the system makes it difficult to guess the overall impact of even small changes. We present here a framework designed to effectively and iteratively improve the accuracy of our linguistic analyzer LIMA by iterative refinements of its linguistic resources. These improvements are continuously assessed by evaluating the analyzer performance against a reference corpus. Our first results show that this framework is really helpful towards this goal.

Mots clés: Evaluation parsing; syntax

Domaines: Informatique et langage [cs.CL]

Gaël de Chalendar, Damien Nouvel. Modular resource development and diagnostic evaluation framework for fast NLP system improvement. North American Chapter of the Association for Computational Linguistics - Human Language Technologies 2009, May 2009, Boulder, United States. ⟨hal-00568775⟩