BEAUFORD: A Benchmark for Evaluation of Formalisation of Definitions in OWL

Cheikh Kacfah Emani (1, 2) , Catarina Ferreira da Silva (2) , Bruno Fies (1) , Parisa Ghodous (2)

1 CSTB - Centre Scientifique et Technique du Bâtiment
2 SOC - Service Oriented Computing

In this paper we present BEAUFORD, a benchmark for methods which aim to provide formal expressions of concepts using the natural language (NL) definition of these concepts. Adding formal expressions of concepts to a given ontology allows reasoners to infer more useful pieces of information or to detect inconsistencies in this given ontology. To the best of our knowledge, BEAUFORD is the first benchmark to tackle this ontology enrichment problem. BEAUFORD allows the breaking down of a given formalisation approach by identifying its key features. In addition, BEAUFORD provides strong mechanisms to evaluate efficiently an approach even in case of ambiguity which is a major challenge in formalisation of NL resources. Indeed, BEAUFORD takes into account the fact that a given NL phrase can be formalised in many ways. Hence, it proposes a suitable specification to represent these multiple formalisations. Taking advantage of this specification, BEAUFORD redefines classical precision and recall and introduces other metrics to take into account the fact that there is not only one unique way to formalise a definition. Finally, BEAUFORD comprises a well-suited dataset to concretely judge of the efficiency of methods of formalisation. Using BEAUFORD, current approaches of formalisation of definitions can be compared accurately using a suitable gold standard.

Mots clés: automatic formalisation; Benchmark; natural language definitions; Short communication: Benchmark

Domaines: Intelligence artificielle [cs.AI]; Traitement du texte et du document; Web

Cheikh Kacfah Emani, Catarina Ferreira da Silva, Bruno Fies, Parisa Ghodous. BEAUFORD: A Benchmark for Evaluation of Formalisation of Definitions in OWL. Open Journal Of Semantic Web, 2015, 2 (1), pp.3-14. ⟨hal-01217090⟩