Skip to Main content Skip to Navigation

Designing a virtual patient dialogue system based on terminology-rich resources: challenges and evaluation

Abstract : Virtual patient software allows health professionals to practice their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we de- signed a patient record model, a knowledge model for the task, and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowl- edge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non- medical evaluators) interact with the system and use 35 cases from 18 specialities. We con- ducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8 per cent. Dialogue management provided on average 74.3 (±9.5) per cent of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All eval- uated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8 per cent of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-02358021
Contributor : Limsi Publications <>
Submitted on : Monday, November 11, 2019 - 10:47:40 AM
Last modification on : Monday, February 10, 2020 - 6:14:09 PM

Identifiers

  • HAL Id : hal-02358021, version 1

Citation

Leonardo Campillos-Llanos, Catherine Thomas, Eric Bilinski, Pierre Zweigenbaum, Sophie Rosset. Designing a virtual patient dialogue system based on terminology-rich resources: challenges and evaluation. Natural Language Engineering, Cambridge University Press (CUP), 2019, pp.1-38. ⟨hal-02358021⟩

Share

Metrics

Record views

37