Reference interpretation in a multimodal environment combining speech and gesture

Nadia Bellalem 1 Laurent Romary 1
1 DIALOGUE - DIALOGUE
CRIN - Centre de Recherche en Informatique de Nancy, UHP - Université Henri Poincaré - Nancy 1, Université Nancy 2, INPL - Institut National Polytechnique de Lorraine
Abstract : The study presented in this paper is dedicated to the integration of pointing gestures within a task oriented man-machine dialogue system. From our point of view, it is natural to see this problem as a sub-part of the more general study of reference in a dialogue since we took the option to limit the analysis of gestures to those explicitly associated with speech and more specifically for singling out objects in the task. Even under these hypotheses, understanding the global message resulting from the combinaison of speech and gesture implies that a precise analysis of the gestural trajectory should be done. This analysis may be split up into two main steps. The first, which we may call structural, is centred upon the shape of the gestural trajectory to mark the meaningful parts to which a specific designation role will be given. The second step has to do with the contextual interpretation of the gesture for which one has to take into account a) the features of the application and the way it is visualized and b) the oral dialogue and more specifically the instructional content of the referring expression accompanying the gesture.
Document type :
Conference papers
Liste complète des métadonnées

Cited literature [8 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-00521585
Contributor : Laurent Romary <>
Submitted on : Tuesday, August 5, 2014 - 3:22:44 PM
Last modification on : Friday, March 22, 2019 - 2:22:12 PM
Document(s) archivé(s) le : Monday, November 17, 2014 - 3:41:55 PM

File

ReferenceInterpretation.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-00521585, version 1

Collections

Citation

Nadia Bellalem, Laurent Romary. Reference interpretation in a multimodal environment combining speech and gesture. First International Workshop on Intelligence and Multimodality in Multimedia Interfaces, 1995, Edinburgh, United Kingdom. 5 p. ⟨hal-00521585⟩

Share

Metrics

Record views

258

Files downloads

68