Skip to Main content Skip to Navigation
Conference papers

Reference interpretation in a multimodal environment combining speech and gesture

Nadia Bellalem 1 Laurent Romary 1
CRIN - Centre de Recherche en Informatique de Nancy, UHP - Université Henri Poincaré - Nancy 1, Université Nancy 2, INPL - Institut National Polytechnique de Lorraine
Abstract : The study presented in this paper is dedicated to the integration of pointing gestures within a task oriented man-machine dialogue system. From our point of view, it is natural to see this problem as a sub-part of the more general study of reference in a dialogue since we took the option to limit the analysis of gestures to those explicitly associated with speech and more specifically for singling out objects in the task. Even under these hypotheses, understanding the global message resulting from the combinaison of speech and gesture implies that a precise analysis of the gestural trajectory should be done. This analysis may be split up into two main steps. The first, which we may call structural, is centred upon the shape of the gestural trajectory to mark the meaningful parts to which a specific designation role will be given. The second step has to do with the contextual interpretation of the gesture for which one has to take into account a) the features of the application and the way it is visualized and b) the oral dialogue and more specifically the instructional content of the referring expression accompanying the gesture.
Document type :
Conference papers
Complete list of metadata

Cited literature [8 references]  Display  Hide  Download
Contributor : Laurent Romary <>
Submitted on : Tuesday, August 5, 2014 - 3:22:44 PM
Last modification on : Friday, February 26, 2021 - 3:26:01 PM
Long-term archiving on: : Monday, November 17, 2014 - 3:41:55 PM


Files produced by the author(s)


  • HAL Id : hal-00521585, version 1



Nadia Bellalem, Laurent Romary. Reference interpretation in a multimodal environment combining speech and gesture. First International Workshop on Intelligence and Multimodality in Multimedia Interfaces, 1995, Edinburgh, United Kingdom. 5 p. ⟨hal-00521585⟩



Record views


Files downloads