Skip to Main content Skip to Navigation
Conference papers

The Role of Gesture in Multimodal Referring Actions

Frédéric Landragin 1
1 LANGUE ET DIALOGUE - Human-machine dialogue with a significant language component
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : When deictic gestures are produced on a touch screen, they can take forms which can lead to several sorts of ambiguities. Considering that the resolution of a multimodal reference requires the identification of the referents and of the context (“reference domain”) from which these referents are extracted, we focus on the linguistic, gestural, and visual clues that a dialogue system may exploit to comprehend the referring intention. We explore the links between words, gestures and perceptual groups, doing so in terms of the clues that delimit the reference domain. We also show the importance of taking the domain into account for dialogue management, particularly for the comprehension of further utterances, when they seem to implicitly use a pre-existing restriction to a subset of objects. We propose a strategy of multimodal reference resolution based on this notion of reference domain, and we illustrate its efficiency with prototypic examples built from a study of significant referring situations extracted from a corpus. We give at last the future directions of our works concerning some linguistic and task aspects that are not integrated here.
Complete list of metadata

Cited literature [9 references]  Display  Hide  Download
Contributor : Frédéric Landragin Connect in order to contact the contributor
Submitted on : Thursday, November 9, 2006 - 4:28:31 PM
Last modification on : Friday, February 26, 2021 - 3:28:03 PM
Long-term archiving on: : Tuesday, April 6, 2010 - 10:03:08 PM


  • HAL Id : hal-00112778, version 1



Frédéric Landragin. The Role of Gesture in Multimodal Referring Actions. Fourth IEEE International Conference on Multimodal Interfaces, 2002, Pittsburgh, PA, United States. pp.173-178. ⟨hal-00112778⟩



Record views


Files downloads