Skip to Main content Skip to Navigation
Conference papers

Referring to Objects with Spoken and Haptic Modalities

Frédéric Landragin 1 Nadia Bellalem 1 Laurent Romary 1
1 LANGUE ET DIALOGUE - Human-machine dialogue with a significant language component
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : The gesture input modality considered in multimodal dialogue systems is mainly reduced to pointing or manipulating actions. With an approach based on the spontaneous character of the communication, the treatment of such actions involves many processes. Without any constraints, the user may use gesture in association with speech, and may exploit the visual context peculiarities, guiding his articulation of gesture trajectories and his choices of words. The semantic interpretation of multimodal utterances also becomes a complex problem, taking into account varieties of referring expressions, varieties of gestural trajectories, structural parameters from the visual context, and also directives from a specific task. Following the spontaneous approach, we propose to give the maximal understanding capabilities to dialogue systems, to ensure that various interaction modes must be taken into account. Considering the development of haptic sense devices (as PHANToM) which increase the capabilities of sensations, particularly tactile and kinesthetic ones, we propose to explore a new domain of research concerning the integration of haptic gesture into multimodal dialogue systems, in terms of its possible associations with speech for objects reference and manipulation. We focus in this paper on the compatibility between haptic gesture and multimodal reference models, and on the consequences of processing this new modality on intelligent system architectures, which is not yet enough studied from a semantic point of view.
Complete list of metadata
Contributor : Frédéric Landragin <>
Submitted on : Thursday, November 9, 2006 - 4:21:46 PM
Last modification on : Friday, February 26, 2021 - 3:28:03 PM
Long-term archiving on: : Tuesday, April 6, 2010 - 7:19:50 PM


  • HAL Id : hal-00112772, version 1



Frédéric Landragin, Nadia Bellalem, Laurent Romary. Referring to Objects with Spoken and Haptic Modalities. Fourth IEEE International Conference on Multimodal Interfaces, 2002, Pittsburgh, PA, United States. pp.99-104. ⟨hal-00112772⟩



Record views


Files downloads