Referring to Objects with Spoken and Haptic Modalities

Frédéric Landragin 1 Nadia Bellalem 1 Laurent Romary 1
1 LANGUE ET DIALOGUE - Human-machine dialogue with a significant language component
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : The gesture input modality considered in multimodal dialogue systems is mainly reduced to pointing or manipulating actions. With an approach based on the spontaneous character of the communication, the treatment of such actions involves many processes. Without any constraints, the user may use gesture in association with speech, and may exploit the visual context peculiarities, guiding his articulation of gesture trajectories and his choices of words. The semantic interpretation of multimodal utterances also becomes a complex problem, taking into account varieties of referring expressions, varieties of gestural trajectories, structural parameters from the visual context, and also directives from a specific task. Following the spontaneous approach, we propose to give the maximal understanding capabilities to dialogue systems, to ensure that various interaction modes must be taken into account. Considering the development of haptic sense devices (as PHANToM) which increase the capabilities of sensations, particularly tactile and kinesthetic ones, we propose to explore a new domain of research concerning the integration of haptic gesture into multimodal dialogue systems, in terms of its possible associations with speech for objects reference and manipulation. We focus in this paper on the compatibility between haptic gesture and multimodal reference models, and on the consequences of processing this new modality on intelligent system architectures, which is not yet enough studied from a semantic point of view.
Type de document :
Communication dans un congrès
IEEE CS Press. Fourth IEEE International Conference on Multimodal Interfaces, 2002, Pittsburgh, PA, United States. pp.99-104, 2002
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-00112772
Contributeur : Frédéric Landragin <>
Soumis le : jeudi 9 novembre 2006 - 16:21:46
Dernière modification le : jeudi 11 janvier 2018 - 06:19:48
Document(s) archivé(s) le : mardi 6 avril 2010 - 19:19:50

Identifiants

  • HAL Id : hal-00112772, version 1

Collections

Citation

Frédéric Landragin, Nadia Bellalem, Laurent Romary. Referring to Objects with Spoken and Haptic Modalities. IEEE CS Press. Fourth IEEE International Conference on Multimodal Interfaces, 2002, Pittsburgh, PA, United States. pp.99-104, 2002. 〈hal-00112772〉

Partager

Métriques

Consultations de la notice

264

Téléchargements de fichiers

135