Skip to Main content Skip to Navigation
Journal articles

Pattern Recognition Letters Learning Off-line vs. On-line Models of Interactive Multimodal Behaviors with Recurrent Neural Networks

Duc Canh Nguyen 1 Gérard Bailly 1 Frédéric Elisei 2, 1
2 GIPSA-Services - GIPSA-Services
GIPSA-lab - Grenoble Images Parole Signal Automatique
Abstract : Human interactions are driven by multi-level perception-action loops. Interactive behavioral models are typically built using rule-based methods or statistical approaches such as Hidden Markov Model (HMM), Dynamic Bayesian Network (DBN), etc. In this paper, we present the multimodal interactive data and our behavioral model based on recurrent neural networks, namely Long-Short Term Memory (LSTM) and Bidirectional LSTM (BiLSTM) models. Speech, gaze and gestures of two subjects involved in a collaborative task are here jointly modeled. The results show that the proposed deep neural networks are more effective than the conventional statistical methods in generating appropriate overt actions for both on-line and off-line prediction tasks.
Document type :
Journal articles
Complete list of metadata

Cited literature [30 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01609535
Contributor : Gérard Bailly Connect in order to contact the contributor
Submitted on : Tuesday, October 3, 2017 - 5:28:47 PM
Last modification on : Wednesday, November 3, 2021 - 7:49:25 AM

File

dan_PRL2017_R2_v2.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Duc Canh Nguyen, Gérard Bailly, Frédéric Elisei. Pattern Recognition Letters Learning Off-line vs. On-line Models of Interactive Multimodal Behaviors with Recurrent Neural Networks. Pattern Recognition Letters, Elsevier, 2017, 100, pp.29-36. ⟨10.1016/j.patrec.2017.09.033⟩. ⟨hal-01609535⟩

Share

Metrics

Les métriques sont temporairement indisponibles