Pattern Recognition Letters Learning Off-line vs. On-line Models of Interactive Multimodal Behaviors with Recurrent Neural Networks

Duc Canh Nguyen 1 Gérard Bailly 1 Frédéric Elisei 2, 1
1 GIPSA-CRISSP - CRISSP
GIPSA-DPC - Département Parole et Cognition
2 GIPSA-Services - GIPSA-Services
GIPSA-lab - Grenoble Images Parole Signal Automatique
Abstract : Human interactions are driven by multi-level perception-action loops. Interactive behavioral models are typically built using rule-based methods or statistical approaches such as Hidden Markov Model (HMM), Dynamic Bayesian Network (DBN), etc. In this paper, we present the multimodal interactive data and our behavioral model based on recurrent neural networks, namely Long-Short Term Memory (LSTM) and Bidirectional LSTM (BiLSTM) models. Speech, gaze and gestures of two subjects involved in a collaborative task are here jointly modeled. The results show that the proposed deep neural networks are more effective than the conventional statistical methods in generating appropriate overt actions for both on-line and off-line prediction tasks.
Document type :
Journal articles
Complete list of metadatas

Cited literature [30 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01609535
Contributor : Gérard Bailly <>
Submitted on : Tuesday, October 3, 2017 - 5:28:47 PM
Last modification on : Wednesday, June 26, 2019 - 10:58:03 AM

File

dan_PRL2017_R2_v2.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Duc Canh Nguyen, Gérard Bailly, Frédéric Elisei. Pattern Recognition Letters Learning Off-line vs. On-line Models of Interactive Multimodal Behaviors with Recurrent Neural Networks. Pattern Recognition Letters, Elsevier, 2017, 100, pp.29-36. ⟨10.1016/j.patrec.2017.09.033⟩. ⟨hal-01609535⟩

Share

Metrics

Record views

286

Files downloads

39