Creating an interactive human/agent loop using multimodal recurrent neural networks

Jieyeon Woo (1, 2) , Catherine I Pelachaud (1, 2) , Catherine Achard (1, 2)

1 ISIR - Institut des Systèmes Intelligents et de Robotique
2 PIROS - Perception, Interaction, Robotique sociales

This paper presents a description of ongoing research that aims to improve the interaction between human and Embodied Conversational Agent (ECA). The main idea is to model the interactive loop between human and agent such as the virtual agent can continuously adapt its behavior according to one's partner. This work, based on recurrent neural network, focuses on non-verbal behavior generation and presents several scientific locks like the multimodality, the intra-personal temporality of multimodal signals or the temporality between partner's social cues. The modeling will be done using the NOXI database containing natural human/human interactions and the nonverbal behavior generation will be tested on the GRETA platform that simulates virtual agents.

Mots clés: deep learning; embodied conversational agent (ECA)

Domaines: Informatique; Neurosciences; Psychologie; Psychologie; Sociologie

Jieyeon Woo, Catherine I Pelachaud, Catherine Achard. Creating an interactive human/agent loop using multimodal recurrent neural networks. WACAI 2021, Centre National de la Recherche Scientifique [CNRS], Oct 2021, Saint Pierre d'Oléron, France. ⟨hal-03377542⟩