Learning 3D Navigation Protocols on Touch Interfaces with Cooperative Multi-Agent Reinforcement Learning

Quentin Debard 1 Jilles Steeve Dibangoye 2 Stephane Canu 3, 4 Christian Wolf 1
1 imagine - Extraction de Caractéristiques et Identification
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
2 CHROMA - Robots coopératifs et adaptés à la présence humaine en environnements dynamiques
Inria Grenoble - Rhône-Alpes, CITI - CITI Centre of Innovation in Telecommunications and Integration of services
3 LITIS - Laboratoire d'Informatique, de Traitement de l'Information et des Systèmes
4 DocApp - LITIS - Equipe Apprentissage
LITIS - Laboratoire d'Informatique, de Traitement de l'Information et des Systèmes
Abstract : Using touch devices to navigate in virtual 3D environments such as computer assisted design (CAD) models or geographical information systems (GIS) is inherently difficult for humans, as the 3D operations have to be performed by the user on a 2D touch surface. This ill-posed problem is classically solved with a fixed and handcrafted interaction protocol, which must be learned by the user.We propose to automatically learn a new interaction protocol allowing to map a 2D user input to 3D actions in virtual environments using reinforcement learning (RL). A fundamental problem of RL methods is the vast amount of interactions often required, which are difficult to come by when humans are involved. To overcome this limitation, we make use of two collaborative agents. The first agent models the human by learning to perform the 2D finger trajectories. The second agent acts as the interaction protocol, interpreting and translating to 3D operations the 2D finger trajectories from the first agent. We restrict the learned 2D trajectories to be similar to a training set of collected human gestures by first performing state representation learning, prior to reinforcement learning. This state representation learning is addressed by projecting the gestures into a latent space learned by a variational auto encoder (VAE).
Keywords : H-C Interfaces Multi-Agent Deep Reinforcement Learning
Document type :
Conference papers
Complete list of metadatas

Cited literature [26 references]  Display  Hide  Download
https://hal.archives-ouvertes.fr/hal-02302554
Contributor : Quentin Debard <>
Submitted on : Tuesday, October 1, 2019 - 4:57:34 PM
Last modification on : Tuesday, November 19, 2019 - 2:38:59 AM

File

sub_213.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02302554, version 1

Collections

EC-LYON | LITIS | CNRS | UNIV-LYON1 | INSA-LYON | INRIA | COMUE-NORMANDIE | INSA-ROUEN | LIRIS | UNIROUEN | UNILEHAVRE | UNIV-LYON2 | INSA-GROUPE | CITI | UDL

Citation

Quentin Debard, Jilles Steeve Dibangoye, Stephane Canu, Christian Wolf. Learning 3D Navigation Protocols on Touch Interfaces with Cooperative Multi-Agent Reinforcement Learning. ECML PKDD 2019 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Sep 2019, Würzburg, Germany. pp.1-16. ⟨hal-02302554⟩

Export

BibTeX TEI DC DCterms EndNote

Share

Metrics

Record views

65

Files downloads

22

Contact

Informations

CCSD