Combining HoloLens and Leap-Motion for Free Hand-Based 3D Interaction in MR Environments

Fakhreddine Ababsa (1) , Junhui He (2) , Jean-Rémy Chardonnet (1)

1 LISPEN - Laboratoire d’Ingénierie des Systèmes Physiques et Numériques
2 ENSEA - Ecole Nationale Supérieure de l'Electronique et de ses Applications

The ability to interact with virtual objects using gestures would allow users to improve their experience in Mixed Reality (MR) environments, especially when they use AR headsets. Today, MR head-mounted displays like the HoloLens integrate hand gesture based interaction allowing users to take actions in MR environments. However, the proposed interactions remain limited. In this paper, we propose to combine a Leap Motion Controller (LMC) with a HoloLens in order to improve gesture interaction with virtual objects. Two main issues are presented: an interactive calibration procedure for the coupled HoloLens-LMC device and an intuitive hand-based interaction approach using LMC data in the HoloLens environment. A set of first experiments was carried out to evaluate the accuracy and the usability of the proposed approach.

Mots clés: Mixed reality; Calibration; Natural interaction

Domaines: Synthèse d'image et réalité virtuelle [cs.GR]; Interface homme-machine [cs.HC]

Fakhreddine Ababsa, Junhui He, Jean-Rémy Chardonnet. Combining HoloLens and Leap-Motion for Free Hand-Based 3D Interaction in MR Environments. 7th International Conference on Augmented Reality, Virtual Reality, and Computer Graphics, Sep 2020, Lecce, Italy. pp.315-327, ⟨10.1007/978-3-030-58465-8_24⟩. ⟨hal-03001870⟩