Towards Real-Time Physical Human-Robot Interaction Using Skeleton Information and Hand Gestures

Osama Mazhar 1 Sofiane Ramdani 1 Benjamin Navarro 1 Robin Passama 1 Andrea Cherubini 1
1 IDH - Interactive Digital Humans
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier
Résumé : For successful physical human-robot interaction, the capability of a robot to understand its environment is imperative. More importantly, the robot should extract from the human operator as much information as possible. A reliable 3D skeleton extraction is essential for a robot to predict the intentions of the operator while s/he moves toward the robot or performs a meaningful gesture. For this purpose, we have integrated a time-of-flight depth camera with a state-of-the-art 2D skeleton extraction library namely Openpose, to obtain 3D skeletal joint coordinates reliably. We have also developed a robust and rotation invariant (in the coronal plane)hand gesture detector using a convolutional neural network. At run time (after having been trained)the detector does not require any pre-processing of the hand images. A complete pipeline for skeleton extraction and hand gesture recognition is developed and employed for real-time physical human-robot interaction, demonstrating the promising capability of the designed framework. This work establishes a firm basis and will be extended for the development of intelligent human intention detection in physical human-robot interaction scenarios, to efficiently recognize a variety of static as well as dynamic gestures.
Complete list of metadatas

Cited literature [16 references]  Display  Hide  Download
Contributor : Andrea Cherubini <>
Submitted on : Sunday, March 25, 2018 - 5:45:01 AM
Last modification on : Wednesday, May 22, 2019 - 5:50:47 PM
Long-term archiving on : Monday, September 10, 2018 - 9:15:57 PM


Files produced by the author(s)




Osama Mazhar, Sofiane Ramdani, Benjamin Navarro, Robin Passama, Andrea Cherubini. Towards Real-Time Physical Human-Robot Interaction Using Skeleton Information and Hand Gestures. IROS: Intelligent Robots and Systems, Oct 2018, Madrid, Spain. pp.1-6, ⟨10.1109/IROS.2018.8594385⟩. ⟨hal-01734739⟩



Record views


Files downloads