Towards Real-Time Physical Human-Robot Interaction Using Skeleton Information and Hand Gestures

Osama Mazhar (1) , Sofiane Ramdani (1) , Benjamin Navarro (1) , Robin Passama (1) , Andrea Cherubini (1)

1 IDH - Interactive Digital Humans

For successful physical human-robot interaction, the capability of a robot to understand its environment is imperative. More importantly, the robot should extract from the human operator as much information as possible. A reliable 3D skeleton extraction is essential for a robot to predict the intentions of the operator while s/he moves toward the robot or performs a meaningful gesture. For this purpose, we have integrated a time-of-flight depth camera with a state-of-the-art 2D skeleton extraction library namely Openpose, to obtain 3D skeletal joint coordinates reliably. We have also developed a robust and rotation invariant (in the coronal plane)hand gesture detector using a convolutional neural network. At run time (after having been trained)the detector does not require any pre-processing of the hand images. A complete pipeline for skeleton extraction and hand gesture recognition is developed and employed for real-time physical human-robot interaction, demonstrating the promising capability of the designed framework. This work establishes a firm basis and will be extended for the development of intelligent human intention detection in physical human-robot interaction scenarios, to efficiently recognize a variety of static as well as dynamic gestures.

Domaines: Robotique [cs.RO]; Automatique / Robotique; Traitement du signal et de l'image [eess.SP]

Osama Mazhar, Sofiane Ramdani, Benjamin Navarro, Robin Passama, Andrea Cherubini. Towards Real-Time Physical Human-Robot Interaction Using Skeleton Information and Hand Gestures. IROS: Intelligent Robots and Systems, Oct 2018, Madrid, Spain. pp.1-6, ⟨10.1109/IROS.2018.8594385⟩. ⟨hal-01734739⟩