Skip to Main content Skip to Navigation
Journal articles

A Real-time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection

Osama Mazhar 1 Benjamin Navarro 1 Sofiane Ramdani 1 Robin Passama 1 Andrea Cherubini 1
1 IDH - Interactive Digital Humans
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier
Abstract : In the light of factories of the future, we present a reliable framework for real-time safe physical human-robot collaboration using static hand gestures. To ensure productive and safe interaction between robot and human coworkers, it is imperative that the robot extracts the essential information about the human coworker. We address this by designing a framework for safe and intuitive robot programming based on hand gesture recognition. First, the OpenPose library is integrated with Microsoft Kinect V2, to obtain a 3D estimation of the human skeleton. With the help of 10 volunteers, we record an image dataset of alphanumeric static hand gestures, taken from the American Sign Language. We name our dataset as OpenSign and release it to the community for bench-marking. The Inception-v3 convolutional neural network is adapted to train the hand gesture detector. To augment the data for training a hand gesture detector, we use OpenPose to localize the hands in the dataset images and segment the backgrounds of hand images using the Kinect depth map. Then, the backgrounds are substituted with random patterns and indoor architecture templates. Fine-tuning of Inception V3 is performed in three phases, to achieve validation accuracy of 99.1% and test accuracy of 98.9%. An asynchronous integration of image acquisition and hand gesture detection is performed to ensure real-time detection of hand gestures. Finally, the proposed framework is integrated in our physical human-robot interaction library OpenPHRI. Using OpenPHRI, we validate the performance of the proposed framework through a complete teaching by demonstration experiment with a robotic manipulator.
Complete list of metadatas

Cited literature [74 references]  Display  Hide  Download
Contributor : Andrea Cherubini <>
Submitted on : Monday, May 20, 2019 - 3:12:58 PM
Last modification on : Thursday, January 30, 2020 - 1:15:58 AM


Files produced by the author(s)




Osama Mazhar, Benjamin Navarro, Sofiane Ramdani, Robin Passama, Andrea Cherubini. A Real-time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection. Robotics and Computer-Integrated Manufacturing, Elsevier, 2019, 60, pp.34-48. ⟨10.1016/j.rcim.2019.05.008⟩. ⟨hal-01823338v2⟩



Record views


Files downloads