A Real-time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2018

A Real-time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection

Osama Mazhar
Benjamin Navarro
Sofiane Ramdani
Robin Passama
Andrea Cherubini

Résumé

In the light of factories of the future, we present a reliable framework for real-time safe physical human-robot collaboration using static hand gestures. To ensure productive and safe interaction between robot and human coworkers, it is imperative that the robot extracts the essential information about the human coworker. We address this by designing a framework for safe and intuitive robot programming based on hand gesture recognition. First, the OpenPose library is integrated with Microsoft Kinect V2, to obtain a 3D estimation of the human skeleton. With the help of 10 volunteers, we record an image dataset of alphanumeric static hand gestures, taken from the American Sign Language. We name our dataset as OpenSign and release it to the community for bench-marking. The Inception-v3 convolutional neural network is adapted to train the hand gesture detector. To augment the data for training a hand gesture detector, we use OpenPose to localize the hands in the dataset images and segment the backgrounds of hand images using the Kinect depth map. Then, the backgrounds are substituted with random patterns and indoor architecture templates. Fine-tuning of Inception V3 is performed in three phases, to achieve validation accuracy of 99.1% and test accuracy of 98.9%. An asynchronous integration of image acquisition and hand gesture detection is performed to ensure real-time detection of hand gestures. Finally, the proposed framework is integrated in our physical human-robot interaction library OpenPHRI. Using OpenPHRI, we validate the performance of the proposed framework through a complete teaching by demonstration experiment with a robotic manipulator.
Fichier principal
Vignette du fichier
mazhar-rcim_submitted_hal.pdf (21.99 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01823338 , version 1 (26-06-2018)
hal-01823338 , version 2 (20-05-2019)

Identifiants

  • HAL Id : hal-01823338 , version 1

Citer

Osama Mazhar, Benjamin Navarro, Sofiane Ramdani, Robin Passama, Andrea Cherubini. A Real-time Human-Robot Interaction Framework with Robust Background Invariant Hand Gesture Detection. 2018. ⟨hal-01823338v1⟩
648 Consultations
624 Téléchargements

Partager

Gmail Facebook X LinkedIn More