Controlling a Mobile Robot with Natural Commands based on Voice and Gesture

A. R. Fardana (1) , S. Jain (1) , Igor Jovančević (1) , Y. Suri (1) , C. Morand (1) , N. M. Robertson (1)

1 HWU - Heriot-Watt University [Edinburgh]

This paper presents a real-time system for the control of a small mobile robot using combined audio (speech) and video (gesture) commands. Commercial hardware is used based on open-source code. Gesture is recognised using a dynamic time warp (DTW) algorithm using skeleton points derived from the RGB-D camera of the Kinect sensor. We present the integration of a faster parallel version of the DTW algorithm. Speech is recognised using a reduced-vocabulary HMM toolkit. Audio beam forming is exploited for localisation of the person relative to the robot. Separate commands are passed to a fusion centre which resolves conflicting and complementary instructions. This means complex commands such as "go there" and "come here" may be recognised without a complex scene model. We provide comprehensive analysis of the performance in an indoor, reverberant environment.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Robotique [cs.RO]

A. R. Fardana, S. Jain, Igor Jovančević, Y. Suri, C. Morand, et al.. Controlling a Mobile Robot with Natural Commands based on Voice and Gesture. IEEE International Conference on Robotics and Automation (ICRA) - Workshop on Human Robot Interaction for Assistance and Industrial Robots, May 2013, Karlsruhe, Germany. ⟨hal-02996999⟩