Towards soundpainting gesture recognition

Thomas Pellegrini (1) , Patrice Guyot (1) , Baptiste Angles (2) , Christophe Mollaret (1) , Christophe Mangou (3)

1 IRIT-SAMoVA - Équipe Structuration, Analyse et MOdélisation de documents Vidéo et Audio
2 IRIT-STORM - Structural Models and Tools in Computer Graphics
3 Ensemble Amalgammes (orchestre)

In this article, we describe our recent research activities on gesture recognition for soundpainting applications. Soundpainting is a multidisciplinary live composing sign language for musicians, actors, dancers, and visual artists. These gestures are produced by a soundpainter, which plays the role of a conductor, in order to lead a live performance. Soundpainting gestures are normalized and well defined, thus they are a very interesting case study in automatic gesture recognition. We describe a first gesture recognition system based on hidden Markov Models. We also report on the creation of a pilot corpus of soundpainting RGB/depth videos. The use of a computer could have many interesting applications listed in the paper. These applications are not limited to live performance, in which the computer would act as a performer. It could also help to investigate the balance between improvisation and planned creation in the particular context of soundpainting.

Mots clés: Gesture recognition; soundpainting; Hidden Markov Models; music; Interactive systems

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Musique, musicologie et arts de la scène

Thomas Pellegrini, Patrice Guyot, Baptiste Angles, Christophe Mollaret, Christophe Mangou. Towards soundpainting gesture recognition. 9th Audio Mostly: A Conference on Interaction With Sound (AM 2014), Oct 2014, Aalborg, France. pp.1-6, ⟨10.1145/2636879.2636899⟩. ⟨hal-03684147⟩