A Multimodal Probabilistic Model for Gesture-based Control of Sound Synthesis

Jules Françoise (1) , Norbert Schnell (2) , Frédéric Bevilacqua (2)

1 IRCAM - Institut de Recherche et Coordination Acoustique/Musique
2 Interactions Son Musique Mouvement

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture-sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.

Mots clés: gesture; hmm; music; music performance; sound synthesis

Domaines: Multimédia [cs.MM]; Son [cs.SD]

Jules Françoise, Norbert Schnell, Frédéric Bevilacqua. A Multimodal Probabilistic Model for Gesture-based Control of Sound Synthesis. 21st ACM international conference on Multimedia (MM'13), Oct 2013, Barcelona, Spain, Spain. pp.705-708, ⟨10.1145/2502081.2502184⟩. ⟨hal-01005538⟩