A Hierarchical Approach for the Design of Gesture-to-Sound Mappings

Jules Françoise (1) , Baptiste Caramiaux (2) , Frédéric Bevilacqua (2)

1 IMTR
2 IRCAM - Institut de Recherche et Coordination Acoustique/Musique

We propose a hierarchical approach for the design of gesture-to-sound mappings, with the goal to take into account multilevel time structures in both gesture and sound processes. This allows for the integration of temporal mapping strategies, complementing mapping systems based on instantaneous relationships between gesture and sound synthesis parameters. As an example, we propose the implementation of Hierarchical Hidden Markov Models to model gesture input, with a flexible structure that can be authored by the user. Moreover, some parameters can be adjusted through a learning phase. We show some examples of gesture segmentations based on this approach, considering several phases such as preparation, attack, sustain, release. Finally we describe an application, developed in Max/MSP, illustrating the use of accelerometer-based sen- sors to control phase vocoder synthesis techniques based on this approach.

Mots clés: gesture; gesture recognition; hidden markov models; interaction; machine learning; mapping; music; music interfaces; sound

Domaines: Son [cs.SD]; Musique, musicologie et arts de la scène

Jules Françoise, Baptiste Caramiaux, Frédéric Bevilacqua. A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. 9th Sound and Music Computing Conference, Jul 2012, Copenhagen, Denmark. pp.233-240. ⟨hal-00847203⟩