Towards a real-time mapping of Finger Gesture to Sound

Vassilios Matsoukas , Sotiris Manitsaris , Athanasios Manitsaris

The scope of this research is to develop a system for real-time and continuous gesture following and recognition given emphasis to the finger approach of gesture control of sound. The proposed methodology can be implemented in a low cost computer vision system, which provides the means to facilitate an interactive composition of contemporary music. A simple web camera is used to capture finger musical gestures in 3D space without any tangible instrument. Afterwards we use the PianOrasis system, which recognizes simultaneously the gestures of all five fingers by extracting meaningful features from each frame sequence. In order to achieve this in real-time, the “gesture following” method is applied. Hidden Markov Models are used to update “continuously” the gesture descriptors. We choose then to map high-level features of finger motions to low-level audio descriptors given emphasis to the temporal representation of mapping rather than the spatial. This is due to the need of real-time performance.

Domaines: Acoustique [physics.class-ph]

Vassilios Matsoukas, Sotiris Manitsaris, Athanasios Manitsaris. Towards a real-time mapping of Finger Gesture to Sound. Acoustics 2012, Apr 2012, Nantes, France. ⟨hal-00811320⟩