Interactive Sound Texture Synthesis Through Semi-Automatic User Annotations

Diemo Schwarz (1) , Baptiste Caramiaux (1)

1 Equipe Interactions musicales temps-réel

We present a way to make environmental recordings controllable again by the use of continuous annotations of the high-level semantic parameter one wishes to control, e.g. wind strength or crowd excitation level. A partial annotation can be propagated to cover the entire recording via cross-modal analysis between gesture and sound by canonical time warping (CTW). The annotations serve as a descriptor for lookup in corpus-based concatenative synthesis in order to invert the sound/annotation relationship. The workflow has been evaluated by a preliminary subject test and results on canonical correlation analysis (CCA) show high consistency between annotations and a small set of audio descriptors being well correlated with them. An experiment of the propagation of annotations shows the superior performance of CTW over CCA with as little as 20 s of annotated material.

Mots clés: corpus-based synthesis; Canonical Correlation Analysis; sound textures; audio descriptors; canonical time warping

Mots clés: Informatique musicale

Domaines: Son [cs.SD]; Interface homme-machine [cs.HC]; Musique, musicologie et arts de la scène; Traitement du signal et de l'image [eess.SP]; Apprentissage [cs.LG]; Intelligence artificielle [cs.AI]; Ingénierie assistée par ordinateur; Multimédia [cs.MM]; Vision par ordinateur et reconnaissance de formes [cs.CV]; Autre [cs.OH]; Traitement du signal et de l'image [eess.SP]

Diemo Schwarz, Baptiste Caramiaux. Interactive Sound Texture Synthesis Through Semi-Automatic User Annotations. Springer International Publishing; Aramaki, M., Derrien, O., Kronland-Martinet, R., Ystad, S. Sound, Music, and Motion, Lecture Notes in Computer Science, Vol. 8905, pp.372-392, 2014. ⟨hal-01161076⟩