Notes on the use of variational autoencoders for speech and audio spectrogram modeling

Laurent Girin (1) , Fanny Roche (2) , Thomas Hueber (1) , Simon Leglaive (3)

1 GIPSA-CRISSP - GIPSA - Cognitive Robotics, Interactive Systems, & Speech Processing
2 Arturia [Montbonnot-Saint-Martin]
3 PERCEPTION - Interpretation and Modelling of Images and Videos

Variational autoencoders (VAEs) are powerful (deep) generative artificial neural networks. They have been recently used in several papers for speech and audio processing, in particular for the modeling of speech/audio spectrograms. In these papers, very poor theoretical support is given to justify the chosen data representation and decoder likelihood function or the corresponding cost function used for training the VAE. Yet, a nice theoretical statistical framework exists and has been extensively presented and discussed in papers dealing with nonnegative matrix factorization (NMF) of audio spectrograms and its application to audio source separation. In the present paper, we show how this statistical framework applies to VAE-based speech/audio spectrogram modeling. This provides the latter insights on the choice and interpretability of data representation and model parameterization.

Domaines: Traitement du signal et de l'image [eess.SP]; Intelligence artificielle [cs.AI]

Laurent Girin, Fanny Roche, Thomas Hueber, Simon Leglaive. Notes on the use of variational autoencoders for speech and audio spectrogram modeling. DAFx 2019 - 22nd International Conference on Digital Audio Effects, Sep 2019, Birmingham, United Kingdom. pp.1-8. ⟨hal-02349385⟩