SING: Symbol-to-Instrument Neural Generator

Abstract : Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as WaveNet and SampleRNN suffer from prohibitive training and inference times because they are based on autoregressive models that generate audio samples one at a time at a rate of 16kHz. In this work, we study the more computationally efficient alternative of generating the waveform frame-by-frame with large strides. We present SING, a lightweight neural audio synthesizer for the original task of generating musical notes given desired instrument, pitch and velocity. Our model is trained end-to-end to generate notes from nearly 1000 instruments with a single decoder, thanks to a new loss function that minimizes the distances between the log spectrograms of the generated and target waveforms. On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01899949
Contributor : Alexandre Defossez <>
Submitted on : Tuesday, October 23, 2018 - 10:58:57 AM
Last modification on : Thursday, February 7, 2019 - 4:55:38 PM
Long-term archiving on : Thursday, January 24, 2019 - 12:37:29 PM

Files

symbol_to_instrument_neural_ge...
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01899949, version 1
  • ARXIV : 1810.09785

Collections

Citation

Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, Francis Bach. SING: Symbol-to-Instrument Neural Generator. Conference on Neural Information Processing Systems (NIPS), Dec 2018, Montréal, Canada. ⟨hal-01899949⟩

Share

Metrics

Record views

277

Files downloads

148