Skip to Main content Skip to Navigation
Conference papers

Robust Training of Vector Quantized Bottleneck Models

Abstract : In this paper we demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs). Discrete latent variable models have been shown to learn nontrivial representations of speech, applicable to unsupervised voice conversion and reaching state-of-the-art performance on unit discovery tasks. For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE). However, training deep discrete variable models is challenging, due to the inherent non-differentiability of the discretization operation. In this paper we focus on VQ-VAE, a state-of-the-art discrete bottleneck model shown to perform on par with its continuous counterparts. It quantizes encoder outputs with on-line k-means clustering. We show that the codebook learning can suffer from poor initialization and non-stationarity of clustered encoder outputs. We demonstrate that these can be successfully overcome by increasing the learning rate for the codebook and periodic date-dependent codeword re-initialization. As a result, we achieve more robust training across different tasks, and significantly increase the usage of latent codewords even for large code-books. This has practical benefit, for instance, in unsupervised representation learning, where large codebooks may lead to disentanglement of latent representations.
Complete list of metadata

Cited literature [38 references]  Display  Hide  Download
Contributor : Antoine LAURENT Connect in order to contact the contributor
Submitted on : Wednesday, August 5, 2020 - 9:41:37 AM
Last modification on : Sunday, June 26, 2022 - 10:16:36 AM
Long-term archiving on: : Monday, November 30, 2020 - 2:36:37 PM


Files produced by the author(s)


  • HAL Id : hal-02912027, version 1



Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, et al.. Robust Training of Vector Quantized Bottleneck Models. IJCNN 2020, Jul 2020, Glasgow, United Kingdom. ⟨hal-02912027⟩



Record views


Files downloads