Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Gaussian-Spherical Restricted Boltzmann Machines

Aurelien Decelle 1 Cyril Furtlehner 1
1 TAU - TAckling the Underspecified
Inria Saclay - Ile de France, LRI - Laboratoire de Recherche en Informatique
Abstract : We consider a special type of Restricted Boltzmann machine (RBM), namely a Gaussian-spherical RBM where the visible units have Gaussian priors while the vector of hidden variables is constrained to stay on an ${\mathbbm L}_2$ sphere. The spherical constraint having the advantage to admit exact asymptotic treatments, various scaling regimes are explicitly identified based solely on the spectral properties of the coupling matrix (also called weight matrix of the RBM). Incidentally these happen to be formally related to similar scaling behaviours obtained in a different context dealing with spatial condensation of zero range processes. More specifically, when the spectrum of the coupling matrix is doubly degenerated an exact treatment can be proposed to deal with finite size effects. Interestingly the known parallel between the ferromagnetic transition of the spherical model and the Bose-Einstein condensation can be made explicit in that case. More importantly this gives us the ability to extract all needed response functions with arbitrary precision for the training algorithm of the RBM. This allows us then to numerically integrate the dynamics of the spectrum of the weight matrix during learning in a precise way. This dynamics reveals in particular a sequential emergence of modes from the Marchenko-Pastur bulk of singular vectors of the coupling matrix.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-02420813
Contributor : Aurélien Decelle <>
Submitted on : Friday, December 20, 2019 - 10:20:00 AM
Last modification on : Wednesday, September 16, 2020 - 5:51:58 PM

Links full text

Identifiers

  • HAL Id : hal-02420813, version 1
  • ARXIV : 1910.14544

Collections

Citation

Aurelien Decelle, Cyril Furtlehner. Gaussian-Spherical Restricted Boltzmann Machines. 2019. ⟨hal-02420813⟩

Share

Metrics

Record views

41