Skip to Main content Skip to Navigation
Journal articles

A Scalable Bayesian Sampling Method Based on Stochastic Gradient Descent Isotropization

Abstract : Stochastic gradient sg-based algorithms for Markov chain Monte Carlo sampling (sgmcmc) tackle large-scale Bayesian modeling problems by operating on mini-batches and injecting noise on sgsteps. The sampling properties of these algorithms are determined by user choices, such as the covariance of the injected noise and the learning rate, and by problem-specific factors, such as assumptions on the loss landscape and the covariance of sg noise. However, current sgmcmc algorithms applied to popular complex models such as Deep Nets cannot simultaneously satisfy the assumptions on loss landscapes and on the behavior of the covariance of the sg noise, while operating with the practical requirement of non-vanishing learning rates. In this work we propose a novel practical method, which makes the sg noise isotropic, using a fixed learning rate that we determine analytically. Extensive experimental validations indicate that our proposal is competitive with the state of the art on sgmcmc.
Document type :
Journal articles
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03592411
Contributor : Centre De Documentation Eurecom Connect in order to contact the contributor
Submitted on : Tuesday, March 1, 2022 - 11:54:36 AM
Last modification on : Wednesday, March 2, 2022 - 3:32:01 AM

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Links full text

Identifiers

Collections

Citation

Giulio Franzese, Dimitrios Milios, Maurizio Filippone, Pietro Michiardi. A Scalable Bayesian Sampling Method Based on Stochastic Gradient Descent Isotropization. Entropy, MDPI, 2021, 23 (11), pp.1426. ⟨10.3390/e23111426⟩. ⟨hal-03592411⟩

Share

Metrics

Record views

7