Service interruption on Monday 11 July from 12:30 to 13:00: all the sites of the CCSD (HAL, EpiSciences, SciencesConf, AureHAL) will be inaccessible (network hardware connection).
Skip to Main content Skip to Navigation
Conference papers

Understanding Priors in Bayesian Neural Networks at the Unit Level

Mariia Vladimirova 1 Jakob Verbeek 2 Pablo Mesejo 3 Julyan Arbel 1 
1 MISTIS - Modelling and Inference of Complex and Structured Stochastic Systems
Inria Grenoble - Rhône-Alpes, Grenoble INP - Institut polytechnique de Grenoble - Grenoble Institute of Technology, LJK - Laboratoire Jean Kuntzmann
2 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : We investigate deep Bayesian neural networks with Gaussian priors on the weights and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, “weight decay”, regularization. Our results indicate a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.
Document type :
Conference papers
Complete list of metadata

Cited literature [14 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02177151
Contributor : Mariia Vladimirova Connect in order to contact the contributor
Submitted on : Monday, July 8, 2019 - 4:27:43 PM
Last modification on : Friday, February 4, 2022 - 3:09:54 AM

File

paper_arXiv_ICML2019-3.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02177151, version 1
  • ARXIV : 1810.05193

Citation

Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, Julyan Arbel. Understanding Priors in Bayesian Neural Networks at the Unit Level. ICML 2019 - 36th International Conference on Machine Learning, Jun 2019, Long Beach, United States. pp.6458-6467. ⟨hal-02177151⟩

Share

Metrics

Record views

162

Files downloads

531