Skip to Main content Skip to Navigation
Conference papers

Understanding Priors in Bayesian Neural Networks at the Unit Level

Mariia Vladimirova 1 Jakob Verbeek 2 Pablo Mesejo 3 Julyan Arbel 1
1 MISTIS - Modelling and Inference of Complex and Structured Stochastic Systems
Inria Grenoble - Rhône-Alpes, Grenoble INP - Institut polytechnique de Grenoble - Grenoble Institute of Technology, LJK - Laboratoire Jean Kuntzmann
2 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : We investigate deep Bayesian neural networks with Gaussian priors on the weights and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, “weight decay”, regularization. Our results indicate a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.
Document type :
Conference papers
Complete list of metadata

Cited literature [14 references]  Display  Hide  Download
Contributor : Mariia Vladimirova <>
Submitted on : Monday, July 8, 2019 - 4:27:43 PM
Last modification on : Tuesday, May 11, 2021 - 11:37:39 AM


Files produced by the author(s)


  • HAL Id : hal-02177151, version 1
  • ARXIV : 1810.05193


Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, Julyan Arbel. Understanding Priors in Bayesian Neural Networks at the Unit Level. ICML 2019 - 36th International Conference on Machine Learning, Jun 2019, Long Beach, United States. pp.6458-6467. ⟨hal-02177151⟩



Record views


Files downloads