 |
Free and accessible knowledge |
Conference papers
Bayesian neural networks become heavier-tailed with depth
Abstract : We investigate deep Bayesian neural networks with Gaussian priors on the weights and ReLU-like nonlinearities, shedding light on novel distribution properties at the level of the neural network units. The main thrust of the paper is to establish that the prior distribution induced on the units before and after activation becomes increasingly heavier-tailed with depth. We show that first layer units are Gaussian, second layer units are sub-Exponential, and we introduce sub-Weibull distributions to characterize the deeper layers units. This result provides new theoretical insight on deep Bayesian neural networks, underpinning their practical potential. The workshop paper is based on the original paper Vladimirova et al. (2018).
Cited literature [1 references]
https://hal.archives-ouvertes.fr/hal-01950658
Contributor : Julyan Arbel Connect in order to contact the contributor
Submitted on : Tuesday, December 11, 2018 - 3:52:54 AM
Last modification on : Friday, February 4, 2022 - 3:20:05 AM
Long-term archiving on: : Tuesday, March 12, 2019 - 12:59:06 PM
Contributor : Julyan Arbel Connect in order to contact the contributor
Submitted on : Tuesday, December 11, 2018 - 3:52:54 AM
Last modification on : Friday, February 4, 2022 - 3:20:05 AM
Long-term archiving on: : Tuesday, March 12, 2019 - 12:59:06 PM
File
BDL.pdf
Files produced by the author(s)