SHADE: Information-Based Regularization for Deep Learning

Abstract : Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. Our second contribution is to derive a stochastic version of the regularizer compatible with deep learning, resulting in a tractable training scheme. We empirically validate the efficiency of our approach to improve classification performances compared to common regularization schemes on several standard architectures.
Document type :
Conference papers
Complete list of metadatas

Cited literature [19 references]  Display  Hide  Download

https://hal.sorbonne-universite.fr/hal-01994740
Contributor : Gestionnaire Hal-Su <>
Submitted on : Friday, January 25, 2019 - 4:29:17 PM
Last modification on : Friday, July 5, 2019 - 3:26:03 PM
Long-term archiving on : Friday, April 26, 2019 - 3:11:20 PM

File

1804.10988.pdf
Files produced by the author(s)

Identifiers

Citation

Michael Blot, Thomas Robert, Nicolas Thome, Matthieu Cord. SHADE: Information-Based Regularization for Deep Learning. ICIP 2018 - 25th IEEE International Conference on Image Processing, Oct 2018, Athènes, Greece. pp.813-817, ⟨10.1109/ICIP.2018.8451092⟩. ⟨hal-01994740⟩

Share

Metrics

Record views

13

Files downloads

27