Skip to Main content Skip to Navigation
Conference papers

Learning Sparse Filters In Deep Convolutional Neural Networks With A l 1 /l 2 Pseudo-Norm

Abstract : While deep neural networks (DNNs) have proven to be efficient for numerous tasks, they come at a high memory and computation cost, thus making them impractical on resource-limited devices. However, these networks are known to contain a large number of parameters. Recent research has shown that their structure can be more compact without compromising their performance. In this paper, we present a sparsity-inducing regularization term based on the ratio l1/l2 pseudo-norm defined on the filter coefficients. By defining this pseudo-norm appropriately for the different filter kernels, and removing irrelevant filters, the number of kernels in each layer can be drastically reduced leading to very compact Deep Convolutional Neural Networks (DCNN) structures. Unlike numerous existing methods, our approach does not require an iterative retraining process and, using this regularization term, directly produces a sparse model during the training process. Furthermore, our approach is also much easier and simpler to implement than existing methods. Experimental results on MNIST and CIFAR-10 show that our approach significantly reduces the number of filters of classical models such as LeNet and VGG while reaching the same or even better accuracy than the baseline models. Moreover, the trade-off between the sparsity and the accuracy is compared to other loss regularization terms based on the l1 or l2 norm as well as the SSL [1], NISP [2] and GAL [3] methods and shows that our approach is outperforming them.
Document type :
Conference papers
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03048750
Contributor : Anthony Berthelier Connect in order to contact the contributor
Submitted on : Wednesday, December 9, 2020 - 3:09:48 PM
Last modification on : Thursday, September 9, 2021 - 2:36:02 PM
Long-term archiving on: : Wednesday, March 10, 2021 - 7:26:06 PM

File

Anthony_ICPR.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03048750, version 1

Citation

Anthony Berthelier, Yongzhe Yan, Thierry Chateau, Christophe Blanc, Stefan Duffner, et al.. Learning Sparse Filters In Deep Convolutional Neural Networks With A l 1 /l 2 Pseudo-Norm. CADL 2020 : Workshop on Computational Aspects of Deep Learning - ICPR 2020, Jan 2021, Milan, Italy. ⟨hal-03048750⟩

Share

Metrics

Record views

88

Files downloads

265