Learning the Structure of Deep Architectures Using L1 Regularization

Praveen Kulkarni 1 Joaquin Zepeda 1 Frédéric Jurie 2 Patrick Pérez 1 Louis Chevallier 1
2 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen
Abstract : We present a method that formulates the selection of the structure of a deep architecture as a penalized, discriminative learning problem. Up to now, the structure of deep architectures has been fixed by hand, and only the weights are learned using discriminative learning. Our work is a first attempt towards a more formal method of deep structure selection. We consider architectures consisting only of fully-connected layers, and our approach relies on diagonal matrices inserted between subsequent layers. By including an L1 norm of the diagonal entries of said matrices as a regularization penalty, we force the diagonals to be sparse, accordingly selecting the effective number of rows (respectively, columns) of the corresponding layer's (next layer's) weights matrix. We carry out experiments on a standard dataset and show that our method succeeds in selecting the structure of deep architectures of multiple layers. One variant of our architecture results in a feature vector of size as little as $36$, while retaining very high image classification performance.
Document type :
Conference papers
Complete list of metadatas

Cited literature [21 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01266462
Contributor : Greyc Référent <>
Submitted on : Friday, October 13, 2017 - 12:45:43 PM
Last modification on : Thursday, February 7, 2019 - 5:45:10 PM
Long-term archiving on : Sunday, January 14, 2018 - 1:21:13 PM

File

hal-01266462.pdf
Files produced by the author(s)

Identifiers

Citation

Praveen Kulkarni, Joaquin Zepeda, Frédéric Jurie, Patrick Pérez, Louis Chevallier. Learning the Structure of Deep Architectures Using L1 Regularization. British Machine Vision Conference, 2015, Sep 2015, swansea, United Kingdom. ⟨10.5244/C.29.23⟩. ⟨hal-01266462⟩

Share

Metrics

Record views

252

Files downloads

248