Top-Down Regularization of Deep Belief Networks

Hanlin Goh (1, 2, 3) , Nicolas Thome (3) , Matthieu Cord (3) , Joo-Hwee Lim (1, 2)

1 Institute for Infocomm Research - I²R [Singapore]
2 IPAL - Image & Pervasive Access Lab
3 MLIA - Machine Learning and Information Access

Designing a principled and effective algorithm for learning deep architectures is a challenging problem. The current approach involves two training phases: a fully unsupervised learning followed by a strongly discriminative optimization. We suggest a deep learning strategy that bridges the gap between the two phases, resulting in a three-phase learning procedure. We propose to implement the scheme using a method to regularize deep belief networks with top-down information. The network is constructed from building blocks of restricted Boltzmann machines learned by combining bottom-up and top-down sampled signals. A global optimization procedure that merges samples from a forward bottom-up pass and a top-down pass is used. Experiments on the MNIST dataset show improvements over the existing algorithms for deep belief networks. Object recognition results on the Caltech-101 dataset also yield competitive results.

Domaines: Apprentissage [cs.LG]

Hanlin Goh, Nicolas Thome, Matthieu Cord, Joo-Hwee Lim. Top-Down Regularization of Deep Belief Networks. Advances in Neural Information Processing Systems 26, Dec 2013, Lake Tahoe, United States. pp.1878-1886. ⟨hal-00947569⟩