LassoNet: A Neural Network with Feature Sparsity

Ismael Lemhadri (1) , Feng Ruan (1) , Louis Abraham (2) , Robert Tibshirani (3)

1 Department of Statistics [Stanford]
2 Gematria Technologies
3 DBDS - Department of Biomedical Data Science [Stanford]

Much work has been done recently to make neural networks more interpretable, and one approach is to arrange for the network to use only a subset of the available features. In linear models, Lasso (or 1-regularized) regression assigns zero weights to the most irrelevant or redundant features, and is widely used in data science. However the Lasso only applies to linear models. Here we introduce LassoNet, a neural network framework with global feature selection. Our approach achieves feature sparsity by adding a skip (residual) layer and allowing a feature to participate in any hidden layer only if its skip-layer representative is active. Unlike other approaches to feature selection for neural nets, our method uses a modified objective function with constraints, and so integrates feature selection with the parameter learning directly. As a result, it delivers an entire regularization path of solutions with a range of feature sparsity. We apply LassoNet to a number of real-data problems and find that it significantly outperforms state-of-the-art methods for feature selection and regression. LassoNet uses projected proximal gradient descent, and generalizes directly to deep networks. It can be implemented by adding just a few lines of code to a standard neural network.

Mots clés: Neural Networks; Feature Selection; Strong Hierarchy; Proximal Gradient Descent

Domaines: Statistiques [math.ST]; Machine Learning [stat.ML]; Apprentissage [cs.LG]

Ismael Lemhadri, Feng Ruan, Louis Abraham, Robert Tibshirani. LassoNet: A Neural Network with Feature Sparsity. Journal of Machine Learning Research, In press, 22 (127). ⟨hal-03454793⟩