Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Proximal boosting and variants

Abstract : Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm, when the empirical risk to minimize is not differentiable, in order to introduce a novel boosting approach, called proximal boosting. Besides being motivated by non-differentiable optimization, the proposed algorithm benefits from algorithmic improvements such as controlling the approximation error and Nesterov's acceleration, in the same way as gradient boosting [Grubb and Bagnell, 2011, Biau et al., 2018]. This leads to two variants, respectively called residual proximal boosting and accelerated proximal boosting. Theoretical convergence is proved for the first two procedures under different hypotheses on the empirical risk and advantages of leveraging proximal methods for boosting are illustrated by numerical experiments on simulated and real-world data. In particular, we exhibit a favorable comparison over gradient boosting regarding convergence rate and prediction accuracy.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-01853244
Contributor : Maxime Sangnier Connect in order to contact the contributor
Submitted on : Tuesday, July 27, 2021 - 11:07:58 PM
Last modification on : Friday, July 30, 2021 - 4:03:07 AM

File

paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01853244, version 3

Citation

Erwan Fouillen, Claire Boyer, Maxime Sangnier. Proximal boosting and variants. 2021. ⟨hal-01853244v3⟩

Share

Metrics

Record views

30

Files downloads

25