Model selection by resampling penalization

Sylvain Arlot (1, 2)

1 LM-Orsay - Laboratoire de Mathématiques d'Orsay
2 SELECT - Model selection in statistical learning

We define a new family of resampling-based penalization procedures for model selection in a very general framework. It generalizes several methods (including Efron's bootstrap penalties and the recently proposed leave-one-out penalties, Arlot (2008)) to any exchangeable weighted bootstrap resampling scheme. In the heteroscedastic regression framework, assuming the models to have a particular structure, we prove that these penalties satisfy a non-asymptotic oracle inequality with a leading constant close to 1. In particular, they are asympotically optimal. We then use these resampling penalties to define an estimator which adapts simultaneously to the smoothness of the regression function and the heteroscedasticity of the noise. This is remarkable because these penalties are general purpose devices, which have not been built specifically to handle heteroscedastic data. We have thus proven that resampling penalties are naturally adaptive to heteroscedasticity. In addition, a simulation study shows that these penalties improve simultaneously V-fold cross-validation, in particular when the signal-to-noise ratio is not large.

Mots clés: non-parametric statistics; resampling; non-asymptotic; exchangeable weighted bootstrap; model selection; penalization; non-parametric regression; adaptivity; heteroscedastic data; histogram

Domaines: Statistiques [math.ST]

Sylvain Arlot. Model selection by resampling penalization. 2008. ⟨hal-00262478v1⟩