Slope heuristics for heteroscedastic regression on a random design

Sylvain Arlot (1, 2) , Pascal Massart (1, 2)

1 LM-Orsay - Laboratoire de Mathématiques d'Orsay
2 SELECT - Model selection in statistical learning

In a recent paper Birgé and Massart (2006) have introduced the notion of minimal penalty in the context of penalized least squares for Gaussian regression. They have shown that for several model selection problems, simply multiplying by 2 the minimal penalty leads to some (nearly) optimal penalty in the sense that it approximately minimizes the resulting oracle inequality. Interestingly, the minimal penalty can be evaluated from the data themselves which leads to a data-driven choice of the penalty that one can use in practice. Unfortunately their approach heavily relies on the Gaussian nature of the stochastic framework that they consider. Our purpose in this paper is twofold: stating a heuristics to design a data-driven penalty (the slope heuristics) which is not sensitive to the Gaussian assumption as in (Birgé and Massart, 2006) and proving that it works for penalized least squares random design regression. As a matter of fact, we could prove some precise mathematical results only for histogram bin-width selection. For some technical reasons which are explained in the paper, we could not work at the level of generality that we were expecting but still this is a first step towards further results and even if the mathematical results hold in some specific framework, the approach and the method that we use are indeed general.

Mots clés: non-parametric regression; non-asymptotic; model selection; penalization; heteroscedastic data; histogram; concentration inequalities

Domaines: Statistiques [math.ST]; Théorie [stat.TH]; Machine Learning [stat.ML]; Méthodologie [stat.ME]

Sylvain Arlot, Pascal Massart. Slope heuristics for heteroscedastic regression on a random design. 2008. ⟨hal-00243116v1⟩