Estimator selection in the Gaussian setting

Yannick Baraud (1) , Christophe Giraud (2) , Sylvie Huet (3)

1 JAD - Laboratoire Jean Alexandre Dieudonné
2 CMAP - Centre de Mathématiques Appliquées - Ecole Polytechnique
3 Unité MIAJ - INRA - Mathématiques et Informatique Appliquées

We consider the problem of estimating the mean $f$ of a Gaussian vector $Y$ with independent components of common unknown variance $\sigma^{2}$. Our estimation procedure is based on estimator selection. More precisely, we start with an arbitrary and possibly infinite collection $\FF$ of estimators of $f$ based on $Y$ and, with the same data $Y$, aim at selecting an estimator among $\FF$ with the smallest Euclidean risk. No assumptions on the estimators are made and their dependencies with respect to $Y$ may be unknown. We establish a non-asymptotic risk bound for the selected estimator. As particular cases, our approach allows to handle the problems of aggregation and model selection as well as those of choosing a window and a kernel for estimating a regression function, or tuning the parameter involved in a penalized criterion. We also derive oracle-type inequalities when $\FF$ consists of linear estimators. For illustration, we carry out two simulation studies. One aims at comparing our procedure to cross-validation for choosing a tuning parameter. The other shows how to implement our approach to solve the problem of variable selection in practice.

Mots clés: Gaussian linear regression; Estimator selection; Model selection; Variable selection; Linear estimator; Kernel estimator; Ridge regression; Lasso; Elastic net; Random Forest; PLS1 regression

Domaines: Statistiques [math.ST]; Théorie [stat.TH]

Yannick Baraud, Christophe Giraud, Sylvie Huet. Estimator selection in the Gaussian setting. Annales de l'Institut Henri Poincaré (B) Probabilités et Statistiques, 2014. ⟨hal-00502156v2⟩