Model selection by bootstrap penalization for classification

Magalie Fromont (1, 2)

1 IRMAR - Institut de Recherche Mathématique de Rennes
2 Laboratoire de statistique

We consider the binary classification problem. Given an i.i.d. sample drawn from the distribution of an chi x {0,1}-valued random pair, we propose to estimate the so-called Bayes classifier by minimizing the sum of the empirical classification error and a penalty term based on Efron's or i.i.d. weighted bootstrap samples of the data. We obtain exponential inequalities for such bootstrap type penalties, which allow us to derive non-asymptotic properties for the corresponding estimators. In particular, we prove that these estimators achieve the global minimax risk over sets of functions built from Vapnik-Chervonenkis classes. The obtained results generalize Koltchinskii (2001) and Bartlett et al.'s (2002) ones for Rademacher penalties that can thus be seen as special examples of bootstrap type penalties. To illustrate this, we carry out an experimental study in which we compare the different methods for an intervals model selection problem.

Mots clés: canonical smooth estimation; Bayesian bootstrap; empirical processes; risk minimization : complexities; bounds; convergence

Domaines: Statistiques [math.ST]; Théorie [stat.TH]

Magalie Fromont. Model selection by bootstrap penalization for classification. Machine Learning, 2007, 66 (2-3), pp.165-207. ⟨10.1007/s10994-006-7679-y⟩. ⟨hal-00457774⟩