Suboptimality of Penalized Empirical Risk Minimization in Classification

Guillaume Lecué (1, 2)

1 LPMA - Laboratoire de Probabilités et Modèles Aléatoires
2 LAMA - Laboratoire d'Analyse et de Mathématiques Appliquées

Let $\cF$ be a set of $M$ classification procedures with values in $[-1,1]$. Given a loss function, we want to construct a procedure which mimics at the best possible rate the best procedure in $\cF$. This fastest rate is called optimal rate of aggregation. Considering a continuous scale of loss functions with various types of convexity, we prove that optimal rates of aggregation can be either $((\log M)/n)^{1/2}$ or $(\log M)/n$. We prove that, if all the $M$ classifiers are binary, the (penalized) Empirical Risk Minimization procedures are suboptimal (even under the margin/low noise condition) when the loss function is somewhat more than convex, whereas, in that case, aggregation procedures with exponential weights achieve the optimal rate of aggregation.

Mots clés: Classification; low noise assumption; aggregation; empirical risk minimization

Domaines: Statistiques [math.ST]; Théorie [stat.TH]

Guillaume Lecué. Suboptimality of Penalized Empirical Risk Minimization in Classification. 2007. ⟨hal-00138840⟩