Consistency of Random Forests

Erwan Scornet (1) , Gérard Biau (1, 2, 3, 4) , Jean-Philippe Vert (5, 6)

1 LSTA - Laboratoire de Statistique Théorique et Appliquée
2 LPMA - Laboratoire de Probabilités et Modèles Aléatoires
3 DMA - Département de Mathématiques et Applications - ENS Paris
4 CLASSIC - Computational Learning, Aggregation, Supervised Statistical, Inference, and Classification
5 CBIO - Centre de Bioinformatique
6 Cancer et génome: Bioinformatique, biostatistiques et épidémiologie d'un système complexe

Random forests are a learning algorithm proposed by Breiman (2001) which combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data-dependent tree structure. In the present paper, we take a step forward in forest exploration by proving a consistency result for Breiman's (2001) original algorithm in the context of additive regression models. Our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity in high-dimensional settings.

Mots clés: Random forests; randomization; consistency; additive model; sparsity; dimension reduction

Domaines: Statistiques [math.ST]; Machine Learning [stat.ML]; Théorie [stat.TH]

Erwan Scornet, Gérard Biau, Jean-Philippe Vert. Consistency of Random Forests. 2014. ⟨hal-00990008v2⟩