On the selection of decision trees in Random Forests

Simon Bernard (1) , Laurent Heutte (1) , Sébastien Adam (1)

1 DocApp - LITIS - Equipe Apprentissage

In this paper we present a study on the random forest (RF) family of ensemble methods. In a ldquoclassicalrdquo RF induction process a fixed number of randomized decision trees are inducted to form an ensemble. This kind of algorithm presents two main drawbacks : (i) the number of trees has to be fixed a priori (ii) the interpretability and analysis capacities offered by decision tree classifiers are lost due to the randomization principle. This kind of process in which trees are independently added to the ensemble, offers no guarantee that all those trees will cooperate effectively in the same committee. This statement rises two questions: are there any decision trees in a RF that provide the deterioration of ensemble performance? If so, is it possible to form a more accurate committee via removal of decision trees with poor performance? The answer to these questions is tackled as a classifier selection problem. We thus show that better subsets of decision trees can be obtained even using a sub-optimal classifier selection method. This proves that ldquoclassicalrdquo RF induction process, for which randomized trees are arbitrary added to the ensemble, is not the best approach to produce accurate RF classifiers. We also show the interest in designing RF by adding trees in a more dependent way than it is traditionally done in ldquoclassicalrdquo RF induction algorithms.

Domaines: Apprentissage [cs.LG]

Simon Bernard, Laurent Heutte, Sébastien Adam. On the selection of decision trees in Random Forests. IEEE International Joint Conference on Neural Networks (IJCNN), Jun 2008, Atlanta, United States. pp.302-307, ⟨10.1109/IJCNN.2009.5178693⟩. ⟨hal-00436355⟩