A Bayesian reassessment of nearest-neighbour classification

Jean-Michel Marin (1) , Christian Robert (2, 3) , Mike Titterington (4)

1 SELECT - Model selection in statistical learning
2 CEREMADE - CEntre de REcherches en MAthématiques de la DEcision
3 CREST - Centre de Recherche en Économie et Statistique
4 University of Glasgow

The k-nearest-neighbour procedure is a well-known method used in supervised classification. While it has been superseded by more recent methods developed in machine learning, it remains an essential tool for classifiers. This paper proposes a reassessment of this approach as a statistical technique derived from a proper probabilistic model; in particular, we modify the assessment made in a previous analysis of this method undertaken by Holmes and Adams (2002, 2003) where the underlying probabilistic model is not completely well-defined. Once clear probabilistic bases of the k-nearest-neighbour procedure are established, we proceed to the derivation of practical computational tools to conduct Bayesian inference on the parameters of the corresponding model. In particular, we assess the difficulties inherent to pseudo-likelihood and to path sampling approximations of a missing normalising constant, and propose a perfect sampling strategy to implement a correct MCMC sampler associated with our model. Illustrations of the performance of the corresponding Bayesian classifier are provided for two benchmark datasets, demonstrating in particular the limitations of the pseudo-likelihood approximation in this set-up.

Domaines: Statistiques [math.ST]

Jean-Michel Marin, Christian Robert, Mike Titterington. A Bayesian reassessment of nearest-neighbour classification. [Research Report] RR-6173, 2007, pp.28. ⟨inria-00143783v1⟩