Estimation of entropies and divergences via nearest neighbors

Nikolai Leonenko (1) , Luc Pronzato (2) , Vippal Savani (1)

1 School of Mathematics [Cardiff]
2 Laboratoire d'Informatique, Signaux, et Systèmes de Sophia-Antipolis (I3S) / Equipe SYSTEMES

We extend the results by Leonenko et al. and show how k-th nearest-neighbor distances in a sample of N i.i.d. vectors distributed with the probability density f can be used to estimate consistently the Rényi and Tsallis entropies of the unknown f under minimal assumptions. The method is extended to the estimation of statistical distances between two distributions in the case when one i.i.d. sample from each is available.

Mots clés: entropy estimation; estimation of statistical distance; estimation of divergence; nearest-neighbor distances; Rényi entropy; Havrda-Chavàt entropy; Tsallis entropy

Domaines: Statistiques [math.ST]; Théorie [stat.TH]

Nikolai Leonenko, Luc Pronzato, Vippal Savani. Estimation of entropies and divergences via nearest neighbors. ProbaStat 2006, Jun 2006, Smolenice, Slovakia. pp.265-273. ⟨hal-00322783⟩