Concentration inequalities in the infinite urn scheme for occupancy counts and the missing mass, with applications

Abstract : An infinite urn scheme is defined by a probability mass function $(p_j)_{j\geq 1}$ over positive integers. A random allocation consists of a sample of $N$ independent drawings according to this probability distribution where $N$ may be deterministic or Poisson-distributed. This paper is concerned with occupancy counts, that is with the number of symbols with $r$ or at least $r$ occurrences in the sample, and with the missing mass that is the total probability of all symbols that do not occur in the sample. Without any further assumption on the sampling distribution, these random quantities are shown to satisfy Bernstein-type concentration inequalities. The variance factors in these concentration inequalities are shown to be tight if the sampling distribution satisfies a regular variation property. This regular variation property reads as follows. Let the number of symbols with probability larger than $x$ be $\vec{\nu}(x) = |\{ j \colon p_j \geq x\}|$. In a regularly varying urn scheme, $\vec{\nu}$ satisfies $\lim_{\tau\rightarrow 0} \vec{\nu}(\tau x)/\vec\nu(\tau) = x^{-\alpha}$ for $\alpha \in [0,1]$ and the variance of the number of distinct symbols in a sample tends to infinity as the sample size tends to infinity. Among other applications, these concentration inequalities allow us to derive tight confidence intervals for the Good-Turing estimator of the missing mass.
Type de document :
Pré-publication, Document de travail
2015
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01101671
Contributeur : Anna Ben-Hamou <>
Soumis le : vendredi 9 janvier 2015 - 12:31:14
Dernière modification le : mardi 11 octobre 2016 - 15:20:17
Document(s) archivé(s) le : vendredi 11 septembre 2015 - 06:24:42

Fichier

main-arxiv.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01101671, version 1
  • ARXIV : 1412.8652

Collections

PMA | INRIA | UPMC | USPC

Citation

Anna Ben-Hamou, Stephane Boucheron, Mesrob I. Ohannessian. Concentration inequalities in the infinite urn scheme for occupancy counts and the missing mass, with applications. 2015. <hal-01101671>

Partager

Métriques

Consultations de
la notice

218

Téléchargements du document

167