Nonparametric estimation of the density of the alternative hypothesis in a multiple testing setup. Application to local false discovery rate estimation

van Hanh Nguyen (1, 2) , Catherine Matias (1)

1 SG - Laboratoire Statistique et Génome
2 LM-Orsay - Laboratoire de Mathématiques d'Orsay

In a multiple testing context, we consider a semiparametric mixture model with two components where one component is known and corresponds to the distribution of $p$-values under the null hypothesis and the other component $f$ is nonparametric and stands for the distribution under the alternative hypothesis. Motivated by the issue of local false discovery rate estimation, we focus here on the estimation of the nonparametric unknown component $f$ in the mixture, relying on a preliminary estimator of the unknown proportion $\theta$ of true null hypotheses. We propose and study the asymptotic properties of two different estimators for this unknown component. The first estimator is a randomly weighted kernel estimator. We establish an upper bound for its pointwise quadratic risk, exhibiting the classical nonparametric rate of convergence over a class of Hölder densities. To our knowledge, this is the first result establishing convergence as well as corresponding rate for the estimation of the unknown component in this nonparametric mixture. The second estimator is a maximum smoothed likelihood estimator. It is computed through an iterative algorithm, for which we establish a descent property. In addition, these estimators are used in a multiple testing procedure in order to estimate the local false discovery rate. Their respective performances are then compared on synthetic data.

Mots clés: semiparametric mixture model.; False discovery rate; kernel estimation; local false discovery rate; maximum smoothed likelihood; multiple testing; $p$-values; semiparametric mixture model

Domaines: Applications [stat.AP]

van Hanh Nguyen, Catherine Matias. Nonparametric estimation of the density of the alternative hypothesis in a multiple testing setup. Application to local false discovery rate estimation. ESAIM: Probability and Statistics, 2014, 18, pp.584-612. ⟨hal-00738555v2⟩