, and the French ministry of foreign and European affairs (EGIDE -PROCOPE project number 21887 NJ). The first author acknowledges the support from the german DFG, under the Research Unit FOR-1735 "Structural Inference in Statistics -Adaptation and Efficiency", and under the Collaborative Research Center SFB-1294 "Data Assimilation". Also, this work was finished while he was a guest at the Institut des HautesÉtudes Scientifiques, Université Paris-Saclay, whose support is also gratefully acknowledged. SUPPLEMENTARY MATERIAL Supplement: Supplement to "Post hoc confidence bounds on false positives using reference families" (doi: COMPLETED BY THE TYPESETTER; .pdf). The supplement provides additional materials: relation to previous work, This work has been supported by CNRS (PEPS FaSciDo)

. References,

S. Arlot, G. Blanchard, and E. Roquain, Some nonasymptotic results on resampling in high dimension, II. Multiple tests. Ann. Statist, vol.38, issue.1, pp.83-99, 2010.
URL : https://hal.archives-ouvertes.fr/hal-00194145

F. Bachoc, D. Preinerstorfer, and L. Steinberger, Uniformly valid confidence intervals post-model-selection

A. Belloni, V. Chernozhukov, and C. Hansen, Inference on treatment effects after selection among high-dimensional controls, Rev. Econ. Stud, vol.81, issue.2, pp.608-650, 2014.

Y. Benjamini and Y. Hochberg, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J. Roy. Statist. Soc. Ser. B, vol.57, issue.1, pp.289-300, 1995.

Y. Benjamini and D. Yekutieli, The control of the false discovery rate in multiple testing under dependency, Ann. Statist, vol.29, issue.4, pp.1165-1188, 2001.

Y. Benjamini and D. Yekutieli, False discovery rate-adjusted multiple confidence intervals for selected parameters, J. Amer. Statist. Assoc, vol.100, issue.469, pp.71-93, 2005.

R. Berk, L. Brown, A. Buja, K. Zhang, and L. Zhao, Valid post-selection inference, Ann. Statist, vol.41, issue.2, pp.802-837, 2013.

G. Blanchard, P. Neuvial, and E. Roquain, , 2018.

G. Blanchard, P. Neuvial, and E. Roquain, Supplement to "post hoc confidence bounds on false positives using reference families, 2018.

P. Bühlmann and J. Mandozzi, High-dimensional variable screening and bias in subsequent inference, with an empirical comparison, Comput. Statist, vol.29, issue.3-4, pp.407-430, 2014.

R. Dezeure, P. Bühlmann, L. Meier, and N. Meinshausen, High-dimensional inference: confidence intervals, p-values and r-software hdi, Statist. Sci, vol.30, issue.4, pp.533-558, 2015.

D. Donoho and J. Jin, Higher criticism for detecting sparse heterogeneous mixtures, Ann. Statist, vol.32, issue.3, pp.962-994, 2004.

S. Dudoit and M. J. Van-der-laan, Multiple testing procedures with applications to genomics, Springer Series in Statistics, 2008.

G. Durand, G. Blanchard, P. Neuvial, and E. Roquain, Post hoc false positive control for spatially structured hypotheses, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01829037

J. Fan, X. Han, and W. Gu, Estimating false discovery proportion under arbitrary covariance dependence, Journal of the American Statistical Association, vol.107, issue.499, pp.1019-1035, 2012.

W. Fithian, D. Sun, T. , and J. , Optimal Inference After Model Selection, 2014.

C. Genovese and L. Wasserman, A stochastic process approach to false discovery control, Ann. Statist, vol.32, issue.3, pp.1035-1061, 2004.

C. R. Genovese and L. Wasserman, Exceedance control of the false discovery proportion, J. Amer. Statist. Assoc, vol.101, issue.476, pp.1408-1417, 2006.

J. J. Goeman and A. Solari, Multiple testing for exploratory research, Statist. Sci, vol.26, issue.4, pp.584-597, 2011.

J. Hemerik and J. J. Goeman, False discovery proportion estimation by permutations: confidence for significance analysis of microarrays, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2017.

J. Hemerik, A. Solari, and J. J. Goeman, Permutation-based simultaneous confidence bounds for the false discovery proportion, Biometrika

E. Katsevich and A. Ramdas, Towards "simultaneous selective inference": posthoc bounds on the false discovery proportion, 2018.

J. D. Lee, D. L. Sun, Y. Sun, and J. E. Taylor, Exact post-selection inference, with application to the Lasso, Ann. Stat, vol.44, issue.3, pp.907-927, 2016.

W. Li, Volcano plots in analyzing differential expressions with mrna microarrays, Journal of bioinformatics and computational biology, vol.10, issue.06, p.1231003, 2012.

N. Meinshausen, False discovery control for multiple tests of association under general dependence, Scand. J. Statist, vol.33, issue.2, pp.227-237, 2006.

N. Meinshausen and P. Bühlmann, Lower bounds for the number of true null hypotheses for multiple testing of associations under general dependence structures, Biometrika, vol.92, pp.893-907, 2005.

. R-core-team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, 2017.

J. P. Romano and M. Wolf, Exact and approximate stepdown methods for multiple hypothesis testing, J. Amer. Statist. Assoc, vol.100, issue.469, pp.94-108, 2005.

R. J. Simes, An improved Bonferroni procedure for multiple tests of significance, Biometrika, vol.73, issue.3, pp.751-754, 1986.

J. Taylor and R. J. Tibshirani, Statistical learning and selective inference, Proc. Natl. Acad. Sci. USA, vol.112, issue.25, pp.7629-7634, 2015.

M. J. Van-der-laan, S. Dudoit, and K. S. Pollard, Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives, Stat. Appl. Genet. Mol. Biol, vol.3, issue.1, pp.1-25, 2004.

P. H. Westfall and S. S. Young, Resampling-Based Multiple Testing. Wiley. Examples and Methods for P -Value Adjustment, 1993.

C. Woo, A. Krishnan, and T. D. Wager, Cluster-extent based thresholding in fMRI analyses: pitfalls and recommendations, Neuroimage, vol.91, pp.412-419, 2014.

, Institut für Mathematik Karl-Liebknecht-Straße 24-25 14476 Potsdam, Germany E-mail: gilles.blanchard@math.uni-potsdam

, CNRS UPS IMT, F-31062 Toulouse Cedex 9

, Statistique et Modélisation, LPSM, 4, Place Jussieu, 75252 Paris cedex 05