Large scale in transit computation of quantiles for ensemble runs

Alejandro Ribes (1) , Théophile Terraz (2) , Bertrand Iooss (3, 4, 5) , Yvan Fournier (1) , Bruno Raffin (6)

1 EDF - EDF
2 DATAMOVE - Data Aware Large Scale Computing
3 EDF R&D PRISME - Performance, Risque Industriel, Surveillance pour la Maintenance et l’Exploitation
4 IMT - Institut de Mathématiques de Toulouse UMR5219
5 GdR MASCOT-NUM - Méthodes d'Analyse Stochastique des Codes et Traitements Numériques
6 UGA [2016-2019] - Université Grenoble Alpes [2016-2019]

While estimating the uncertainties of numerical simulation model outputs, quantiles are important statistical quantities that can be used in risk analysis, outlier detection or computation of confidence intervals. Quantiles being order statistics, the classical approach for their computation requires availability of the full sample before ranking it. In numerical simulation, this approach is not suitable at exascale as large ensembles of model runs would need to gather a prohibitively large amount of data. This paper solves this problem by using an iterative approach based on the stochastic quan-tile algorithm of Robbins-Monro whose parameters are finely tuned in order to gain robustness. The computational part of the approach relies on the Melissa framework, a file avoiding, adaptive, fault tolerant and elastic framework. Quantiles are updated on-the-fly as soon as the in transit parallel server receives results from one of the running simulations. We validate our approach on a use case based on 3000 fluid dynamics parallel simulations of 6M hexahedra and 100 time-steps. This validation case was executed on two supercomputers, avoiding 11 TB of file storage per execution. Ubiquitous spatio-temporal maps of quantiles and inter-quantile based intervals are then produced via our robustly tuned Robbins-Monro algorithm.

Mots clés: In Situ Data Processing; Parametric Studies; Uncertainty Quantification; Iterative Statistics; Quantile; Robbins-Monro

Domaines: Statistiques [math.ST]

Alejandro Ribes, Théophile Terraz, Bertrand Iooss, Yvan Fournier, Bruno Raffin. Large scale in transit computation of quantiles for ensemble runs. 2019. ⟨hal-02016828v1⟩