Model predictivity assessment: incremental test-set selection and accuracy evaluation

Elias Fekhari (1) , Bertrand Iooss (1, 2, 3) , Joseph Muré (1) , Luc Pronzato (4, 3) , Maria Joao Rendas (4)

1 EDF R&D PRISME - Performance, Risque Industriel, Surveillance pour la Maintenance et l’Exploitation
2 IMT - Institut de Mathématiques de Toulouse UMR5219
3 GdR MASCOT-NUM - Méthodes d'Analyse Stochastique des Codes et Traitements Numériques
4 I3S - Laboratoire d'Informatique, Signaux, et Systèmes de Sophia Antipolis

Unbiased assessment of the predictivity of models learnt by supervised machine-learning methods requires knowledge of the learned function over a reserved test set (not used by the learning algorithm). The quality of the assessment depends, naturally, on the properties of the test set and on the error statistic used to estimate the prediction error. In this work we tackle both issues, proposing a new predictivity criterion that carefully weights the individual observed errors to obtain a global error estimate, and using incremental experimental design methods to "optimally" select the test points on which the criterion is computed. Several incremental constructions are studied, including greedy-packing (coffee-house design), support points and kernel herding techniques. Our results show that the incremental and weighted versions of the latter two, based on Maximum Mean Discrepancy concepts, yield superior performance. An industrial test case provided by the historical French electricity supplier (EDF) illustrates the practical relevance of the methodology, indicating that it is an efficient alternative to expensive cross-validation techniques.

Mots clés: Design of experiments; Discrepancy; Gaussian process; Machine learning; Metamodel; Validation

Domaines: Statistiques [math.ST]

Elias Fekhari, Bertrand Iooss, Joseph Muré, Luc Pronzato, Maria Joao Rendas. Model predictivity assessment: incremental test-set selection and accuracy evaluation. 2022. ⟨hal-03523695v1⟩