Validation of Machine Learning Prediction Models

Luc Pronzato (1, 2) , Maria Joao Rendas (1)

1 Laboratoire d'Informatique, Signaux, et Systèmes de Sophia-Antipolis (I3S) / Equipe SYSTEMES
2 GdR MASCOT-NUM - Méthodes d'Analyse Stochastique des Codes et Traitements Numériques

We address the estimation of the Integrated Squared Error (ISE) of a predictor η(x) of an unknown function f learned using data acquired on a given design Xn. We consider ISE estimators that are weighted averages of the residuals of the predictor η(x) on a set of selected points Zm. We show that, under a stochastic model for f, minimisation of the mean squared error of these ISE estimators is equivalent to minimisation of a Maximum Mean Discrepancy (MMD) for a nonstationary kernel that is adapted to the geometry of Xn. Sequential Bayesian quadrature then yields sequences of nested validation designs that minimise, at each step of the construction, the relevant MMD. The optimal ISE estimate can be written in terms of the integral of a linear reconstruction, for the assumed model, of the square of the interpolator residuals over the domain of f. We present an extensive set of numerical experiments which demonstrate the good performance and robustness of the proposed solution. Moreover, we show that the validation designs obtained are space-filling continuations of Xn , and that correct weighting of the observed interpolator residuals is more important than the precise configuration Zm of the points at which they are observed.

Mots clés: Model Validation; Bayesian Quadrature; Maximum Mean Discrepancy; Experimental Design

Domaines: Statistiques [math.ST]

Luc Pronzato, Maria Joao Rendas. Validation of Machine Learning Prediction Models. The New England Journal of Statistics in Data Science, In press. ⟨hal-03818234⟩