Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Gaël Varoquaux (1, 2) , Pradeep A Reddy Raamana (3) , Denis A Engemann (4) , Andrés A Hoyos-Idrobo (2, 1) , Yannick A Schwartz (1, 2) , Bertrand A Thirion (1)

1 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
2 NEUROSPIN - Service NEUROSPIN
3 RRI - Rotman Research Institute at the Baycrest Centre
4 UPMC - Université Pierre et Marie Curie - Paris 6

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets –anatomical and functional MRI and MEG– and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.

Mots clés: cross-validation; decoding; fMRI; model selection; sparse; bagging; MVPA

Domaines: Ingénierie biomédicale; Sciences cognitives; Neurosciences; Machine Learning [stat.ML]; Imagerie médicale

Gaël Varoquaux, Pradeep A Reddy Raamana, Denis A Engemann, Andrés A Hoyos-Idrobo, Yannick A Schwartz, et al.. Assessing and tuning brain decoders: cross-validation, caveats, and guidelines. NeuroImage, 2016, ⟨10.1016/j.neuroimage.2016.10.038⟩. ⟨hal-01332785v2⟩