Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross‐validation, a method also used to tune decoders' hyper‐parameters. This paper is a review on cross‐validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within‐ and across‐subject predictions, on multiple datasets –anatomical and functional MRI and MEG– and simulations. Theory and experiments outline that the popular “leave‐one‐out” strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross‐validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross‐validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non‐sparse decoders. Graphical abstract Figure. No Caption available. HighlightsWe give a primer on cross‐validation to measure decoders predictive power.We assess on many datasets its practical use for decoding selection and tuning.Cross‐validation displays large confidence intervals, in particular leave one out.Default parameters on standard decoders can outperform parameter tuning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Self-citationThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,464
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940 · cited 4×
“Figure 4: Varying amount of regularization on the face vs house discrimination in the Haxby 2001 data [19].”
BackgroundMethodInfluentialMEG and EEG data analysis with MNE-Python
Gramfort, A. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00267 · Cited by 3,784
Statistical Parametric Mapping: The Analysis of Functional Brain Images
Penny, W. D., Friston, K. J., Ashburner, J., Kiebel, S. J., & Thomas, N. (2007) · UCL Discovery (University College London) · Cited by 3,672
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,663
Cited by
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
“Practically a distribution of performance measures could be produced by, for example, multiple times differently/randomly splitting the data to CV folds [27].”
BackgroundMachine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“Technically, this test is done via cross-validation: the available data is split in two, a first part, the train set used to fit themodel, and a second part, the test set used to test themodel (Pereira et al., 2009; Varoquaux et al., 2017).”
BackgroundMethodSelf-citationKong, R., Li, J., Orban, C., Sabuncu, M. R., Liu, H., Schaefer, A., Sun, N., Zuo, X.-N., Holmes, A. J., Eickhoff, S. B., et al. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhy123 · Cited by 593
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381
“More generally, best practices appropriate to the stage of the discovery (i.e., confirmatory or exploratory) alleviate many multiple testing concerns (Wagenmakers et al., 2012).”
BackgroundLi, J., Kong, R., Liégeois, R., Orban, C., Tan, Y., Sun, N., Holmes, A. J., Sabuncu, M. R., Ge, T., & Yeo, B. T. T. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.04.016 · Cited by 380
The need to approximate the use-case in clinical machine learning
Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix019 · Cited by 367
“For CV to be valid, training and test sets need to be independent [30].”
Gilron, R., Little, S., Perrone, R., Wilt, R., de Hemptinne, C., Yaroshinsky, M. S., Racine, C. A., Wang, S. S., Ostrem, J. L., Larson, P. S., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00897-5 · Cited by 343
Benchmarking functional connectome-based predictive models for resting-state fMRI
Dadi, K., Rahim, M., Abraham, A., Chyzhyk, D., Milham, M., Thirion, B., & Varoquaux, G. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.062 · Cited by 333
Self-citationRobust EEG-based cross-site and cross-protocol classification of states of consciousness
Engemann, D. A., Raimondo, F., King, J.-R., Rohaut, B., Louppe, G., Faugeras, F., Annen, J., Cassol, H., Gosseries, O., Fernandez-Slezak, D., et al. (2018) · Brain · doi:10.1093/brain/awy251 · Cited by 329
Self-citationSchulz, M.-A., Yeo, B. T. T., Vogelstein, J. T., Mourao-Miranada, J., Kather, J. N., Kording, K., Richards, B., & Bzdok, D. (2020) · Nature Communications · doi:10.1038/s41467-020-18037-z · Cited by 315
Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior
Kong, R., Yang, Q., Gordon, E., Xue, A., Yan, X., Orban, C., Zuo, X.-N., Spreng, N., Ge, T., Holmes, A., et al. (2021) · Cerebral Cortex · doi:10.1093/cercor/bhab101 · Cited by 254
Evaluating Machine Learning Models and Their Diagnostic Value
Varoquaux, G., & Colliot, O. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_20 · Cited by 242
Self-citationChen, J., Tam, A., Kebets, V., Orban, C., Ooi, L. Q. R., Asplund, C. L., Marek, S., Dosenbach, N. U. F., Eickhoff, S. B., Bzdok, D., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-29766-8 · Cited by 224
Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging
Kragel, P. A., Koban, L., Barrett, L. F., & Wager, T. D. (2018) · Neuron · doi:10.1016/j.neuron.2018.06.009 · Cited by 215
Data leakage inflates prediction performance in connectome-based machine learning models
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · Nature Communications · doi:10.1038/s41467-024-46150-w · Cited by 236
Variable selection and validation in multivariate modelling
Shi, L., Westerhuis, J. A., Rosén, J., Landberg, R., & Brunius, C. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty710 · Cited by 205
“…results highlight the potential impact of false positive findings in small datasets and indicate that even the most careful cross-validation scheme may not be able to fully safeguard against overfitting in under-determined systems with a low number of samples (Rao and Fung, 2006; Varoquaux, 2017).”
Koster, R., Chadwick, M. J., Chen, Y., Berron, D., Banino, A., Düzel, E., Hassabis, D., & Kumaran, D. (2018) · Neuron · doi:10.1016/j.neuron.2018.08.009 · Cited by 191
Deep learning for small and big data in psychiatry
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0767-z · Cited by 189
Inference in the age of big data: Future perspectives on neuroscience
Bzdok, D., & Yeo, B. T. T. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.061 · Cited by 189
“19 practice, cross-validation procedures are frequently used to quantify out-of-sample performance by an unbiased estimate of a model's capacity to generalize to data samples acquired in the future (Dwork et al., 2015; Varoquaux et al., 2016).”
Wilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
“This also contributed to a relatively lower reduction in optimistic bias when using the theoretically validated approach to reduce optimistic bias resulting from model tuning in cross-validation (nested methods) [12,30,32].”
BackgroundMethodTAPAS: An Open-Source Software Package for Translational Neuromodeling and Computational Psychiatry
Frässle, S., Aponte, E. A., Bollmann, S., Brodersen, K. H., Do, C. T., Harrison, O. K., Harrison, S. J., Heinzle, J., Iglesias, S., Kasper, L., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.680811 · Cited by 169
Samper-González, J., Burgos, N., Bottani, S., Fontanella, S., Lu, P., Marcoux, A., Routier, A., Guillon, J., Bacci, M., Wen, J., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.08.042 · Cited by 169
I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data
Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.09.036 · Cited by 165
Recommendations and future directions for supervised machine learning in psychiatry
Cearns, M., Hahn, T., & Baune, B. T. (2019) · Translational Psychiatry · doi:10.1038/s41398-019-0607-2 · Cited by 127
Dinga, R., Marquand, A. F., Veltman, D. J., Beekman, A. T. F., Schoevers, R. A., van Hemert, A. M., Penninx, B. W. J. H., & Schmaal, L. (2018) · Translational Psychiatry · doi:10.1038/s41398-018-0289-1 · Cited by 125
Systematic misestimation of machine learning performance in neuroimaging studies of depression
Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D. M. A., Emden, D., Winter, N. R., Leenings, R., Eickhoff, S. B., Kircher, T., et al. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01020-7 · Cited by 120
“observed that LOOCV leads to unstable and biased estimates, concluding that repeated random splits should be preferred [14].”
Ansart, M., Epelbaum, S., Bassignana, G., Bône, A., Bottani, S., Cattai, T., Couronné, R., Faouzi, J., Koval, I., Louis, M., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101848 · Cited by 116
Kamagata, K., Zalesky, A., Hatano, T., Ueda, R., Di Biase, M. A., Okuzumi, A., Shimoji, K., Hori, M., Caeyenberghs, K., Pantelis, C., et al. (2017) · Human Brain Mapping · doi:10.1002/hbm.23628 · Cited by 106
“This has been shown to provide less biased and more stable classification accuracy estimates than leave-one-out methods [Varoquaux et al., 2017].”
MethodProgress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
Dhamala, E., Yeo, B. T. T., & Holmes, A. J. (2023) · Biological Psychiatry · doi:10.1016/j.biopsych.2022.09.024 · Cited by 91
“Guidelines for implementing predictive models to study behavioral variability based on neuroimaging data have been proposed(5-11), but there has not yet been a review of how different methodological considerations can influence the models.”
Vahid, A., Mückschel, M., Stober, S., Stock, A.-K., & Beste, C. (2020) · Communications Biology · doi:10.1038/s42003-020-0846-z · Cited by 87
Pulini, A. A., Kerr, W. T., Loo, S. K., & Lenartowicz, A. (2019) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2018.06.003 · Cited by 85
“Potential pitfalls of applying classification approaches to neuropsychiatric data have been discussed extensively (1,3,41,42).”
BackgroundInfluentialSanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.009 · Cited by 82
Albizu, A., Fang, R., Indahlastari, A., O’Shea, A., Stolte, S. E., See, K. B., Boutzoukas, E. M., Kraft, J. N., Nissim, N. R., & Woods, A. J. (2020) · Brain Stimulation · doi:10.1016/j.brs.2020.10.001 · Cited by 79
Ghasemzadeh, H., Hillman, R. E., & Mehta, D. D. (2024) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2023_jslhr-23-00273 · Cited by 72
Riganello, F., Larroque, S. K., Bahri, M. A., Heine, L., Martial, C., Carrière, M., Charland-Verville, V., Aubinet, C., Vanhaudenhuyse, A., Chatelle, C., et al. (2018) · Frontiers in Neurology · doi:10.3389/fneur.2018.00769 · Cited by 70
“According to the best standards in machine learning for neuroimagery, we conducted a 10-fold cross-validation test to evaluate the generalizable performance of this model (85) (Table 2), which showed that this model is quite robust and reliable, with 90% accuracy, 7% false positive rate, 13% false negative rate and a…”
MethodHow to remove or control confounds in predictive models, with applications to brain biomarkers
Chyzhyk, D., Varoquaux, G., Milham, M., & Thirion, B. (2022) · GigaScience · doi:10.1093/gigascience/giac014 · Cited by 65
“Prediction accuracy is ideally measured on an independent validation set, but most often, no large independent validation set is available and a cross-validation procedure, which iteratively separates train and test sets [21],is used.”
InfluentialSelf-citationHe, C., Bai, Y., Wang, Z., Fan, D., Wang, Q., Liu, X., Zhang, H., Zhang, H., Zhang, Z., Yao, H., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117428 · Cited by 44
Chen, Z., Liu, X., Yang, Q., Wang, Y.-J., Miao, K., Gong, Z., Yu, Y., Leonov, A., Liu, C., Feng, Z., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1671 · Cited by 43
Reporting details of neuroimaging studies on individual traits prediction: A literature survey
Yeung, A. W. K., More, S., Wu, J., & Eickhoff, S. B. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119275 · Cited by 36
Chen, Z., Hu, B., Liu, X., Becker, B., Eickhoff, S. B., Miao, K., Gu, X., Tang, Y., Dai, X., Li, C., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-02941-4 · Cited by 35
Model-based prediction of muscarinic receptor function from auditory mismatch negativity responses
Schöbi, D., Homberg, F., Frässle, S., Endepols, H., Moran, R. J., Friston, K. J., Tittgemeyer, M., Heinzle, J., & Stephan, K. E. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118096 · Cited by 25
Zhutovsky, P., Vijverberg, E. G. B., Bruin, W. B., Thomas, R. M., Wattjes, M. P., Pijnenburg, Y. A. L., van Wingen, G. A., & Dols, A. (2019) · Journal of Alzheimer's Disease · doi:10.3233/jad-181004 · Cited by 20
“Additionally, having more data would allow for a bigger test set in each iteration of the cross-validation procedure which would allow for less variance in the estimation of the generalization performance of the classifier [40].”
Donini, M., Monteiro, J. M., Pontil, M., Hahn, T., Fallgatter, A. J., Shawe-Taylor, J., & Mourão-Miranda, J. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.053 · Cited by 19
Jahanshad, N., Lenzini, P., & Bijsterbosch, J. (2025) · Neuropsychopharmacology · doi:10.1038/s41386-024-01938-8 · Cited by 12
“Prior work has established important best practice guidelines for such models [52 – 56].”
MethodTeghipco, A., Newman-Norlund, R., Fridriksson, J., Rorden, C., & Bonilha, L. (2024) · Communications Medicine · doi:10.1038/s43856-024-00541-8 · Cited by 11
“Partitions used to tune models were nested to estimate model generalizability more reliably and mitigate the risk of over fi tting 65 – 67 .”
Metadata
- Authors
- Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2017-01
- Volume
- 145
- Issue
- Pt B
- Pages
- 166-179
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2421101021
- Cited by
- 792
- References
- 63
- FWCI
- 32.32
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.