The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Open accessAbstract
Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross‐validation, a method also used to tune decoders' hyper‐parameters. This paper is a review on cross‐validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within‐ and across‐subject predictions, on multiple datasets –anatomical and functional MRI and MEG– and simulations. Theory and experiments outline that the popular “leave‐one‐out” strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross‐validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross‐validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non‐sparse decoders. Graphical abstract Figure. No Caption available. HighlightsWe give a primer on cross‐validation to measure decoders predictive power.We assess on many datasets its practical use for decoding selection and tuning.Cross‐validation displays large confidence intervals, in particular leave one out.Default parameters on standard decoders can outperform parameter tuning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Self-citationThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940
MEG and EEG data analysis with MNE-Python
Gramfort, A. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00267 · Cited by 3,784
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
Statistical Parametric Mapping: The Analysis of Functional Brain Images
Penny, W. D., Friston, K. J., Ashburner, J., Kiebel, S. J., & Thomas, N. (2007) · UCL Discovery (University College London) · Cited by 3,656
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
Machine learning for neuroimaging with scikit-learn
Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., Gramfort, A., Thirion, B., & Varoquaux, G. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00014 · Cited by 2,384
Self-citationBeyond mind-reading: multi-voxel pattern analysis of fMRI data
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.07.005 · Cited by 2,271
Meinshausen, N., & Bühlmann, P. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00740.x · Cited by 2,215
Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python
Gorgolewski, K., Burns, C. D., Madison, C., Clark, D., Halchenko, Y. O., Waskom, M. L., & Ghosh, S. S. (2011) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2011.00013 · Cited by 2,186
Marcus, D. S., Wang, T. H., Parker, J., Csernansky, J. G., Morris, J. C., & Buckner, R. L. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.9.1498 · Cited by 2,163
Information-based functional brain mapping
Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600244103 · Cited by 2,113
MNE software for processing MEG and EEG data
Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., Parkkonen, L., & Hämäläinen, M. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.027 · Cited by 1,951
Self-citationChoosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
Decoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · cited 1×
“Decoding, ie predicting behavior or phenotypes from brain images or signals, has become a central tool in neuroimage data processing [20, 21, 24, 42, 58, 61].”
BackgroundDecoding mental states from brain activity in humans
Haynes, J.-D., & Rees, G. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1931 · Cited by 1,817
Prefrontal-Subcortical Pathways Mediating Successful Emotion Regulation
Wager, T. D., Davidson, M. L., Hughes, B. L., Lindquist, M. A., & Ochsner, K. N. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.006 · Cited by 1,761 · cited 1×
“We discriminate across subjects i) affective content with data from [59], ii) visual from narrative with data from [39], iii) famous, familiar, and scrambled faces from a visual-presentations dataset [22], and iv) left and right saccades in data from [26].”
Machine learning classifiers and fMRI: A tutorial overview
Pereira, F., Mitchell, T., & Botvinick, M. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.11.007 · Cited by 1,624
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180
Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.07.073 · Cited by 905
Mourão-Miranda, J., Bokde, A. L. W., Born, C., Hampel, H., & Stetter, M. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.06.070 · Cited by 704
Submodel Selection and Evaluation in Regression. The X-Random Case
Breiman, L., & Spector, P. (1992) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403680 · Cited by 669
Sitt, J. D., King, J.-R., El Karoui, I., Rohaut, B., Faugeras, F., Gramfort, A., Cohen, L., Sigman, M., Dehaene, S., & Naccache, L. (2014) · Brain · doi:10.1093/brain/awu141 · Cited by 608
Diffeomorphic registration using geodesic shooting and Gauss–Newton optimisation
Ashburner, J., & Friston, K. J. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.12.049 · Cited by 503
PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI Data
Hanke, M., Halchenko, Y. O., Sederberg, P. B., Hanson, S. J., Haxby, J. V., & Pollmann, S. (2009) · Neuroinformatics · doi:10.1007/s12021-008-9041-y · Cited by 483
Recruitment of an Area Involved in Eye Movements During Mental Arithmetic
Knops, A., Thirion, B., Hubbard, E. M., Michel, V., & Dehaene, S. (2009) · Science · doi:10.1126/science.1171599 · Cited by 407
Self-citationA Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives
Haynes, J.-D. (2015) · Neuron · doi:10.1016/j.neuron.2015.05.025 · Cited by 402
Support vector machines for temporal classification of block design fMRI data
LACONTE, S., STROTHER, S., CHERKASSKY, V, ANDERSON, J., & HU, X. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.01.048 · Cited by 379
Toward open sharing of task-based fMRI data: the OpenfMRI project
Poldrack, R. A., Barch, D. M., Mitchell, J. P., Wager, T. D., Wagner, A. D., Devlin, J. T., Cumba, C., Koyejo, O., & Milham, M. P. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00012 · Cited by 342
Fu, C. H. Y., Mourao-Miranda, J., Costafreda, S. G., Khanna, A., Marquand, A. F., Williams, S. C. R., & Brammer, M. J. (2008) · Biological Psychiatry · doi:10.1016/j.biopsych.2007.08.020 · Cited by 337
Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns
Yamashita, O., Sato, M.-A., Yoshioka, T., Tong, F., & Kamitani, Y. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.05.050 · Cited by 327
Face Repetition Effects in Implicit and Explicit Memory Tests as Measured by fMRI
Henson, R. N. A., Shallice, T., Gorno-Tempini, M. L., & Dolan, R. J. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.2.178 · Cited by 264
Sparse logistic regression for whole-brain classification of fMRI data
Ryali, S., Supekar, K., Abrams, D. A., & Menon, V. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.02.040 · Cited by 263
Strother, S. C., Anderson, J., Hansen, L. K., Kjems, U., Kustra, R., Sidtis, J., Frutiger, S., Muley, S., LaConte, S., & Rottenberg, D. (2002) · NeuroImage · doi:10.1006/nimg.2001.1034 · Cited by 248
Decoding the Large-Scale Structure of Brain Function by Classifying Mental States Across Individuals
Poldrack, R. A., Halchenko, Y. O., & Hanson, S. J. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02460.x · Cited by 239
Adding dynamics to the Human Connectome Project with MEG
Larson-Prior, L. J., Oostenveld, R., Della Penna, S., Michalareas, G., Prior, F., Babajani-Feremi, A., Schoffelen, J.-M., Marzetti, L., de Pasquale, F., Di Pompeo, F., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.056 · Cited by 223
Strother, S., Anderson, J., Frutiger, S., Muley, S., Rottenberg, D., Kjems, U., & Hansen, L. K. (2000) · NeuroImage · doi:10.1016/s1053-8119(00)91523-4 · Cited by 217
Social-Cognitive Deficits in Normal Aging
Moran, J. M., Jolly, E., & Mitchell, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5511-11.2012 · Cited by 205
Interpretable whole-brain prediction analysis with GraphNet
Grosenick, L., Klingenberg, B., Katovich, K., Knutson, B., & Taylor, J. E. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2012.12.062 · Cited by 201
Total Variation Regularization for fMRI-Based Prediction of Behavior
Michel, V., Gramfort, A., Varoquaux, G. P., Eger, E., & Thirion, B. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2113378 · Cited by 165
Self-citationPrediction and interpretation of distributed neural activity with sparse models
CARROLL, M., CECCHI, G., RISH, I., GARG, R., & RAO, A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.08.020 · Cited by 162
Consistency and variability in functional localisers
Duncan, K. J. K., Pattamadilok, C., Knierim, I., & Devlin, J. T. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.014 · Cited by 143
Total activation: fMRI deconvolution through spatio-temporal regularization
Karahanoğlu, F. I., Caballero‐Gaudes, C., Lazeyras, F., & Ville, D. V. D. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.01.067 · Cited by 140
How machine learning is shaping cognitive neuroimaging
Varoquaux, G. P., & Thirion, B. (2014) · GigaScience · doi:10.1186/2047-217x-3-28 · Cited by 132
Self-citationModel sparsity and brain pattern interpretation of classification models in neuroimaging
Rasmussen, P. M., Hansen, L. K., Madsen, K. H., Churchill, N. W., & Strother, S. C. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.09.011 · Cited by 126
The Evaluation of Preprocessing Choices in Single-Subject BOLD fMRI Using NPAIRS Performance Metrics
LaConte, S. M., Anderson, J., Muley, S., Ashe, J., Frutiger, S. A., Rehm, K., Hansen, L. K., Yacoub, E. S., Hu, X. P., Rottenberg, D. A., et al. (2002) · NeuroImage · doi:10.1006/nimg.2002.1300 · Cited by 124
Demirci, O., Clark, V. P., Magnotta, V. A., Andreasen, N. C., Lauriello, J., Kiehl, K. A., Pearlson, G. D., & Calhoun, V. D. (2008) · Brain Imaging and Behavior · doi:10.1007/s11682-008-9028-1 · Cited by 116
Classifier ensembles for fMRI data analysis: an experiment
Kuncheva, L. I., & Rodríguez, J. J. (2010) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2009.12.021 · Cited by 93
Identifying Predictive Regions from fMRI with TV-L1 Prior
Gramfort, A., Thirion, B., & Varoquaux, G. (2013) · International Workshop on Pattern Recognition in NeuroImaging · doi:10.1109/prni.2013.14 · Cited by 88
Self-citationAnalysis of Functional Magnetic Resonance Imaging in Python
Millman, J., & Brett, M. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.46 · Cited by 75
Detecting stable distributed patterns of brain activation using Gini contrast
Langs, G., Menze, B. H., Danial Lashkari, A., & Golland, P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.07.074 · Cited by 71
Varoquaux, G. P., Gramfort, A., & Thirion, B. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6447 · Cited by 62
Self-citationThickness network features for prognostic applications in dementia
Raamana, P. R., Weiner, M. W., Wang, L., & Beg, M. F. (2014) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2014.05.040 · Cited by 54
Self-citationExploring predictive and reproducible modeling with the single‐subject FIAC dataset
Chen, X., Pereira, F., Lee, W., Strother, S. C., & Mitchell, T. M. (2006) · Human Brain Mapping · doi:10.1002/hbm.20243 · Cited by 27
Churchill, N. W., Yourganov, G., & Strother, S. C. (2014) · Human Brain Mapping · doi:10.1002/hbm.22490 · Cited by 23
Improving Sparse Recovery on Structured Images with Bagged Clustering
Hoyos-Idrobo, A., Schwartz, Y., Varoquaux, G. P., & Thirion, B. (2015) · International Workshop on Pattern Recognition in NeuroImaging · doi:10.1109/prni.2015.30 · Cited by 15
Self-citationMapping cognitive ontologies to and from the brain
Schwartz, Y., Thirion, B., & Varoquaux, G. P. (2014) · Frontiers in Neuroinformatics · doi:10.3389/conf.fninf.2014.18.00084 · Cited by 14
Self-citationNew Results - Mapping cognitive ontologies to and from the brain
Varoquaux, G. P., Thirion, B., & Schwartz, Y. (2013) · Cited by 2
Self-citation
Cited by
Machine learning algorithm validation with a limited sample size
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019) · PLOS ONE · doi:10.1371/journal.pone.0224365 · Cited by 1,901
“Practically a distribution of performance measures could be produced by, for example, multiple times differently/randomly splitting the data to CV folds [27].”
BackgroundMachine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897
“Technically, this test is done via cross-validation: the available data is split in two, a first part, the train set used to fit themodel, and a second part, the test set used to test themodel (Pereira et al., 2009; Varoquaux et al., 2017).”
BackgroundMethodSelf-citationKong, R., Li, J., Orban, C., Sabuncu, M. R., Liu, H., Schaefer, A., Sun, N., Zuo, X.-N., Holmes, A. J., Eickhoff, S. B., et al. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhy123 · Cited by 593
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381
Metadata
- Authors
- Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2017-01
- Volume
- 145
- Issue
- Pt B
- Pages
- 166-179
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2421101021
- Cited by
- 792
- References
- 63
- FWCI
- 32.32
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.