Establishment of Best Practices for Evidence for Prediction: A Review
Open accessAbstract
Importance: Great interest exists in identifying methods to predict neuropsychiatric disease states and treatment outcomes from high-dimensional data, including neuroimaging and genomics data. The goal of this review is to highlight several potential problems that can arise in studies that aim to establish prediction. Observations: A number of neuroimaging studies have claimed to establish prediction while establishing only correlation, which is an inappropriate use of the statistical meaning of prediction. Statistical associations do not necessarily imply the ability to make predictions in a generalized manner; establishing evidence for prediction thus requires testing of the model on data separate from those used to estimate the model's parameters. This article discusses various measures of predictive performance and the limitations of some commonly used measures, with a focus on the importance of using multiple measures when assessing performance. For classification, the area under the receiver operating characteristic curve is an appropriate measure; for regression analysis, correlation should be avoided, and median absolute error is preferred. Conclusions and Relevance: To ensure accurate estimates of predictive validity, the recommended best practices for predictive modeling include the following: (1) in-sample model fit indices should not be reported as evidence for predictive accuracy, (2) the cross-validation procedure should encompass all operations applied to the data, (3) prediction analyses should not be performed with samples smaller than several hundred observations, (4) multiple measures of prediction accuracy should be examined and reported, (5) the coefficient of determination should be computed using the sums of squares formulation and not the correlation coefficient, and (6) k-fold cross-validation rather than leave-one-out cross-validation should be used.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Potential wrongful inflation of accuracy estimates (e.g. related to small sample sizes, absence of testing model performance in an external sample, and problematic validation methods) precludes reliable immediate implementation in clinical practice.
“potential wrongful inflation of accuracy estimates (e.g. related to small sample sizes, absence of testing model performance in an external sample, and problematic validation methods), precluding reliable immediate implementation in clinical practice (Hosseini et al., 2020; Jacobucci, Littlefield, Millner, Kleiman, & Steinley, 2020; Poldrack, Huckins, & Varoquaux, 2020).”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Poldrack et al., 2020
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredIn predictive neuroimaging, ten of 57 studies may have leaked information by performing dimensionality reduction across the whole dataset prior to train/test splitting.
“In another review specific to predictive neuroimaging, ten of the 57 studies may have leaked information by performing dimensionality reduction across the whole dataset prior to train/test splitting18.”
From Data leakage inflates prediction performance in connectome-based machine learning models (2024) · cites this paper as [18]
The ML framework requires minimal assumptions and is more robust than parametric approaches regarding data heterogeneity.
“The ML framework requires minimal assumptions and is more robust than parametric approaches regarding data heterogeneity (Makridakis et al., 2018; Poldrack et al., 2019).”
From Multiclass characterization of frontotemporal dementia variants via multimodal brain network computational inference (2023) · cites this paper as Poldrack et al., 2019
The study used 80% of the sample for training and validation and 20% for testing, following best practices in ML.
“Following best practices in ML (Müller & Guido, 2016; Poldrack et al., 2019), we used 80% of the sample for training and validation and 20% of the sample for testing.”
From Multiclass characterization of frontotemporal dementia variants via multimodal brain network computational inference (2023) · cites this paper as Poldrack et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Self-citationThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
(1966) · Medical Entomology and Zoology · Cited by 10,293 · see citing works
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616 · see citing works
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696 · see citing works
Genome-wide association analysis identifies 13 new risk loci for schizophrenia
Multicenter Genetic Studies of Schizophrenia Consortium, Ripke, S., Psychosis Endophenotypes International Consortium, Wellcome Trust Case Control Consortium 2, O'Dushlaine, C., Chambert, K., Moran, J. L., Kähler, A. K., Akterin, S., Bergen, S. E., et al. (2013) · Nature Genetics · doi:10.1038/ng.2742 · Cited by 1,496 · see citing works
Building better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056 · see citing works
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962 · see citing works
Predicting Age Using Neuroimaging: Innovative Brain Ageing Biomarkers
Cole, J. H., & Franke, K. (2017) · Trends in Neurosciences · doi:10.1016/j.tins.2017.10.001 · Cited by 911 · see citing works
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897 · see citing works
Self-citationBest practices in data analysis and sharing in neuroimaging using MRI
Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4500 · Cited by 714 · see citing works
Self-citationCross-trial prediction of treatment outcome in depression: a machine learning approach
Chekroud, A. M., Zotti, R. J., Shehzad, Z., Gueorguieva, R., Johnson, M. K., Trivedi, M. H., Cannon, T. D., Krystal, J. H., & Corlett, P. R. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00471-x · Cited by 676 · see citing works
Lehman, C. D., Arao, R. F., Sprague, B. L., Lee, J. M., Buist, D. S. M., Kerlikowske, K. M., Henderson, L. M., Onega, T., Tosteson, A. N. A., Rauscher, G. H., et al. (2016) · Radiology · doi:10.1148/radiol.2016161174 · Cited by 667 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
Building a Science of Individual Differences from fMRI
Dubois, J., & Adolphs, R. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.014 · Cited by 610 · see citing works
Prediction as a Humanitarian and Pragmatic Contribution from Human Cognitive Neuroscience
Gabrieli, J. D. E., Ghosh, S., & Whitfield, S. (2015) · Neuron · doi:10.1016/j.neuron.2014.10.047 · Cited by 526 · see citing works
Koutsouleris, N., Meisenzahl, E. M., Davatzikos, C. A., Bottlender, R., Frodl, T., Scheuerecker, J., Schmitt, G., Zetzsche, T., Decker, P., Reiser, M. F., et al. (2009) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2009.62 · Cited by 400 · see citing works
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Carp, J. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00149 · Cited by 381 · see citing works
Ten simple rules for predictive modeling of individual differences in neuroimaging
Scheinost, D., Noble, S., Horien, C., Greene, A. S., Lake, E. M., Salehi, M., Gao, S., Shen, X., O'Connor, D., Barron, D. S., et al. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.057 · Cited by 381 · see citing works
A distributed brain network predicts general intelligence from resting-state human neuroimaging data
Dubois, J., Galdi, P., Paul, L. K., & Adolphs, R. (2018) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2017.0284 · Cited by 318 · see citing works
Berkman, E. T., & Falk, E. B. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721412469394 · Cited by 288 · see citing works
Neuroprediction of future rearrest
Aharoni, E., Vincent, G., Harenski, C., Calhoun, V., Sinnott-Armstrong, W., Gazzaniga, M., & Kiehl, K. (2013) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1219302110 · Cited by 238 · see citing works
Annual prevalence of diagnosed schizophrenia in the USA: a claims data analysis approach
Wu, E. Q., Shi, L., Birnbaum, H. G., Hudson, T. J., & Kessler, R. C. (2006) · Psychological Medicine · doi:10.1017/s0033291706008191 · Cited by 232 · see citing works
Luedtke, A., Sadikova, E., & Kessler, R. C. (2019) · Clinical Psychological Science · doi:10.1177/2167702618815466 · Cited by 143 · see citing works
Sprague, B. L., Arao, R. F., Miglioretti, D. L., Henderson, L. M., Buist, D. S. M., Onega, T., Rauscher, G. H., Lee, J. M., Tosteson, A. N. A., Kerlikowske, K. M., et al. (2017) · Radiology · doi:10.1148/radiol.2017161519 · Cited by 125 · see citing works
I TRIED A BUNCH OF THINGS: THE DANGERS OF UNEXPECTED OVERFITTING IN CLASSIFICATIONPreprint
Powell, M. A., Hosseini, M. N. G., Collins, J. J., Flintoft, C. C., Jones, W., Bowman, H., & Wyble, B. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/078816 · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [24]Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. Paper presented at: International Joint Conference on Artificial Intelligence; August 20–25, 1995; Montreal, Quebec, Canada.
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Evaluating Machine Learning Models and Their Diagnostic Value
Varoquaux, G., & Colliot, O. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_20 · Cited by 242 · see citing works
Self-citationData leakage inflates prediction performance in connectome-based machine learning models
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · Nature Communications · doi:10.1038/s41467-024-46150-w · Cited by 236 · see citing works
A distributed fMRI-based signature for the subjective experience of fear
Zhou, F., Zhao, W., Qi, Z., Geng, Y., Yao, S., Kendrick, K. M., Wager, T. D., & Becker, B. (2021) · Nature Communications · doi:10.1038/s41467-021-26977-3 · Cited by 155 · see citing works
I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data
Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.09.036 · Cited by 165 · see citing works
Ansart, M., Epelbaum, S., Bassignana, G., Bône, A., Bottani, S., Cattai, T., Couronné, R., Faouzi, J., Koval, I., Louis, M., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101848 · Cited by 116 · see citing works
Sanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.009 · Cited by 82 · see citing works
How to remove or control confounds in predictive models, with applications to brain biomarkers
Chyzhyk, D., Varoquaux, G., Milham, M., & Thirion, B. (2022) · GigaScience · doi:10.1093/gigascience/giac014 · Cited by 65 · see citing works
Self-citationStudying the neural representations of uncertainty
Walker, E. Y., Pohl, S., Denison, R. N., Barack, D. L., Lee, J., Block, N., Ma, W. J., & Meyniel, F. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01444-y · Cited by 56 · see citing works
Moment-to-Moment Brain Signal Variability Reliably Predicts Psychiatric Treatment Outcome
Månsson, K. N. T., Waschke, L., Manzouri, A., Furmark, T., Fischer, H., & Garrett, D. D. (2022) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.09.026 · Cited by 49 · see citing works
Chen, Z., Liu, X., Yang, Q., Wang, Y.-J., Miao, K., Gong, Z., Yu, Y., Leonov, A., Liu, C., Feng, Z., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1671 · Cited by 43 · see citing works
Postpartum depression: a developed and validated model predicting individual risk in new mothers
Munk-Olsen, T., Liu, X., Madsen, K. B., Kjeldsen, M.-M. Z., Petersen, L. V., Bergink, V., Skalkidou, A., Vigod, S. N., Frokjaer, V. G., Pedersen, C. B., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02190-8 · Cited by 36 · see citing works
Reporting details of neuroimaging studies on individual traits prediction: A literature survey
Yeung, A. W. K., More, S., Wu, J., & Eickhoff, S. B. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119275 · Cited by 36 · see citing works
Campa, F., Matias, C. N., Moro, T., Cerullo, G., Casolo, A., Teixeira, F. J., & Paoli, A. (2023) · Nutrients · doi:10.3390/nu15020278 · Cited by 33 · see citing works
Irremediability in psychiatric euthanasia: examining the objective standard
Nicolini, M. E., Jardas, E., Zarate, C. A., Gastmans, C., & Kim, S. Y. H. (2023) · Psychological Medicine · doi:10.1017/s0033291722002951 · Cited by 31 · see citing works
Addressing machine learning bias to foster energy justice
Chen, C.-F., Napolitano, R., Hu, Y., Kar, B., & Yao, B. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103653 · Cited by 24 · see citing works
Roesmann, K., Leehr, E. J., Böhnlein, J., Steinberg, C., Seeger, F., Schwarzmeier, H., Gathmann, B., Siminski, N., Herrmann, M. J., Dannlowski, U., et al. (2022) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2021.07.006 · Cited by 25 · see citing works
Santamaría-García, H., Baez, S., Aponte-Canencio, D. M., Pasciarello, G. O., Donnelly-Kehoe, P. A., Maggiotti, G., Matallana, D., Hesse, E., Neely, A., Zapata, J. G., et al. (2021) · Patterns · doi:10.1016/j.patter.2020.100176 · Cited by 18 · see citing works
Jahanshad, N., Lenzini, P., & Bijsterbosch, J. (2025) · Neuropsychopharmacology · doi:10.1038/s41386-024-01938-8 · Cited by 12 · see citing works
Teghipco, A., Newman-Norlund, R., Fridriksson, J., Rorden, C., & Bonilha, L. (2024) · Communications Medicine · doi:10.1038/s43856-024-00541-8 · Cited by 11 · see citing works
Gonzalez-Gomez, R., Ibañez, A., & Moguilner, S. (2023) · Network Neuroscience · doi:10.1162/netn_a_00285 · Cited by 10 · see citing works
- Authors
- Poldrack, R. A., Huckins, G., & Varoquaux, G.
- Affiliations
- Stanford University (US)
- Centre Inria de Saclay (FR)
- Venue
- JAMA Psychiatry
- Publisher
- American Medical Association (AMA)
- Published
- 2020-05-01
- Volume
- 77
- Issue
- 5
- Pages
- 534
- ISSN
- 2168-622X (ISSN-L)
- PMC
- PMC7250718
- OpenAlex
- W2990091959
- Cited by
- 738
- References
- 27
- Citations by year
- 2026: 98 · 2025: 126 · 2024: 144 · 2023: 118 · 2022: 116 · 2021: 94 · 2020: 42
- FWCI
- 49.05
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- JAMA Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (T32MH020016)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Keywords
- predictive validity, predictive modeling, cross-validation, area under the receiver operating characteristic, R-squared, k-fold cross-validation, leave-one-out cross-validation, neuroimaging, genomics, neuropsychiatric disorders, treatment outcome prediction, best practices
- MeSH terms
- Mental Disorders, Data Interpretation, Statistical, Humans, Predictive Value of Tests, Regression Analysis, ROC Curve, Statistics as Topic, Reproducibility of Results, Bias, Treatment Outcome, Practice Guidelines as Topic, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
