Systematic misestimation of machine learning performance in neuroimaging studies of depression
Open accessThe DOI matches its Crossref record; 27 of the 28 references it cites were found in a registry or an index.
Abstract
We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls based on neuroimaging data. Drawing upon structural MRI data from a balanced sample of N = 1868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes ( N = 4 to N = 150) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes ( N = 20), we observe accuracies of up to 95%. For medium sample sizes ( N = 100) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“This is particularly true given recent discussions about how machine learning methods could misestimate results in smaller studies (Flint et al., 2021; Nielsen et al., 2020).”
From Understanding and predicting future relapse in depression from resting state functional connectivity and self-referential processing (2023) · cites this paper as Flint et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“Finally, a linear support-vector machine (SVM) with default parameters (Scikit-learn [22], v0.”
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Murray, D. D., Suzuki, K., Law, M., Trebicka, J., Neuhaus, J., Wentworth, D., Johnson, M., Vjecha, M. J., Kelleher, A. D., Emery, S., et al. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139981 · Cited by 2,740
Small sample size effects in statistical pattern recognition: Recommendations for practitioners
Raudys, S. J., & Jain, A. K. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.75512 · Cited by 1,384 · cited 1×
“, measured variables) and a complex classification rule being fit to a dataset [11].”
Wïttchen, H., Wunderlich, U., Gruschwitz, S., & Zaudig, M. (1997) · Max Planck Institute for Plasma Physics · Cited by 1,106
Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls
Arbabshirani, M. R., Plis, S., Sui, J., & Calhoun, V. D. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.02.079 · Cited by 912 · cited 3×
“With a median N of less than 100 in many published studies [10], this may seem challenging.”
InfluentialCross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897 · cited 3×
“Although these findings have sparked in-depth conceptual considerations [15], empirical investigations of this problem have been limited to specific cross-validation schemes and small test set sizes [16].”
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints
van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-137 · Cited by 809 · cited 1×
“Recent works have further reiterated this point [12].”
Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.038 · Cited by 792 · cited 1×
“observed that LOOCV leads to unstable and biased estimates, concluding that repeated random splits should be preferred [14].”
Understanding heterogeneity in clinical cohorts using normative models: Beyond case-control studies
Marquand, A. F., Rezek, I., Buitelaar, J., & Beckmann, C. F. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.12.023 · Cited by 683 · cited 1×
“, biotype identification [19] and normative modelling approaches [23]) remains an open question.”
Cited by
Digital medicine and the curse of dimensionality
Berisha, V., Krantsevich, C., Hahn, P. R., Hahn, S., Dasarathy, G., Turaga, P., & Liss, J. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00521-5 · Cited by 344
“14 ; this is indeed closely related to the fi le-drawer effect that has been observed in several fi elds 17 – 19 .”
BackgroundMethodResultInfluentialDeep learning for small and big data in psychiatry
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0767-z · Cited by 189
“The larger the test set, of course, the lower will be the uncertainty about the mean PE estimate, i.e., its variance or standard error (see also [80]).”
BackgroundMethodA Systematic Evaluation of Machine Learning–Based Biomarkers for Major Depressive Disorder
Winter, N. R., Blanke, J., Leenings, R., Ernsting, J., Fisch, L., Sarink, K., Barkhau, C., Emden, D., Thiel, K., Flinkenflügel, K., et al. (2024) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2023.5083 · Cited by 157
Self-citationQin, K., Lei, D., Pinaya, W. H. L., Pan, N., Li, W., Zhu, Z., Sweeney, J. A., Mechelli, A., & Gong, Q. (2022) · EBioMedicine · doi:10.1016/j.ebiom.2022.103977 · Cited by 126
“10 In recent years, the rapid development of multi-site collaborations has increased the availability of large-scale datasets, which is critical for training a reliable and generalizable model.”
BackgroundModern Methods of Diagnostics and Treatment of Neurodegenerative Diseases and Depression
Shusharina, N. N., Yukhnenko, D., Botman, S., Sapunov, V., Savinov, V., Kamyshov, G., Sayapin, D., & Voznyuk, I. A. (2023) · Diagnostics · doi:10.3390/diagnostics13030573 · Cited by 119
“The phenomenon of sustainable overestimation of the effectiveness of machine learning methods developed for use in psychiatry was studied in [132].”
Background
Metadata
- Authors
- Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D. M. A., Emden, D., Winter, N. R., Leenings, R., Eickhoff, S. B., Kircher, T., Krug, A., Nenadic, I., Arolt, V., Clark, S., Baune, B. T., Jiang, X., Dannlowski, U., & Hahn, T.
- Affiliations
- University of Münster (DE)
- The University of Melbourne (AU)
- Forschungszentrum Jülich (DE)
- Philipps University of Marburg (DE)
- The University of Adelaide (AU)
- Corresponding author
- Dannlowski, Udo
- ORCID
- Venue
- Neuropsychopharmacology
- Published
- 2021-05-06
- Volume
- 46
- Issue
- 8
- Pages
- 1510-1517
- ISSN
- 0893-133X (ISSN-L)
- 1740-634X
- PMC
- PMC8209109
- OpenAlex
- W2996053974
- Cited by
- 120
- References
- 28
- Citations by year
- 2026: 17 · 2025: 21 · 2024: 20 · 2023: 26 · 2022: 20 · 2021: 14
- FWCI
- 10.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Neuropsychopharmacology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- UWA Profiles and Research Repository (UWA) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Interdisciplinary Center for Clinical Research (IZKF) of the medical faculty of Münster: SEED 11/18
- Deutsche Forschungsgemeinschaft (KI 588/14–1, KI 588/14–2, KI 588/15-1, KI 588/17-1, KR 3822/5–1, NE 2254/3-1, NE2254/4-1, DA 1151/5–1, DA 1151/5–2, DA 1151/6-1, SFB-TRR58, HA 7070/2–2, HA7070/3, HA7070/4)
- Interdisziplinäres Zentrum für Klinische Forschung, Universitätsklinikum Würzburg (Dan3/012/17)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Pharmacology (medical) › Treatment of Major Depression
- Keywords
- major depressive disorder, machine learning model performance, sample size, structural MRI, neuroimaging classification, test set size, prediction accuracy
- MeSH terms
- Machine Learning, Depression, Depressive Disorder, Major, Major Depressive Disorder, Humans, Magnetic Resonance Imaging, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Systematic misestimation of machine learning performance in neuroimaging studies of depression. https://app.cochat.ai/papers/10.1038/s41386-021-01020-7
- The paper
- Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D. M. A., Emden, D., Winter, N. R., Leenings, R., Eickhoff, S. B., Kircher, T., Krug, A., Nenadic, I., Arolt, V., Clark, S., Baune, B. T., Jiang, X., Dannlowski, U., & Hahn, T. (2021). Systematic misestimation of machine learning performance in neuroimaging studies of depression. Neuropsychopharmacology. https://doi.org/10.1038/s41386-021-01020-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.