Illusory generalizability of clinical prediction models
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model’s success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, such models based on data from specific populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
From Personalized machine learning guided intervention for optimizing lifestyle behaviors in depression: a pilot study (2026) · cites this paper as [26]
In the citing paper
“AI agents show significant potential in the realm of personalized treatment by leveraging a variety of patient-specific data, including genetic information, medical history, lifestyle factors, and treatment responses.77,88,89,90,91,92”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
In the citing paper
“AI-driven real-time patient monitoring systems are revolutionizing clinical care by enabling continuous, data-driven tracking of patient health.90,104,105”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987) · Schizophrenia Bulletin · doi:10.1093/schbul/13.2.261 · Cited by 21,378
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
(1976) · Medical Entomology and Zoology · Cited by 8,899
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
The clinical global impressions scale: applying a research tool in clinical practice.
Busner, J., & Targum, S. D. (2007) · PubMed Central · Cited by 2,760
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Research works printed but not linked: 3
- 10.1176/appi.ajp.2017.16121358 · DOI · Paper page
- 10.1093/schbul/sbv167 · DOI · Paper page
- 10.1001/jamapsychiatry.2017.0025 · DOI · Paper page
Cited by
Items 101–200 of 252
Neuromelanin-Sensitive MRI: A Biomarker for Treatment-Resistant Schizophrenia?
Vano, L. J., Veronese, M., McCutcheon, R. A., & Howes, O. D. (2024) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20240278 · Cited by 4
Are Psychiatric Nosologies Limiting the Success of Clinical Prediction Models?
Scott, I., & Nelson, B. (2024) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2024.2662 · Cited by 7
Facal, F., & Costas, J. (2024) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2024.111183 · Cited by 6
Byun, A. J. S., Lane, E., Langholm, C., Flathers, M., Hall, M., & Torous, J. B. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03054-5 · Cited by 5
“Although this study does not make clinical predictions, schizophrenia research in general involves an oversaturation of prediction models [30, 31] with little replication.”
Consistent differential effects of bupropion and mirtazapine in major depression
Strobl, E. V. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.119551 · Cited by 8
Learning Outcomes That Maximally Differentiate Psychiatric Treatments
Strobl, E. V., & Kim, S. (2026) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.70074 · Cited by 6
“However, the ever‐ increasing number of variables and complexity of machine learning makes it more difficult to deploy, generalize, interpret, and maintain these algorithms in everyday clinical practice (Chekroud et al. 2024; Falkai and Koutsouleris 2024).”
Lam, N. F. D., Cai, J., & Ng, K. H. (2025) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-025-01535-z · Cited by 5
“For example, when an AI model for predicting patient response to antipsychotic medication was tested using external, independent data sets [163], it demonstrated results much poorer than those obtained from the dataset on which the model was trained.”
Calderon, A., Irwin, M., Simon, N. M., Shear, M. K., Mauro, C., Zisook, S., Reynolds, C. F., & Malgaroli, M. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.119536 · Cited by 6
Artificial Intelligence and Psychotherapy
Bal, F. (2025) · Psikiyatride Guncel Yaklasimlar - Current Approaches in Psychiatry · doi:10.18863/pgy.1538432 · Cited by 7
Jin, Q., Jia, Q., Zhou, X., Jin, D., Song, X., Xie, Z., Alimujiang, A., Li, Y., Huang, J., & Ma, C. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3597467 · Cited by 5
“Therefore, there is an urgent need for DL models basedonclinicaldatafeaturestoimproveclassificationaccuracy, enhanceinterpretability,andovercomeillusorygeneralizability problems[18].”
Yakimov, V., Neuwinger, L., Weber, M. M., Brantl, M., Maurus, I., Sautner, J., John, M., Karslı, B., Hasanaj, G., Bungard, A., et al. (2025) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-025-02100-1 · Cited by 4
“To address the replication crisis in psychiatry [8, 28 ], we specifically aim to 1) externally validate previously identified prognostic biomarker candi - dates.”
InfluentialSelf-citationModel-Based Approaches to Investigating Mismatch Responses in Schizophrenia
Gütlin, D., McDermott, H. H., Grundei, M., & Auksztulewicz, R. (2024) · Clinical EEG and Neuroscience · doi:10.1177/15500594241253910 · Cited by 8
“Indeed, several studies have shown that many proposed procedures in schizophrenia prediction 176 – 178 and other clinical predictions 179,180 fail to generalize across different contexts, such as across subjects or across time ranges.”
Comprehensive evaluation of pure and hybrid collaborative filtering in drug repurposing
Réda, C., Vie, J.-J., & Wolkenhauer, O. (2025) · Scientific Reports · doi:10.1038/s41598-025-85927-x · Cited by 5
“This problem is pervasive in machine learning, particularly in health-related applications 62 , where differences in technicians and measurement protocols can induce a shift in the distribution of values in the data.”
Developing algorithmic psychiatry via multi-level spanning computational models
Halassa, M. M., Frank, M. J., Garety, P. A., Öngür, D., Airan, R. D., Sanacora, G., Dzirasa, K., Suresh, S., Fitzpatrick, S. M., & Rothman, D. L. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102094 · Cited by 8
Zhang, T., Zhao, X., Yeo, B. T. T., Huo, X., Eickhoff, S. B., & Ji, C. (2026) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaf218 · Cited by 5
Feyaerts, J., Brar, P. S., Sass, L. A., & Nelson, B. (2025) · The Lancet Psychiatry · doi:10.1016/s2215-0366(25)00244-5 · Cited by 6
Machine Learning Model for Response to Internet-Delivered CBT vs Antidepressant Medication
Lee, C. T., Richards, D., Heinzle, J., Hanlon, A. K., Lynch, K., Harty, S., Claus, N., O’Keane, V., Stephan, K. E., Whelan, R., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.41639 · Cited by 7
Rassaby, M. M., Spaulding, I. G., Stein, M. B., & Taylor, C. T. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.120733 · Cited by 4
A digital motor score for sensitive detection of progression in Huntington’s disease
Giboin, L., Simillion, C., Rennig, J., Bamdadian, A., Kinsella, F. C., Smith, A. V., McColgan, P., Lindemann, M., Lipsmeier, F., Wild, E. J., et al. (2025) · Brain · doi:10.1093/brain/awaf127 · Cited by 6
“29 One limitation is that the HDDMS was constructed partly using data from existing therapeutic trials that excluded people with HD who are early and late in their disease course.”
Aglinskas, A., Bergeron, A., & Anzellotti, S. (2025) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13829 · Cited by 3
“29 In order to test whether CVAE models can generalize and to avoid issues associated with k-fold cross-validation, 29 we adopted an out-of-sample approach.”
Artificial Intelligence and Schizophrenia: Crossing the Limits of the Human Brain
Melo, A., Romão, J., & Duarte, T. A. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1004805 · Cited by 5
Kim, B., Chae, M., Kim, Y., Kong, S., Kim, Y., Jung, T., Jeong, J., Park, S., Cho, C., Yeom, J. W., et al. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822654 · Cited by 4
“At the same time, the goal is to develop a generalizable prediction model by incorporating various generalization strategies, ensuring that the model can be effectively applied in real-world clinical settings [34, 35].”
InfluentialDominicus, L. S., Zandstra, M. G., Franse, J., Otte, W. M., Hillebrand, A., de Graaf, S., Ambrosen, K. S., Glenthoj, B. Y., Zalesky, A., Bojesen, K. B., et al. (2025) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13791 · Cited by 4
“9 We consider the strict inclusion criteria of AP-naïve individuals experiencing their fi rst episode of psychosis, de fi ned medication regimens, and the addition of EEG characteristics in the prediction model, as used in the present study, to be potential solutions for some of the problems…”
Langhammer, T., Unterfeld, C., Blankenburg, F., Erk, S., Fehm, L., Haynes, J., Heinzel, S., Hilbert, K., Jacobi, F., Kathmann, N., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-094110 · Cited by 4
“17 18 The reliance on small sample sizes from clinical trials, prioritising internal over external validity, provides a biased foundation for developing algorithms meant for broader healthcare application.”
Koutsouleris, N., Buciuman, M.-O., Vetter, C. S., Weyer, C., Zhutovsky, P., Tovar-Perdomo, S., Khuntia, A. T., Milaneschi, Y., Popovic, D., Ruef, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-3949072/v1 · Cited by 4
Self-citationRichter, M. F., Emden, D., Leenings, R., Winter, N. R., Mikolajczyk, R., Massag, J., Zwiky, E., Borgers, T., Redlich, R., Koutsouleris, N., et al. (2024) · medRxiv · doi:10.1101/2024.04.04.24305250 · Cited by 3
Self-citationA Biomarker-Centric Framework for the Prediction of Future Chronic PainPreprint
Fillingim, M., Tanguay-Sabourin, C., Parisien, M., Zare, A., Guglietti, G. V., Norman, J., Petre, B., Bortsov, A. V., Ware, M. A., Pérez, J., et al. (2024) · medRxiv · doi:10.1101/2024.04.19.24306101 · Cited by 3
Parola, A., Jessen, E. T., Rybner, A. S., Mortensen, M. D., Larsen, S. N., Simonsen, A., Lin, J. M., Zhou, Y., Wang, H., Koelkebeck, K., et al. (2024) · medRxiv · doi:10.1101/2024.11.06.24316839 · Cited by 3
“Furthermore, schizophrenia is a highly heterogeneous disorder, and models trained on small samples with homogeneous features, may not generalize well when applied to patients with other clinical or socio-demographic characteristics (Chekroud et al., 2024).”
DeepCRC: Precision Colorectal Cancer Diagnosis Using Deep Learning and Metagenomic Biomarkers
Wang, H., Li, Y., Feng, Y. X., Shen, K., Lu, C. F., & Xie, Z. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822422 · Cited by 3
“This finding underscores the importance of considering clinical context in predictive models, as these factors can provide essential insights into individual patient risk profiles [48].”
A commentary on frontostriatal salience network expansion in individuals in depression
Buehler, S. K., Zhang, R., & Roiser, J. P. (2025) · Psychoradiology · doi:10.1093/psyrad/kkaf005 · Cited by 2
Li, J., & Armstrong, B. C. (2025) · Open Mind · doi:10.1162/opmi.a.21 · Cited by 2
Hieronymus, F., Hieronymus, M., Sjöstedt, A., Nilsson, S., Näslund, J., Lisinski, A., & Østergaard, S. D. (2025) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.70037 · Cited by 2
Siekmeier, P. J., & Coyle, J. T. (2025) · Translational Psychiatry · doi:10.1038/s41398-025-03604-z · Cited by 6
“It should be noted that recent research [71] has indicated that well-validated AI models, when applied in new clinical settings — even with similar patient populations and outcome measures — may underperform [10].”
Behrens, S., Do, W. L., Wang, L., Revsine, M., Maestri, E., Jacob, A. R., Chang, C.-W., Forgues, M., Sardoo, A. M., Budhu, A., et al. (2025) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2025-011742 · Cited by 2
“Serum samples were obtained at the time of diagnosis and stored at −80°C.”
A brain-computer interface roadmap for diagnosing and treating neurological disorders
Sun, G., Wang, Y., Liu, H., Weng, L., Li, X., He, W., Sun, J., Liang, S., Kong, W., Dong, J.-Z., et al. (2025) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2025.100016 · Cited by 2
Pena-Gralle, A. P. B., Schnitzer, M. E., Boureguaa, S.-N., Morin, F., Legault, M., Sirois, C., Dragomir, A., & Blais, L. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103312 · Cited by 2
Crema, C., Boccali, A., Martinelli, A., Francesco, S. D., Meloni, S., Baronio, C. M., Gasparotti, R., Pedrini, L., Lanfredi, M., Pievani, M., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-31550-9 · Cited by 3
“Many studies fail to incorporate an independent test set, making it difficult to evaluate real-world applicability 15,16 .”
Liu, Z., Chen, W., Wang, X. Q., Huang, J., Gou, M., Han, H., Wang, J. P., Wang, D.-H., Chen, S., Xie, T., et al. (2026) · Schizophrenia Bulletin · doi:10.1093/schbul/sbag009 · Cited by 3
Adkinson, B. D., Rosenblatt, M., Sun, H., Dadashkarimi, J., Tejavibulya, L., Horien, C. L., Westwater, M. L., Rodriguez, R. X., Noble, S., & Scheinost, D. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02447-y · Cited by 6
“Neuroimaging predictive models will achieve real-world utility only if they can overcome these dataset-specific idiosyncrasies (that is, “dataset shifts”) 24,56,94–96 .”
Cui, J., He, Z., Lim, E., & Lim, C. E. D. (2026) · Digital Health · doi:10.1177/20552076261443585 · Cited by 7
“6,9,10 Chekroud et al. 11 highlighted the phenomenon of “illusory universality” in clinical prediction models.”
Μιλτιάδους, Α., Ntetska, A., Aspiotis, V., Moustakli, E., Tsipouras, M. G., Tzallas, A. T., Γιαννακέας, Ν., Glavas, E., Angelidis, P., & Tzimourta, K. D. (2026) · Cognitive Neurodynamics · doi:10.1007/s11571-026-10464-w · Cited by 6
“…internal subject-level validation (e.g., LOSO) does not guarantee transport-ability to independent clinical cohorts, as models may still fail when evaluated on fully external datasets—a phenomenon described as “illusory generalizability” in clinical prediction research (Chekroud et al. 2024).”
Generative AI and exacerbated suicide risk among vulnerable populations
Shannonhouse, L., Zhai, Y., Gutierrez, P. M., & Lin, Y.-W. D. (2026) · The Lancet Regional Health - Americas · doi:10.1016/j.lana.2026.101528 · Cited by 3
Mehareen, J., Lim, L. H. M., Adibi, A., Amegadzie, J. E., Xia, Y., Vera, M. A. D., Law, M. R., Sin, D. D., Bhatt, S. P., Quint, J. K., et al. (2026) · Thorax · doi:10.1136/thorax-2026-224814 · Cited by 2
“23 In contrast, models such as the Birmingham Lung Improvement Studies (BLISS) aim to predict any respiratory admissions among patients with established COPD, offering a more holistic, health-service utilisation, rather than a disease-specific outcome whose prevention is the corner-stone of COPD…”
Zhukovsky, P., Kuhn, M., Borchers, L. R., Ren, B., Woronko, S. E., Li, M., Lam, C. S. T., Zhang, E. M., Ressler, K. J., Brennan, B. P., et al. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00671-z · Cited by 3
“Data harmonization and cross-trial generalizability have been a challenge in previous studies 19 , and we did observe higher response rates in SMART-D compared with EMBARC.”
Wang, X., Dou, Y., Wang, Y., Sun, K., & Zhou, G. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1798803 · Cited by 6
“…emphasize multicenter external validation (7, 8, 11 – 14 Transportability Assessment: Evaluating model performance in populations with characteristics that differ from the development cohort, such as ethnicity, tumor subtype, disease stage, treatment era, or healthcare system (10, 30, 32, 34).”
Association Rule Mining as Knowledge Infusion into the Mechanics of an eXtended Classifier System
Novak, D., Verber, D., Fister, I., & Fister, I. (2024) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines63318.2024.10629091 · Cited by 2
Enhancing Brain Tumor Diagnosis through Adaptive Feature Aggregation based Transfer Learning
Iffath, F., Dey, L., & Gavrilova, M. L. (2024) · IEEE Conference on Industrial Electronics and Applications (ICIEA) · doi:10.1109/iciea61579.2024.10664936 · Cited by 2
“These techniques encompass a wide array of applications, including diagnostic imaging, clinical decision support systems, predictive analytics, and personalized medicine [2], [3].”
BackgroundPucher, G., Rostalski, T., Nensa, F., Kleesiek, J., Reinhardt, H. C., & Sauer, C. M. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.105526 · Cited by 6
“14 Using the example of an innovative diagnostic support algorithm for acute leukaemia classi fi cation, AI-PAL(7), our work showed a signi fi cantly lower performance during a simulated clinical implementation than the published data.”
Medical AI can transform medicine — but only if we carefully track the data it touches
Vaid, A. (2025) · Nature · doi:10.1038/d41586-025-01946-8 · Cited by 2
Harvey, A., Moreau, C. A., Kumar, K., Huguet, G., Urchs, S. G. W., Sharmarke, H., Jizi, K., Martin, C.-O., Younis, N., Tamer, P., et al. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00222 · Cited by 3
“While many studies have reported promising results ( Iyortsuun et al., 2023), generalising the prediction of ML models to completely independent data can lead to a sharp decrease in performance, as was recently evidenced for clinical trial stratification ( Chekroud et al., 2024).”
ResultMedical AI falters when assessing patients it hasn’t seen
Naddaf, M. (2024) · Nature · doi:10.1038/d41586-024-00094-9 · Cited by 3
Brain-phenotype predictions can survive across diverse real-world dataPreprint
Adkinson, B. D., Rosenblatt, M., Dadashkarimi, J., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.23.576916 · Cited by 2
Mkrtchian, A., Qiu, Z., Abir, Y., Erdmann, T., Dercon, Q., Sedlinská, T., Browning, M. D., Costello, H., & Huys, Q. J. M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.08.631868 · Cited by 2
Predicting the Predictable in the Psychiatric High RiskPreprint
Strobl, E. V. (2025) · medRxiv · doi:10.1101/2025.04.11.25325553 · Cited by 2
“Numerous models have unfortunately achieved only modest accuracy despite incorporating tens or even hundreds of predictors (Chekroud et al., 2024).”
Nan, J., Purpura, S., Jaiswal, S., Afshar, H., Maric, V., Manchanda, J. K., Taylor, C. T., Ramanathan, D., & Mishra, J. (2026) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-026-00062-3 · Cited by 1
“Further, such models based on data from speci fi c populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
Psychiatry and psychotherapy: the science of sitting with uncertainty
McLoughlin, A. (2024) · Irish Journal of Medical Science (1971 -) · doi:10.1007/s11845-024-03687-5 · Cited by 1
„Data-driven-Intensivmedizin“: Mangel an umfassenden Datensätzen
Hardenberg, J.-H. (2024) · Medizinische Klinik - Intensivmedizin und Notfallmedizin · doi:10.1007/s00063-024-01141-z · Cited by 1
“Das große Problem der Prädiktions-modelle ist die oft schlechte externe Validität [16, 17].”
Moving towards precision psychiatry: the hard nut of depression
Dukart, J., Lotter, L. D., & Eickhoff, S. B. (2024) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-024-02023-8 · Cited by 2
Fridell, A., Olsson, N. C., Coco, C., Bölte, S., & Jonsson, U. (2025) · Autism · doi:10.1177/13623613251331993 · Cited by 2
“Replication in other contexts and samples is needed, which is further underscored by recent research pointing to the elusive nature of reliable predictors of treatment outcomes in psychiatric research (Chekroud et al., 2024).”
How to bridge the gap between evidence-based and clinical management of endometriosis
Koninckx, P. R., Ussia, A., Miller, C. E., Guo, S., Lesaffre, E. M. E. H., Wattiez, A., & Adamyan, L. V. (2025) · Journal of Endometriosis and Uterine Disorders · doi:10.1016/j.jeud.2025.100123 · Cited by 1
Targeted Learning for Optimal Patient Assignment to Psychotherapy
Malkki, V. K., Saarni, S. E., Lutz, W., & Rosenström, T. H. (2025) · Psychotherapy Research · doi:10.1080/10503307.2025.2517567 · Cited by 1
“Overall, the gen - eralizability of clinical prediction algorithms has been worse than expected (Chekroud et al., 2024), and only a few attempts have been made to implement personalized treatment models into clinical practice (Delgadillo et al., 2020, 2021; Lutz et al., 2022a).”
Hammelrath, L., Brose, A., Heinrich, M., Zagorscak, P., Burchert, S., Langhammer, T., & Knaevelsrud, C. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-102392 · Cited by 4
“11 Most existing prediction studies relied on cross-sectional data (ie, pre-treatment self-reports).”
Functional Coupling and Longitudinal Outcome Prediction in First-Episode Psychosis
Pope, I. Z., Chopra, S., Holmes, A., Francey, S. M., O'Donoghue, B. N., Cropley, V., Nelson, B., Yuen, H. P., Allott, K., Álvarez‐Jiménez, M., et al. (2025) · Biological Psychiatry Global Open Science · doi:10.1016/j.bpsgos.2025.100589 · Cited by 1
“Therefore, the reliability and generalizability of our finding that resting-state FC did not predict clinical outcomes of patients with FEP should be assessed by performing cross-validation on a large, representative sample (83,100,101).”
Assmann, N., Kaiser, T., Herzog, P., Arntz, A. R., Klein, J. P., Faßbinder, E., & Schaich, A. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104899 · Cited by 1
“Models that have undergone proper external validation have often produced disappointing results (Chekroud et al., 2024; van Bronswijk, Bruijniks, et al., 2021).”
Coutts, F., Mena, S., Ucur, E., Fleischhacker, W. W., Kahn, R. S., Lieberman, J. A., Hasan, A., Howes, O. D., Correll, C. U., Koutsouleris, N., et al. (2025) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13914 · Cited by 3
“17 Here we present the fi rst models of antipsychotic response to generalize across continents and disease stages, representing a crucial step forward for precision psychiatry.”
Self-citationBurkhardt, G., Goerigk, S. A., Bulubas, L., Dechantsreiter, E., Keeser, D., Vogelmann, U., von Wartensleben, K., Wolf, J., Plewnia, C., Fallgatter, A. J., et al. (2025) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2025.111600 · Cited by 5
Self-citationXiao, Y., Guo, Y.-X., Yuan, Y.-W., & Ma, Z.-R. (2026) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2026.104957 · Cited by 1
Xiong, K., Li, X., Ding, S., Xie, S., Li, B., Zhang, Y., & Yang, L. (2026) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2026.03.002 · Cited by 1
Ruisong, W. A. N. G., Ding, X., Zhang, W., & Tieliu, S. (2026) · Pediatric Investigation · doi:10.1002/ped4.70061 · Cited by 2
“38,39 A systematic review of ML models for predicting extubation failure in preterm neonates exemplifies this concern, finding that most studies were limited by a high risk of bias and lacked external validation.”
Nilsson, A., Eijsbroek, V. C., Gu, Z., Kjell, K., Giorgi, S., Kotov, R., Ganesan, A. V., Schwartz, H. A., & Kjell, O. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261419036 · Cited by 4
“SEMP aims to address concerns that predictive models often under-perform in independent or prospective samples (e.g., Chekroud et al., 2024; Kernbach & Staartjes, 2022; Spasic & Nenadic, 2020; also see Essential Caveat About Generalization section).”
Fosset, M., Pensier, J., Jung, B., Siddiqi, A., Mamdani, M. M., & Celi, L. A. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001621 · Cited by 2
“This finding indicates that a successful single center RCT does not constitute evidence of widespread clinical applicability [38].”
Germani, É., Zeineddine, F., Mourad, C. J., & Albarqouni, S. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104258 · Cited by 3
Oliver, D., Arribas, M., Logeswaran, Y., Fusar‐Poli, L., Kempton, M. J., Hedges, E. P., Albiñana, C., Vassos, E., Sariaslan, A., Pettersson‐Yeo, W., et al. (2026) · Schizophrenia Bulletin · doi:10.1093/schbul/sbag088 · Cited by 2
“33 , 34 Despite this, it is crucial to externally validate these findings in independent datasets, 65 , 72 , 73 and potentially refit the models in larg er samples (eg, AMP-SCZ).”
Bergonzo, S., Mouchabac, S., Auxéméry, Y., Kulakovskyi, A., Altunlu, E., Leboyer, M., Bonnot, O., Bourla, A., Koutsouleris, N., & Frileux, S. (2026) · L Encéphale · doi:10.1016/j.encep.2026.08.007 · Cited by 2
Self-citationBrimacombe, M. B. (2024) · Bioengineering · doi:10.3390/bioengineering11121189 · Cited by 3
“However, while accurate in large datasets, supervised ANN predictive classification models may be only moderately predictive in small-to-moderate sample sizes and their results are not generalizable [27].”
Crema, C., Boccali, A., Martinelli, A., Francesco, S. D., Meloni, S., Baronio, C. M., Pedrini, L., Lanfredi, M., Archetti, D., Redolfi, A., et al. (2024) · medRxiv · doi:10.1101/2024.07.25.24310985 · Cited by 2
Lohner, V., Badhwar, A., Detcheverry, F. E., García, C. L., Gellersen, H. M., Khodakarami, Z., Lattmann, R., Li, R., Low, A., Mazo, C., et al. (2024) · medRxiv · doi:10.1101/2024.12.17.24319166 · Cited by 1
Xiao, W., Jones, M., Milne, E., & Dede, A. J. O. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.13.632659 · Cited by 1
Harvey, A., Moreau, C. A., Kumar, K., Guillaumeaf, Urchs, S. G. W., Sharmarke, H., Jizi, K., Martin, C.-O., Younis, N., Tamer, P., et al. (2023) · doi:10.31234/osf.io/srmx7
Searching for answers to the problem of TIME in human ARTs
Albertini, D. F. (2024) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-024-03063-3
“For those of you interested in a reality check, have a look at the papers by Chekroud et al., [6] and Petzchner [7] who take, respectively, a deep dive into the AI conun-drum in contemporary medicine and an even closer look into the practical aspects of moving forward in any subdiscipline…”
The path toward generalizable clinical prediction modelsPreprint
Hieronymus, F., Hieronymus, M., Sjöstedt, A., Nilsson, S., Näslund, J., Lisinski, A., & Østergaard, S. D. (2024) · medRxiv · doi:10.1101/2024.04.16.24305902
“9 From this, Chekroud et al. concluded that models predicting treatment outcomes in schizophrenia are highly context-dependent and have limited generalizability between trials.”
MethodInfluentialData-driven brain atrophy staging in spinocerebellar ataxia type 3Preprint
Baumeister, H., Schaprian, T., Wegner, P., Ferreira, M., Kuegler, D., França, M. C., de Rezende, T. J. R., Martinez, A. R. M., Jiang, H. F., Grobe‐Einsler, M., et al. (2024) · medRxiv · doi:10.1101/2024.05.29.24307992
Optimizing the Use of Artificial Intelligence in Cardiology in 2024
Ellis, S. G., & Kattan, M. W. (2024) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2024.05.016
Biological Characteristics of Treatment Outcomes in Schizophrenia
Das, M., Chakraborty, S., Das, N., & Mehta, U. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-32035-4_80-1
Collins, A. C., Price, G. D., Dainer‐Best, J., Haddox, D., Beevers, C. G., & Jacobson, N. C. (2024) · Cognitive Therapy and Research · doi:10.1007/s10608-024-10544-3
“…this limitation was circumvented by using nested CV, recent research has noted limitations with machine learning when generalizing to other samples (Chekroud et al., 2024), so it is important for future work to further investigate generalization in other samples, including within persons who meet…”
Predicting conversion to psychosis using machine learning: response to Cannon
Smucny, J., Cannon, T. D., Bearden, C. E., Addington, J., Cadenhead, K. S., Cornblatt, B. A., Keshavan, M. S., Mathalon, D. H., Perkins, D. O., Stone, W. S., et al. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1520173 · Cited by 1
“Although all algorithms here did perform above chance, the fact that they did not perform as well as a model trained within one dataset as in our previous study is in conceptual agreement with the fi ndings of Chekroud et al. (15).”
Medical Decision-Making Amid Insufficient, Unreliable or Conflicting Empirical EvidencePreprint
Tozzi, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5132501
Adkinson, B. D., Rosenblatt, M., Sun, H.-L., Dadashkarimi, J., Tejavibulya, L., Horien, C., Westwater, M., Rodriguez, R., Noble, S., & Scheinost, D. (2025) · bioRxiv · doi:10.1101/2025.03.12.642878 · Cited by 3
“Neuroimaging predictive models will achieve real-world utility only if they can overcome these dataset-specific idiosyncrasies (i.e., “dataset shifts”) 25,46,83–85 .”
Hammelrath, L., Brose, A., Heinrich, M., Zagorscak, P., Burchert, S., Langhammer, T., & Knaevelsrud, C. (2025) · medRxiv · doi:10.1101/2025.03.14.25323957
“Thus far, however, results are unsatisfactory: depending on the features of input, predictive accuracies often only slightly exceed chance level, like 62% for neuroimaging markers (8) and/or fail to generalize to external samples (9).”
Flores, R., Nihalani, R., Umur, S., Vigneault, F., & Novak, R. (2025) · Research Square · doi:10.21203/rs.3.rs-6130021/v1
Brimacombe, M. B. (2025) · International Journal of Statistics and Probability · doi:10.5539/ijsp.v14n1p58
Salvatore, M., Mondul, A. M., Friese, C. R., Hanauer, D. A., Xu, H., Pearce, C. L., & Mukherjee, B. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104853
Biological Characteristics of Treatment Outcomes in Schizophrenia
Das, M., Chakraborty, S., Das, N., & Mehta, U. M. (2025) · Springer eBooks · doi:10.1007/978-3-031-73368-0_80
Smucny, J. (2025) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.06.004
Trials for computational psychiatry
Huys, Q. J. M., & Browning, M. D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00879-6
“Examples again abound, but include the prediction of treatment response from standard electroencephalography recordings and clinical measures 3 .”
Kaplan, D. (2025) · Springer eBooks · doi:10.1007/978-3-032-08228-2_4
Comparative Analysis of Foundation Models for EEG-Based Alzheimer’s Disease Detection
Huang, N., Tzallas, A. T., He, Z., & Zhang, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228726
Auditing coherence through the principle of a single functional mappingPreprint
Haddadi, H., Kloskowski, A., & Mironowicz, P. (2025) · Research Square · doi:10.21203/rs.3.rs-8072048/v1
Womer, F. Y., Xiao, Y., Ngo, V., Zheng, J., & Wang, F. (2025) · Journal of Translational Internal Medicine · doi:10.1515/jtim-2025-0045 · Cited by 2
“…et al . tacitly tested the accuracy of supervised learning method in predicting clinical diagnoses or treatment effectiveness regarding MPDs. [169,170] Regretfully, despite the endeavors of both methodologies and data resources that have been made, neither of them have achieved satisfying model…”
May, P., Jung, J., Högner, M., & Bassermann, F. (2025) · Amyloid · doi:10.1080/13506129.2025.2605534 · Cited by 1
“This failure is an example of illusory generalizability, a phenomenon where models learn superficial, dataset-specific correlations that do not hold true when applied to out-of-sample data [20].”
Metadata
- Authors
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M.
- Affiliations
- Yale University (US)
- New York City Council (US)
- University of Augsburg (DE)
- University of Cologne (DE)
- Ludwig-Maximilians-Universität München (DE)
- Yale New Haven Hospital (US)
- Laureate Institute for Brain Research (US)
- Corresponding author
- Chekroud, Adam M.
- ORCID
- Venue
- Science
- Published
- 2024-01-12
- Volume
- 383
- Issue
- 6679
- Pages
- 164-167
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4390734451
- Cited by
- 251
- References
- 37
- Citations by year
- 2026: 76 · 2025: 96 · 2024: 78
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, generalizability, out-of-sample forecasting, antipsychotic drugs, schizophrenia, machine learning, context sensitivity, treatment outcome prediction, clinical decision-making
- MeSH terms
- Machine Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Female, Humans, Male, Middle Aged, Prognosis, Schizophrenia, Antipsychotic Agents, Models, Statistical, Treatment Outcome, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.