Illusory generalizability of clinical prediction models
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model’s success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, such models based on data from specific populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
From Personalized machine learning guided intervention for optimizing lifestyle behaviors in depression: a pilot study (2026) · cites this paper as [26]
In the citing paper
“AI agents show significant potential in the realm of personalized treatment by leveraging a variety of patient-specific data, including genetic information, medical history, lifestyle factors, and treatment responses.77,88,89,90,91,92”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
In the citing paper
“AI-driven real-time patient monitoring systems are revolutionizing clinical care by enabling continuous, data-driven tracking of patient health.90,104,105”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987) · Schizophrenia Bulletin · doi:10.1093/schbul/13.2.261 · Cited by 21,378
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
(1976) · Medical Entomology and Zoology · Cited by 8,899
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
The clinical global impressions scale: applying a research tool in clinical practice.
Busner, J., & Targum, S. D. (2007) · PubMed Central · Cited by 2,760
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Research works printed but not linked: 3
- 10.1176/appi.ajp.2017.16121358 · DOI · Paper page
- 10.1093/schbul/sbv167 · DOI · Paper page
- 10.1001/jamapsychiatry.2017.0025 · DOI · Paper page
Cited by
Items 201–252 of 252
Azizmalayeri, M., de Keizer, N. F., Termorshuizen, F., Abu–Hanna, A., & Cinà, G. (2026) · Critical Care Medicine · doi:10.1097/ccm.0000000000006976
Parchure, S., Gupta, A., Kelkar, A. S., Vnenchak, L., Faseyitan, O. K., Medaglia, J. D., Harvey, D. Y., Coslett, H. B., & Hamilton, R. H. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.02.703416
D'Souza, A., Davidson, I. N., & Smucny, J. (2026) · Psychiatry Research · doi:10.1016/j.psychres.2026.117032
Cheng, A., Konova, A. B., Powers, A. R., Corlett, P. R., Levy, I., Gu, X., Huys, Q. J. M., Pushkarskaya, H., Fineberg, S. K., Hauser, T. U., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.02.009
Self-citationKhantee, W., Somsuphaprungyos, S., & Natho, P. (2025) · International Conference on Robotics, Automation and Artificial Intelligence (RAAI) · doi:10.1109/raai67517.2025.11423419
“At the same time, recent analyses warn that models boasting high internal accuracy may exhibit illusory generalizability across cohorts, underscoring the value of regularization, transparent feature handling, and prospective validation [14].”
YAPAY ZEKA İLE TERAPİ: TERAPÖTİK SÜREÇTE ALGORİTMALARIN ETKİSİ VE ETİK SORUNLAR
Pamfili, C., & Namli, N. I. (2026) · Dokuz Eylül Üniversitesi Sosyal Bilimler Enstitüsü Dergisi · doi:10.16953/deusosbil.1673372
Yang, X.-L., Zhang, T.-Y., Ren, X.-L., Wang, Y., Lv, H., Su, M.-F., Huo, X.-N., & Zhao, X. (2026) · European Journal of Psychiatry · doi:10.1016/j.ejpsy.2026.100351 · Cited by 1
Evaluating biomarkers and prediction models with E2P SimulatorPreprint
Karvelis, P., Felsky, D., Abi-Dargham, A., Horga, G., & Diaconescu, A. O. (2026) · eLife · doi:10.7554/elife.110646.1
Evaluating biomarkers and prediction models: predictive utility analysis with E2P SimulatorPreprint
Karvelis, P., Felsky, D., Abi-Dargham, A., Horga, G., & Diaconescu, A. O. (2026) · eLife · doi:10.7554/elife.110646
Sodnomdavaa, T., & Gantumur, U. (2026) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph23040496 · Cited by 1
“In particular, models evaluated using random data splits may perform well under controlled conditions but show reduced reliability when applied to future data due to temporal changes in patient characteristics and clinical practice [6].”
Ding, J., Chiu, J. C., Moon, S., Ren, Y., Turner, D., Shoval, G., Niv, Y., & Berwian, I. M. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1774560
“While our test dataset does not constitute a fully external dataset, as highlighted in guidance on prediction model evaluation and generalizability (83, 84), this approach has the following advantages given our dataset size: (cid:129) Preventing data leakage, which often occurs with improper…”
Weng, Y., Boyle, R. T., Lee, C. T., Quinn, D. M., Earley, C., Splittgerber, M., Zhang, L., Franzen, L., Banaschewski, T., Bokde, A. L. W., et al. (2026) · Neuropsychopharmacology · doi:10.1038/s41386-026-02401-6 · Cited by 1
Kjell, O., Ganesan, A. V., Boyd, R. L., Oltmanns, J. R., Rivero, A., Feltman, S. M., Carr, M. A., Alves, J., Luft, B. J., Kotov, R., et al. (2026) · Clinical Psychological Science · doi:10.1177/21677026261439026 · Cited by 1
Thompson, K. N., Couvy‐Duchesne, B., Lee, S., Geurgas, R., Jeong, Y., Arirangan, S., Newman, S., Tropf, F. C., & Wedow, R. (2026) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-026-03098-0 · Cited by 1
Higson‐Sweeney, N., Peycheva, A., & Breedvelt, J. (2026) · BMJ Mental Health · doi:10.1136/bmjment-2025-302327
Self‐harm in individuals with substance use disorders: Predictive factors and risk model
Yu, R., Tracy, D. K., Sinclair, J., Brikell, I., Lichtenstein, P., Larsson, H., & Fazel, S. (2026) · Addiction · doi:10.1111/add.70465 · Cited by 1
“Research has highlighted challenges in generalizability and practical utility for predictive models in mental health [26].”
Machine Learning for Predicting Clinical Trial Outcomes
Aldhahab, K., Elgamal, Z. M., & Barakat, M. (2025) · International Conference on Business Intelligence for Technology Innovation (ICBITI) · doi:10.1109/icbiti65527.2025.11500912
“Chekroud et al.[16] performed an analysis of the generalizability of machine learning algorithms in forecasting the outcomes of schizophrenia therapies.”
Task-aligned outcome learning in psychiatry: reducing endpoint dilution
Strobl, E. V. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1832978 · Cited by 4
“These strategies can help, but they also reduce interpretability (33), raise deployment burden (34), and often weaken transportability across hospitals or health systems (35, 36).”
Schuyteneer, E. D., Leenaerts, N., Vrieze, E., Meule, A., Kolar, D., Wuttke, E. P., & Voderholzer, U. (2026) · International Journal of Eating Disorders · doi:10.1002/eat.70129 · Cited by 1
Zhang, T., Huo, X., Chen, J., & Zhao, X. (2026) · BMC Medicine · doi:10.1186/s12916-026-04955-0
“This systematic assessment was to ensure that our findings were not driven by idiosyncrasies in how we defined treatment response and helps establish the robustness of our subtyping in indicating remission status [77, 78].”
Schatzkästchen oder Büchse der Pandora? – Klinische Ethik und Künstliche Intelligenz
Beil, M., Janssens, U., & Fuest, K. (2026) · Springer eBooks · doi:10.1007/978-3-662-72378-4_16
AI-based Psychiatric Prediction in Youth: Neuroimaging Provides Minimal Gains Beyond ConfoundsPreprint
Gijsen, S., Ibrahim, A., Tochadse, M., & Ritter, K. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.22.727174
Goldberg, S. B., Baldwin, S. A., Flynn, A. W. P., Babins‐Wagner, R., Caperton, D. D., Williams, C. Y., Hamm, E. H., Lam, S. U., Cozart, J. K., Deole, G., et al. (2026) · Clinical Psychological Science · doi:10.1177/21677026261424222
Pre-treatment Chemotherapy Resistance Prediction in Endometrial Cancer: A Machine Learning Approach
Niny, S., Kissi, C., Reddy, C. P., & Srifi, M. N. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28819-6_24
Jensen, C. M., Marandi, R. Z., Moestrup, K. S., Mourad, A., Lora, A. J. M., Sherman, B. T., Vock, D. M., Nordwall, J. A., Carson, J. M., Peiffer‐Smadja, N., et al. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag079
“14,15 This study has several strengths.”
Wei, Q., Cui, M., Qi, Y., Zhang, Q., Liu, Z., Yu, C., Hobbs, F. D. R., Christiani, D. C., Li, Y., Zhang, R., et al. (2026) · The Innovation Medicine · doi:10.59717/j.xinn-med.2026.100229
Chu, A., & Pettine, W. W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30710-1_49
Wei, S., Gao, M., & Wang, Y. (2026) · Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.joms.2026.07.004
Fu, Y., Deng, Q., Gao, T., Lyv, X., Zhou, F., Zhao, Z., Dai, J., Xu, H., & Tong, Z. (2026) · Psychiatry · doi:10.1080/00332747.2026.2696219
“However, they have been rarely studied in schizophrenia, and their predictive accuracy remains unclear (Chekroud et al., 2024).”
Wu, J., Zhang, X., Hu, S., Li, X., Yang, T., Yao, J., He, Y., Gao, J., Wang, Y., & Lei, Y. (2026) · Frontiers in Toxicology · doi:10.3389/ftox.2026.1829277
“We then analyzed the top 10 features, their dependencies, and individual patient risk profiles to show how features contribute to re-elevation risk in individual patients (Stiglic et al., 2020; Chekroud et al., 2024; Apley and Zhu, 2020).”
MUNTEANU-FORIJ, O., Poetar, C.-R., Zika, O., Predescu, E., Șipoș, R., Dobrean, A., & Månsson, K. N. T. (2026) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2026.112303
Collecting the Data in Psychological Sciences
Kovač, N., Blagojević, M., Watson, P., Farahani, H., Done, L.-C. W., & Ratković, K. (2026) · Springer eBooks · doi:10.1007/978-3-032-18312-5_4
Understanding Institutional Variability in ICU Data for Robust Machine Learning
Salinas, F., Kessler, S., Agristean, M., Abdulkader, M., Aubin, H., Lichtenberg, A., & Falko Schmid, A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-34459-5_22
Tochadse, M., Klawohn, J., Kaufmann, C., Grützmann, R., Riesel, A., Heinzel, S., Rane, R. P., da Silva, S. P., Wellan, S. A., Gijsen, S., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-64405-y
“Two recent treatment outcome prediction studies in mental disorders 54,55 examined generalization performance of their ML models on independent clinical cohorts.”
Andrés, M. S., Maharahan, V., Riverola, A. P., Rodriguez, L. G., Thuny, F., Ramalingam, S., Howe, M., Mamas, M. A., Banerjee, A., Manisty, C., et al. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag127
Kong, W., Tan, S. M. X., Lokadjaja, M. C., Kho, J. J., Yang, Z., Yee, J. Y., Lam, M., Lee, J., & Goh, W. W. B. (2026) · PLOS mental health. · doi:10.1371/journal.pmen.0000686
“However, deployment practicability is another matter, and requires careful evaluations on gen‑ eralizability, resource cost, and data availability [21,22].”
Towards precision psychiatry: initial foundations and future directions
Etkin, A., O’Donnell, P., Ressler, K. J., & Manji, H. K. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01509-0 · Cited by 3
“This raises important issues both around the need for prospective replication of any finding, given the high likelihood of false positives or over-fitting when using machine learning 9 , and around the clinical relevance of a correlation.”
Farhat, L. C., Sampaio, E. K., Nunes, D. D. A., Curiki, G. C., Taylor, J. H., Bloch, M. H., & Polanczyk, G. V. (2026) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2026.09.002 · Cited by 1
Weller, P. D., Hilbert, K., Alpers, G. W., Bartholdy, S., Berger, E., Brakemeier, E., Bräscher, A., Brockmeyer, T., Christiansen, H., Equit, M., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-70047-x
“Whilst we had hoped that the use of a more naturalistic and diverse dataset in the current study would lead to improved generalization in comparison to previous attempts 17 this did not materialize, suggesting that more factors play a role in model generalizability.”
The impact of deep learning and omics data in transforming precision therapy for brain cancer
Munquad, S., Upadhyay, A., & Das, A. B. (2026) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2026.1811813
“Model performance can be significantly impacted by variations in diagnostic tools, laboratory protocols, and standard treatment guidelines (Chekroud et al., 2024; Ly, 2024 (Raiaan et al., 2024).”
van den Eshof, J. W., Leferink, R. G., Van, H. L., van Grieken, R. A., Arntz, A. R., & Cremers, H. R. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2721657
Wiebel, C., Eijsbroek, V. C., Varadarajan, V., Kjell, K., Schwartz, H. A., & Kjell, O. (2026) · JMIR Mental Health · doi:10.2196/105460
“…on training data, they often fail to generalize to unseen data, which is a problem well-documented across predictive modeling in clinical science [44], suggesting that without rigorous validation procedures, such as testing preregistered models on held-out samples, reported accuracies may be…”
Lim, S. Y., Lee, Y., Kantabanlang, Y., & Jiang, Y. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106762
Ontology-Based Safety Management for Next-Generation Gaseous Fuel Infrastructure: A Review
Yoon, C., Lee, J., Min, M., & Jung, S. (2026) · Energies · doi:10.3390/en19204842
Wu, Q., Fan, X., & Qi, X. (2026) · Health Nexus Digital Health and Medical AI · doi:10.63802/hndh.v1.i1.358
Zhu, W., Zheng, X., Tian, D., Yan, L., Bai, Z., Wu, J., Hu, P., & Chen, R. (2026) · doi:10.2196/preprints.112699
Omigbodun, F. T. (2026) · Journal of Medicine Surgery and Public Health · doi:10.1016/j.glmedi.2026.100258
‘From exacerbation counts to calibrated risk: lessons from ACCEPT 3.0-UK for COPD prediction models’
Antón, A. J., Romero, P., & Navarrete, B. A. (2026) · Thorax · doi:10.1136/thorax-2026-225547
Karalunas, S. L., Faraone, S. V., Mooney, M. A., Weiskopf, N. G., Doyle, A. E., Kurtz‐Nelson, E. C., Keehn, R. M., Sibley, M. H., Wozniak, J. R., Zaback, T., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.10.006
Weyer, C., Diederichs, N., Meisenzahl, E., & (336400), N. K. (2026) · Der Nervenarzt · doi:10.1007/s00115-026-02023-9
Naqvi, A., & Naqvi, F. N. (2026) · JMIR mhealth and uhealth · doi:10.2196/103711
Metadata
- Authors
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M.
- Affiliations
- Yale University (US)
- New York City Council (US)
- University of Augsburg (DE)
- University of Cologne (DE)
- Ludwig-Maximilians-Universität München (DE)
- Yale New Haven Hospital (US)
- Laureate Institute for Brain Research (US)
- Corresponding author
- Chekroud, Adam M.
- ORCID
- Venue
- Science
- Published
- 2024-01-12
- Volume
- 383
- Issue
- 6679
- Pages
- 164-167
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4390734451
- Cited by
- 251
- References
- 37
- Citations by year
- 2026: 76 · 2025: 96 · 2024: 78
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, generalizability, out-of-sample forecasting, antipsychotic drugs, schizophrenia, machine learning, context sensitivity, treatment outcome prediction, clinical decision-making
- MeSH terms
- Machine Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Female, Humans, Male, Middle Aged, Prognosis, Schizophrenia, Antipsychotic Agents, Models, Statistical, Treatment Outcome, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.