The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Illusory generalizability of clinical prediction models
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model’s success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, such models based on data from specific populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
From Personalized machine learning guided intervention for optimizing lifestyle behaviors in depression: a pilot study (2026) · cites this paper as [26]
In the citing paper
“AI agents show significant potential in the realm of personalized treatment by leveraging a variety of patient-specific data, including genetic information, medical history, lifestyle factors, and treatment responses.77,88,89,90,91,92”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
In the citing paper
“AI-driven real-time patient monitoring systems are revolutionizing clinical care by enabling continuous, data-driven tracking of patient health.90,104,105”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987) · Schizophrenia Bulletin · doi:10.1093/schbul/13.2.261 · Cited by 21,378
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
(1976) · Medical Entomology and Zoology · Cited by 8,899
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
The clinical global impressions scale: applying a research tool in clinical practice.
Busner, J., & Targum, S. D. (2007) · PubMed · Cited by 2,760
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Research works printed but not linked: 3
- 10.1176/appi.ajp.2017.16121358 · DOI · Paper page
- 10.1093/schbul/sbv167 · DOI · Paper page
- 10.1001/jamapsychiatry.2017.0025 · DOI · Paper page
Cited by
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
“This reinforces the potential of roles such as that of digital navigators 11,13 in im proving engagement rates and therapeutic outcomes 198 .”
A review of model evaluation metrics for machine learning in genetics and genomics
Miller, C., Portlock, T., Nyaga, D. M., & O'Sullivan, J. M. (2024) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2024.1457619 · Cited by 199
Zantvoort, K., Nacke, B., Görlich, D., Hornstein, S., Jacobi, C., & Funk, B. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01360-w · Cited by 116
“Further, especially for complex features or ones with unknown or low information value, having a reasonablysizedtestsetisindispensable 18,46 .”
InfluentialFügener, A., Walzner, D. D., & Gupta, A. (2025) · Management Science · doi:10.1287/mnsc.2024.05684 · Cited by 160
Practical AI application in psychiatry: historical review and future directions
Sun, J., Lu, T., Shao, X., Han, Y., Xia, Y., Zheng, Y., Wang, Y., Li, X., Ravindran, A. V., Fan, L., et al. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03072-3 · Cited by 98
“For example, Adam et al. analyzed the performance of AI models across multiple independent clinical trials that involved antipsychotic medication for schizophrenia [58].”
Metadata
- Authors
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M.
- Affiliations
- Yale University (US)
- New York City Council (US)
- University of Augsburg (DE)
- University of Cologne (DE)
- Ludwig-Maximilians-Universität München (DE)
- Yale New Haven Hospital (US)
- Laureate Institute for Brain Research (US)
- Corresponding author
- Chekroud, Adam M.
- ORCID
- Venue
- Science
- Published
- 2024-01-12
- Volume
- 383
- Issue
- 6679
- Pages
- 164-167
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4390734451
- Cited by
- 251
- References
- 37
- Citations by year
- 2026: 76 · 2025: 96 · 2024: 78
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, generalizability, out-of-sample forecasting, antipsychotic drugs, schizophrenia, machine learning, context sensitivity, treatment outcome prediction, clinical decision-making
- MeSH terms
- Machine Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Female, Humans, Male, Middle Aged, Prognosis, Schizophrenia, Antipsychotic Agents, Models, Statistical, Treatment Outcome, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Illusory generalizability of clinical prediction models. https://app.cochat.ai/papers/10.1126/science.adg8538
- The paper
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M. (2024). Illusory generalizability of clinical prediction models. Science. https://doi.org/10.1126/science.adg8538
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.