Illusory generalizability of clinical prediction models
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model’s success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, such models based on data from specific populations may not be generalizable and thus may fail to properly assign the optimal treatment [26].”
From Personalized machine learning guided intervention for optimizing lifestyle behaviors in depression: a pilot study (2026) · cites this paper as [26]
In the citing paper
“AI agents show significant potential in the realm of personalized treatment by leveraging a variety of patient-specific data, including genetic information, medical history, lifestyle factors, and treatment responses.77,88,89,90,91,92”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
In the citing paper
“AI-driven real-time patient monitoring systems are revolutionizing clinical care by enabling continuous, data-driven tracking of patient health.90,104,105”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [90]
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987) · Schizophrenia Bulletin · doi:10.1093/schbul/13.2.261 · Cited by 21,378
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
(1976) · Medical Entomology and Zoology · Cited by 8,899
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
The clinical global impressions scale: applying a research tool in clinical practice.
Busner, J., & Targum, S. D. (2007) · PubMed · Cited by 2,759
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Lack of group-to-individual generalizability is a threat to human subjects research
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711978115 · Cited by 1,099
Dose Equivalents for Antipsychotic Drugs: The DDD Method: Table 1.
Leucht, S., Samara, M., Heres, S., & Davis, J. M. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbv167 · Cited by 762
Cross-trial prediction of treatment outcome in depression: a machine learning approach
Chekroud, A. M., Zotti, R. J., Shehzad, Z., Gueorguieva, R., Johnson, M. K., Trivedi, M. H., Cannon, T. D., Krystal, J. H., & Corlett, P. R. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00471-x · Cited by 676
Self-citationLeucht, S., Leucht, C., Huhn, M., Chaimani, A., Mavridis, D., Helfer, B., Samara, M., Rabaioli, M., Bächer, S., Cipriani, A., et al. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2017.16121358 · Cited by 542
DeRubeis, R. J., Cohen, Z. D., Forand, N. R., Fournier, J. C., Gelfand, L. A., & Lorenzo-Luaces, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0083875 · Cited by 509
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 459
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
Chekroud, A. M., Gueorguieva, R., Krumholz, H. M., Trivedi, M. H., Krystal, J. H., & McCarthy, G. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0025 · Cited by 276
Self-citationKoutsouleris, N., Kahn, R. S., Chekroud, A. M., Leucht, S., Falkai, P., Wobrock, T., Derks, E. M., Fleischhacker, W. W., & Hasan, A. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(16)30171-7 · Cited by 217
Self-citationEnhancing the Personalization of Psychotherapy With Dynamic Assessment and Modeling
Fisher, A. J., & Boswell, J. F. (2016) · Assessment · doi:10.1177/1073191116638735 · Cited by 204
Remission in schizophrenia: validity, frequency, predictors, and patients' perspective 5 years later
Lambert, M., Karow, A., Leucht, S., Schimmelmann, B. G., & Naber, D. (2010) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2010.12.3/mlambert · Cited by 203
Leucht, S. (2014) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.13049su1c.02 · Cited by 174
Open trial of a personalized modular treatment for mood and anxiety
Fisher, A. J., Bosley, H. G., Fernandez, K. C., Reeves, J. W., Soyster, P. D., Diamond, A. E., & Barkin, J. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.01.010 · Cited by 156
Zhu, Y., Li, C., Huhn, M., Rothe, P., Krause, M., Bighelli, I., Schneider‐Thoma, J., & Leucht, S. (2017) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2017.06.011 · Cited by 147
Is the PANSS used correctly? a systematic review
Obermeier, M., Schennach-Wolff, R., Meyer, S., Möller, H.-J., Riedel, M., Krause, D., & Seemüller, F. (2011) · BMC Psychiatry · doi:10.1186/1471-244x-11-113 · Cited by 83
Data-Driven Taxonomy for Antipsychotic Medication: A New Classification System
McCutcheon, R. A., Harrison, P. J., Howes, O. D., McGuire, P. K., Taylor, D. M., & Pillinger, T. (2023) · Biological Psychiatry · doi:10.1016/j.biopsych.2023.04.004 · Cited by 76
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Qu, C., Omar, R. Z., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 56
Bondar, J., Babich Morrow, C., Gueorguieva, R., Brown, M., Hawrilenko, M., Krystal, J. H., Corlett, P. R., & Chekroud, A. M. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.16349 · Cited by 53
Self-citationChekroud, A. M., Foster, D., Zheutlin, A. B., Gerhard, D. M., Roy, B., Koutsouleris, N., Chandra, A., Esposti, M. D., Subramanyan, G., Gueorguieva, R., et al. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201800094 · Cited by 51
Self-citationTraining replicable predictors in multiple studies
Patil, P., & Parmigiani, G. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708283115 · Cited by 48
Evidence-Based Pragmatic Psychiatry—A Call to Action
Paulus, M. P. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.2439 · Cited by 43
Self-citationCourt hears how drug giant Merck tried to "neutralise" and "discredit" doctors critical of Vioxx
Moynihan, R. (2009) · BMJ · doi:10.1136/bmj.b1432 · Cited by 40
Applications of prediction models
Steyerberg, E. W. (2008) · Statistics in the health sciences · doi:10.1007/978-0-387-77244-8_2 · Cited by 39
Bigger Data, Harder Questions—Opportunities Throughout Mental Health Care
Chekroud, A. M. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.3333 · Cited by 29
Self-citationThe perilous path from publication to practice
Chekroud, A. M., & Koutsouleris, N. (2017) · Molecular Psychiatry · doi:10.1038/mp.2017.227 · Cited by 26
Self-citationGenerating Accurate Personalized Predictions of Future Behavior: A Smoking ExemplarPreprint
Fisher, A. J., & Soyster, P. D. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/e24v6 · Cited by 24
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
Research works printed but not linked: 3
- 10.1176/appi.ajp.2017.16121358 · DOI · Paper page
- 10.1093/schbul/sbv167 · DOI · Paper page
- 10.1001/jamapsychiatry.2017.0025 · DOI · Paper page
Cited by
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
“This reinforces the potential of roles such as that of digital navigators 11,13 in im proving engagement rates and therapeutic outcomes 198 .”
A review of model evaluation metrics for machine learning in genetics and genomics
Miller, C., Portlock, T., Nyaga, D. M., & O'Sullivan, J. M. (2024) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2024.1457619 · Cited by 199
Zantvoort, K., Nacke, B., Görlich, D., Hornstein, S., Jacobi, C., & Funk, B. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01360-w · Cited by 116
“Further, especially for complex features or ones with unknown or low information value, having a reasonablysizedtestsetisindispensable 18,46 .”
InfluentialFügener, A., Walzner, D. D., & Gupta, A. (2025) · Management Science · doi:10.1287/mnsc.2024.05684 · Cited by 161
Practical AI application in psychiatry: historical review and future directions
Sun, J., Lu, T., Shao, X., Han, Y., Xia, Y., Zheng, Y., Wang, Y., Li, X., Ravindran, A. V., Fan, L., et al. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03072-3 · Cited by 99
“For example, Adam et al. analyzed the performance of AI models across multiple independent clinical trials that involved antipsychotic medication for schizophrenia [58].”
Metadata
- Authors
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M.
- Affiliations
- Yale University (US)
- New York City Council (US)
- University of Augsburg (DE)
- University of Cologne (DE)
- Ludwig-Maximilians-Universität München (DE)
- Yale New Haven Hospital (US)
- Laureate Institute for Brain Research (US)
- Corresponding author
- Chekroud, Adam M.
- ORCID
- Venue
- Science
- Published
- 2024-01-12
- Volume
- 383
- Issue
- 6679
- Pages
- 164-167
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4390734451
- Cited by
- 251
- References
- 37
- Citations by year
- 2026: 76 · 2025: 96 · 2024: 78
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Psychiatry and Mental health › Schizophrenia research and treatment
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, generalizability, out-of-sample forecasting, antipsychotic drugs, schizophrenia, machine learning, context sensitivity, treatment outcome prediction, clinical decision-making
- MeSH terms
- Machine Learning, Adolescent, Adult, Aged, Aged, 80 and over, Child, Female, Humans, Male, Middle Aged, Prognosis, Schizophrenia, Antipsychotic Agents, Models, Statistical, Treatment Outcome, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.