The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
External validation of AI models in health should be replaced with recurring local validation
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One strategy to address this concern and limit inaccuracies is to evaluate AI within health care organizations using their patients’ data.4,20,21”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [20]
In the citing paper
“There are widespread concerns that the performance of these tools in early validation studies may not reflect their use in practice.19,20”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [20]
In the citing paper
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [103]
In the citing paper
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [49]
In the citing paper
“This phenomenon reflects that the training material may prime the dexterity of the resultant module49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [49]
Citations
Cites
Porter, M. E. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1011024 · Cited by 5,084
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 477
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294
Self-citationThere is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119
Self-citationThe epic sepsis model falls short-the importance of external validation
Habib, A. R., Lin, A. L., & Grant, R. W. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.3333 · Cited by 101
Singh, H., Mhasawade, V., & Chunara, R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000023 · Cited by 67
Granlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53
Cited by
Sorted by citations
FDA Perspective on the Regulation of Artificial Intelligence in Health Care and Biomedicine
Warraich, H. J., Tazbaz, T., & Califf, R. M. (2025) · JAMA · doi:10.1001/jama.2024.21451 · Cited by 332
Transforming Cardiovascular Care With Artificial Intelligence: From Discovery to Practice
Khera, R., Oikonomou, E. K., Nadkarni, G. N., Morley, J., Wiens, J., Butte, A. J., & Topol, E. J. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.05.003 · Cited by 324
AI, Health, and Health Care Today and Tomorrow: The JAMA Summit Report on Artificial Intelligence
Angus, D. C., Khera, R., Lieu, T., Liu, V., Ahmad, F. S., Anderson, B., Bhavani, S. V., Bindman, A., Brennan, T., Celi, L. A., et al. (2025) · JAMA · doi:10.1001/jama.2025.18490 · Cited by 195
Self-citationA Nationwide Network of Health AI Assurance Laboratories
Shah, N. H., Halamka, J. D., Saria, S., Pencina, M., Tazbaz, T., Tripathi, M., Callahan, A., Hildahl, H., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2023.26930 · Cited by 114
Self-citationVan Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Metadata
- Authors
- Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H.
- Affiliations
- University of Oxford (GB)
- Duke University (US)
- Stanford Medicine (US)
- Corresponding author
- Youssef, Alexey
- Venue
- Nature Medicine
- Published
- 2023-10-18
- Volume
- 29
- Issue
- 11
- Pages
- 2686-2687
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4387763575
- Cited by
- 170
- References
- 9
- Citations by year
- 2026: 61 · 2025: 61 · 2024: 44 · 2023: 4
- FWCI
- 6.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- external validation, local validation, artificial intelligence models, machine learning, medical device regulation, FDA review, model transparency, childhood cancer, hematopoietic stem cell transplantation, AI governance, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
