The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
External validation of AI models in health should be replaced with recurring local validation
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One strategy to address this concern and limit inaccuracies is to evaluate AI within health care organizations using their patients’ data.4,20,21”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [20]
In the citing paper
“There are widespread concerns that the performance of these tools in early validation studies may not reflect their use in practice.19,20”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [20]
In the citing paper
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [103]
In the citing paper
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [49]
In the citing paper
“This phenomenon reflects that the training material may prime the dexterity of the resultant module49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [49]
Citations
Cites
Porter, M. E. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1011024 · Cited by 5,084
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 477
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294
Self-citationThere is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119
Self-citationThe epic sepsis model falls short-the importance of external validation
Habib, A. R., Lin, A. L., & Grant, R. W. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.3333 · Cited by 101
Singh, H., Mhasawade, V., & Chunara, R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000023 · Cited by 67
Granlund, T., Stirbu, V., & Mikkonen, T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00726-1 · Cited by 53
Cited by
Sorted by citations
Items 101–172 of 172
Altıntaş, Ö., & Deni̇zli̇, A. (2026) · Electronics · doi:10.3390/electronics15153388 · Cited by 2
Deep learning to predict fetal acidemia: a response
McCoy, J. A., & Cava, W. L. (2024) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2024.07.037 · Cited by 2
The potential risks of artificial intelligence in cardiovascular care
Göçer, H., & Durukan, A. B. (2025) · Turkish Journal of Thoracic and Cardiovascular Surgery · doi:10.5606/tgkdc.dergisi.2025.26912 · Cited by 2
Role and Use of Race in Artificial Intelligence and Machine Learning Models Related to Health
Were, M. C., Li, A., Malin, B., Yin, Z., Coco, J., Collins, B. X., Clayton, E. W., Novak, L. L., Hendricks‐Sturrup, R., Oluyomi, A., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73996 · Cited by 4
AI ethics for the everyday intensivist
Tan, S. C., Modra, L. J., & Hensman, T. (2025) · Critical Care and Resuscitation · doi:10.1016/j.ccrj.2025.100115 · Cited by 2
Jagadeesh, A. T., Aramrat, C., Rai, S., Maqsood, F. H., Madhukeshwar, A. K., Bhogadi, S., Lieber, J., Mahajan, H., Banjara, S. K., Lewin, A., et al. (2025) · The Lancet Regional Health - Southeast Asia · doi:10.1016/j.lansea.2025.100644 · Cited by 2
A Distinct Approach to Clinical GenAI OversightPreprint
Thiers, F. A., & Lucy, K. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/vm6zy · Cited by 2
Singh, J., Alsaidan, O. A., Aodah, A. H., Alrobaian, M., Almalki, W. H., Almujri, S. S., Sahoo, A., Alam, K., Lal, J. A., Barkat, M. A., et al. (2025) · Future Oncology · doi:10.1080/14796694.2025.2606642 · Cited by 9
Chen, Z., Li, W., & Lin, Z. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001283 · Cited by 5
Moldakhmetova, S., Bimurat, B., Berdaly, A., Makhammajanov, Z., Salykov, A., Bukasov, R., & Gaipov, A. (2026) · International Journal of Molecular Sciences · doi:10.3390/ijms27083648 · Cited by 1
Automated Triage for New Keratoconus Referrals Using Multimodal Deep Learning
Balal, S., Kandakji, L., Leucci, M. T., Keane, P. A., Gore, D. M., Pontikos, N., & Allan, B. D. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101208 · Cited by 1
Vijayasimha, M., Dubey, A. K., Yaqoob, S. I., & Charan, L. S. (2026) · Forensic Science Medicine and Pathology · doi:10.1007/s12024-026-01256-z · Cited by 1
Asmara, O. D., Steenhuis, E. G. M., de Jong, K., Joseph, A., Timmer, M., Tenda, E. D., Boerma, E. C., Heuvelmans, M. A., & van Geffen, W. H. (2026) · Radiology Artificial Intelligence · doi:10.1148/ryai.250331 · Cited by 3
Integration of artificial intelligence applications in clinical pharmacy services: A scoping review
Al‐Taie, A., Abdullah, A. S., & Arueyingho, O. (2026) · Future Healthcare Journal · doi:10.1016/j.fhj.2026.100547 · Cited by 1
Can Machine Learning Personalize Cardiovascular Therapy in Sepsis?
Catling, F. J. R., Nagendran, M., Festor, P., Bien, Z., Harris, S. K., Faisal, A. A., Gordon, A. C., & Komorowski, M. (2024) · Critical Care Explorations · doi:10.1097/cce.0000000000001087 · Cited by 1
Lumumba, V. W., Muriithi, D. K., Njoroge, E. W., Langat, A. K., Mwebesa, E., & Wanyama, M. (2025) · Biomedical Statistics and Informatics · doi:10.11648/j.bsi.20251003.12 · Cited by 1
Health AI Assurance Laboratories—Reply
Shah, N. H., Halamka, J. D., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2024.1087 · Cited by 1
Self-citationHazard-aware adaptations bridge the generalization gap in large language models: a nationwide studyPreprint
Wu, J. T.-Y., Conover, S., Su, C. C., Corrigan, J., Culnan, J. M., Liu, Y., Kelley, M., Do, N., Arya, S., Harris, A. H. S., et al. (2025) · medRxiv · doi:10.1101/2025.02.14.25322312 · Cited by 1
Ferreira, F. S., Lannou, E. L., Post, B., Haar, S., Kadirvelu, B., Brett, S. J., & Faisal, A. A. (2024) · medRxiv · doi:10.1101/2024.05.28.24308024
Katz, S., Suijker, J., Skrede, S., Vries, A. M., Pijpe, A., Norrby‐Teglund, A., Medina, L. M. P., Damås, J. K., Hyldegaard, O., Solligård, E., et al. (2024) · medRxiv · doi:10.1101/2024.06.05.24308478
Jagadeesh, A., Aramrat, C., Rai, S., Maqsood, F. H., Madhukeshwar, A. K., Bhogadi, S., Lieber, J., Mahajan, H., Banjara, S. K., Lewin, A., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5153451
Lee, M. C. M., Farahvash, A., & Zezos, P. (2025) · Inflammatory Bowel Diseases · doi:10.1093/ibd/izaf096
Generalizability of Risk Models for Treatment-Resistant Depression Across Three Health SystemsPreprint
Walsh, C. G., Ripperger, M. A., McCoy, T. H., Castro, V. M., Hu, Y., Kirchner, H. L., Ruderfer, D. M., & Perlis, R. H. (2025) · medRxiv · doi:10.1101/2025.05.21.25328089
Artificial intelligence in thoracic pathology: diagnostic and predictive applications
von der Thüsen, J. H. (2025) · Diagnostic histopathology · doi:10.1016/j.mpdhp.2025.06.008
Portability of an artificial intelligence model for self-harm detection across hospital settingsPreprint
Rozova, V. S., Witt, K., Conway, M., Robinson, J., & Verspoor, K. M. (2025) · medRxiv · doi:10.1101/2025.07.10.25331160
Lefebvre, O., Lemyre, F. C., Éthier, J., Chikouche, L. H., Amriou, L., Poenaru, D., & Vallières, M. (2025) · medRxiv · doi:10.1101/2025.08.22.25334254
Wang, X., Jahagirdar, S., Kemp, B., Gross, J. J., Bruckmaier, R. M., Saccenti, E., & van Knegsel, A. T. M. (2025) · Metabolites · doi:10.3390/metabo15110698
Yan, A. P., Guo, L., Patel, P., Schechter, T., Arciniegas, S. E., Inoue, J., Vettese, E., Jessa, K., Cardiff, B., TOMLINSON, G. A., et al. (2025) · BMC Cancer · doi:10.1186/s12885-025-15137-1
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Learning Patient Similarity from Genomics for Precision OncologyPreprint
Shady, M., Reardon, B. M., Jiang, S., Pimenta, E. M., O’Meara, T. A., Park, J., Kehl, K. L., Elmarakeby, H., Sunyaev, S. R., & Allen, E. M. V. (2025) · medRxiv · doi:10.64898/2025.12.17.25342480
Li, D., Wang, H., Gao, F., Fu, X., & Han, J. (2025) · Hepatoma Research · doi:10.20517/2394-5079.2025.67
Faviez, C., Chen, X., Garcelon, N., Zaidan, M., Billot, K., Petzold, F., Faour, H., Douillet, M., Rozet, J., Cormier‐Daire, V., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25902422
Faviez, C., Chen, X., Garcelon, N., Zaidan, M., Billot, K., Petzold, F., Faour, H., Douillet, M., Rozet, J., Cormier‐Daire, V., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25902330
Katz, S., Suijker, J., Skrede, S., Vries, A. M., Pijpe, A., Norrby‐Teglund, A., Medina, L. M. P., Damås, J. K., Hyldegaard, O., Solligård, E., et al. (2026) · BMC Medicine · doi:10.1186/s12916-025-04593-y
Artificial Intelligence to Propel Evidence Generation
Dhruva, S. S., Curtis, L. H., & Califf, R. M. (2026) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2025.11.011
Feng, R., Richter, F., Mari, E., Gleason, A., Le, C., Kellner, C. P., Shrivastava, R. K., Fields, M. C., Rapoport, B. I., Bederson, J. B., et al. (2026) · Neurosurgery · doi:10.1227/neu.0000000000003899
Abubakar, A. M., Ferhoune, I., Ndaba, S., Bilal, M. T., & Abdul, A. Z. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-36412-9.00010-8
Pang, C., Jiayong, D., Jiang, D., Wang, Y., Li, N., & Ren, D. (2026) · Scientific Reports · doi:10.1038/s41598-026-37913-0
Michopoulou, S. K., Prosser, A. M. J., O’Brien, N., Dickson, J. C., Guy, M., Teeling, J. L., & Kipps, C. (2026) · Journal of Alzheimer s Disease · doi:10.1177/13872877251413790
Deep learning for estimating right ventricular function from routine coronary angiography
Rostami, B., Bhattarai, P., Al-Abcha, A. K., Lee, E., Chang, S.-S., Sarraf, M., Attia, Z. I., Fetterly, K. A., Thaden, J. J., Friedman, P. A., et al. (2026) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztag004
Karthikeyan, K., Flythe, J. E., Pun, P. H., Winkelmayer, W. C., & Carlson, D. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf152
Lefebvre, O., Lemyre, F. C., Éthier, J., Chikouche, L. H., Amriou, L., Poenaru, D., & Vallières, M. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag034 · Cited by 1
Management and Prediction of Acute Pancreatitis Severity Using AI: A Surgical Perspective
Dumitrascu, I., Zărnescu, N. O., Marchegiani, G., Ilie, A., Zarnescu, E. C., & Costea, R. (2026) · Diagnostics · doi:10.3390/diagnostics16091350
Zhang, T., Huo, X., Chen, J., & Zhao, X. (2026) · BMC Medicine · doi:10.1186/s12916-026-04955-0
Dor, B. B., & Sonis, S. T. (2026) · Head and Neck Pathology · doi:10.1007/s12105-026-01919-x
MHub.ai: A Standardized Platform for Reproducible AI Research in Medical ImagingPreprint
Aerts, H. J. W. L., Nürnberg, L., Bontempi, D., Pai, S., Lisle, C. R., Pieper, S., Kikinis, R., van de Leemput, S. C., Soni, R., Murugesan, G., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9480032/v1
Quennelle, S., Malekzadeh-Milani, S. G., Garcelon, N., Burgun, A., Bonnet, D., & Neuraz, A. (2026) · BMC Cardiovascular Disorders · doi:10.1186/s12872-026-06065-4
Toward Effective Human Oversight in a Human-in-the-Loop AI System: Multi-Model AI and Model Labeling
Chang, J., Oh, Z. X., & Syafiie, S. (2026) · IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS) · doi:10.1109/i2cacis69435.2026.11600358
Abdulkerimova, S. F., & Danilova, Y. A. (2026) · Real-World Data & Evidence · doi:10.37489/2782-3784-myrwd-107
Cross-database validation reveals distinct layers of transportability in ICU delirium predictionPreprint
Ni, S., & Sato, K. (2026) · medRxiv · doi:10.64898/2026.07.19.26358409
Li, X., Alexandrov, N., Veltman, Y., van Walsum, T., Bos, D., Majoie, C. B. L. M., van Zwam, W. H., Dippel, D. W. J., Roozenbeek, B., Lingsma, H. F., et al. (2026) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2026-025297
Systematic assessment of mixed imputation methods and explainable machine learning
Li, J.-Y., & Zhao, Y. (2026) · World Journal of Gastrointestinal Surgery · doi:10.4240/wjgs.v18.i7.119087
Pan, L., Zhao, G., Yu, Y., & Qin, G. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01285-x
Champeaux, S., Booth, J., Brown, A. P. Y., Sebire, N. J., Drobnjak, I., & Bowyer, S. A. (2026) · medRxiv · doi:10.64898/2026.08.18.26360666
Moradi, Z., Kiani, I., Rahmani, S. A. M. S., Mirzai, M., Elhaie, M., Dahaj, M. E., Rahimi, M., Mosadegh, M., Mohammadzadeh, S., & Gity, M. (2026) · European Journal of Radiology Open · doi:10.1016/j.ejro.2026.100808
Wang, M., Li, J., Cheng, A. S. K., Lo, H. K. Y., & Ng, P. H. F. (2026) · doi:10.2196/preprints.108436
Huang, J., Ong, W., Zheng, A. L. W., Tan, T. S. E., Zheng, G. T. C., Teo, E. C., Tan, J. H., Kumar, N., & Hallinan, J. T. P. D. (2026) · Cancers · doi:10.3390/cancers18172794
van den Bergh, F. J. P. N., van Rietbergen, B., van den Bergh, J. P., Biver, E., Wyers, C. E., & Bevers, M. S. A. M. (2026) · Bone · doi:10.1016/j.bone.2026.118076
Tian, Y., Tian, X., Li, Z., Wang, Z., Guo, H., Meng, F., & Wang, Z. (2026) · Digital Health · doi:10.1177/20552076261487029
Zhang, W., Li, X., Yuan, R., Zhang, Y., Yang, Y., Xu, Y., Zhong, R., Chen, S., Liu, D., Gao, L., et al. (2026) · Digital Health · doi:10.1177/20552076261487079
Huo, B., Chartash, D., Tsang, L., Collins, P. G. S., Cacciamani, G. E., Ruurda, J. P. H., Mayol, J. M., Hinchliffe, R. J., & Guyatt, G. (2026) · Nature Medicine · doi:10.1038/s41591-026-04647-5
Alhalawany, R. M., Alnufaie, R. F., & Khatatbeh, Y. M. (2026) · Healthcare · doi:10.3390/healthcare14182924
Temporal and cross-site validation of an AI system for self-harm detection
Rozova, V. S., Chen, L., Witt, K., Conway, D. M., Robinson, J., & Verspoor, K. M. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001667
Responsible Implementation of EHR-Based Clinical Prediction Models—Beyond Performance
Zhao, S. J., & Deardorff, W. J. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.32906
Nouman, M., & Haraguchi, R. (2026) · BioMedInformatics · doi:10.3390/biomedinformatics6050074
In silico biomedicine in the era of artificial intelligence: opportunities and challenges
Mustaffa, F.-E., Sajid, T., Ali, M., & Ashraf, N. M. (2026) · In Silico Research in Biomedicine · doi:10.1016/j.insi.2026.100629
Teti, F. S. L., Keke, K. J.-B., Yao, K. S., Dongo, K. Y. S., Fionko, Y. B., & Haïdara, A. (2026) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v5i9.1664
AI Prognosis in Healthcare, Bias, Misprediction, and Governance Gaps
Düffelmeyer, M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0403-6.ch008
Cappellani, F., Zeppieri, M., Antonio, L., Mazzotta, C. G., Gattazzo, I., D’Esposito, F., Gagliano, C., & Capobianco, M. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15207785
Cross-Dataset Wearable Sleep Staging under Source-Only and Target-Normalized Conditions
Fei, Y., Shen, C., & Xiao, F. (2026) · International Conference on Computer Network, Electronic and Automation (ICCNEA) · doi:10.1109/iccnea70678.2026.11722303
Rural Hospital AI Use and Evaluation
Cruikshank, J. A., Whitacre, B. E., & Davis, A. F. (2026) · The Journal of Rural Health · doi:10.1111/jrh.70237
AI algorithms and external validation: promises, pitfalls, and a practical blueprint
Thambiraj, G., & Armoundas, A. A. (2026) · European Heart Journal · doi:10.1093/eurheartj/ehag816
Metadata
- Authors
- Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H.
- Affiliations
- University of Oxford (GB)
- Duke University (US)
- Stanford Medicine (US)
- Corresponding author
- Youssef, Alexey
- Venue
- Nature Medicine
- Published
- 2023-10-18
- Volume
- 29
- Issue
- 11
- Pages
- 2686-2687
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4387763575
- Cited by
- 170
- References
- 9
- Citations by year
- 2026: 61 · 2025: 61 · 2024: 44 · 2023: 4
- FWCI
- 6.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- external validation, local validation, artificial intelligence models, machine learning, medical device regulation, FDA review, model transparency, childhood cancer, hematopoietic stem cell transplantation, AI governance, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review