Cited by 286Statistical Methods in EpidemiologyArtificial Intelligence in HealthcareMeta-analysis and systematic reviews

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

There is no such thing as a validated prediction model

Open access

Van Calster, Ben, Steyerberg, Ewout W., Wynants, Laure, van Smeden, Maarten·2023·BMC Medicine

DOI 10.1186/s12916-023-02779-wPMC PMC9951847

Open this paper in CoChat — free account Open PDF

Abstract

BACKGROUND: Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? MAIN BODY: We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. CONCLUSION: Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1186/s12916-023-02779-w

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Fig. 1 Distribution of patient age in the 9 largest centers from the ovarian cancer study. Histograms, density estimates, and mean (standard deviation) are given per centerCC BY · PubMed Central

Fig. 2 Distribution of maximum lesion diameter in the 9 largest centers from the ovarian cancer study. Histograms, density estimates, and median (interquartile range) are given per centerCC BY · PubMed Central

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citation
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Derivation of a Simple Clinical Model to Categorize Patients Probability of Pulmonary Embolism: Increasing the Models Utility with the SimpliRED D-dimer
Anderson, D., Rodger, M., Ginsberg, J., Kearon, C., Gent, M., Turpie, A., Bormanis, J., Weitz, J., Chamberlain, M., Bowie, D., et al. (2000) · Thrombosis and Haemostasis · doi:10.1055/s-0037-1613830 · Cited by 1,651
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Self-citation
Understanding receiver operating characteristic (ROC) curves
Fan, J., Upadhye, S., & Worster, A. (2006) · Canadian Journal of Emergency Medicine · doi:10.1017/s1481803500013336 · Cited by 843
Sample size considerations for the external validation of a multivariable prognostic model: A resampling study
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665

Show all 56 references

Sorted by citations

TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
Method
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods
(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
External validation of AI models in health should be replaced with recurring local validation
Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H. (2023) · Nature Medicine · doi:10.1038/s41591-023-02540-z · Cited by 170
APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support
Kwong, J. C. C., Khondker, A., Lajkosz, K., McDermott, M. B. A., Borrat, X., McCradden, M. D., Mamdani, M., Kulkarni, G. S., & Johnson, A. E. W. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.35377 · Cited by 159
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149
Generalizability assessment of AI models across hospitals in a low-middle and high income country
Yang, J., Dung, N. T., Thạch, P. N., Phong, N. T., Phu, V. D., Dong, P. K., Yen, L. M., Thy, D. B. X., Soltan, A. A. S., Thwaites, L., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-52618-6 · Cited by 120
Current Use And Evaluation Of Artificial Intelligence And Predictive Models In US Hospitals: Article examines uses and evaluation of artificial intelligence and predictive models in US hospitals
Nong, P., Adler-Milstein, J., Apathy, N. C., Holmgren, A. J., & Everson, J. (2025) · Health Affairs · doi:10.1377/hlthaff.2024.00842 · Cited by 104
Implementing precision methods in personalizing psychological therapies: Barriers and possible ways forward
Deisenhofer, A., Barkham, M., Beierl, E. T., Schwartz, B., Doorn, K. A., Beevers, C. G., Berwian, I. M., Blackwell, S. E., Bockting, C., Brakemeier, E., et al. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104443 · Cited by 123
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Perspectives on validation of clinical predictive algorithms
de Hond, A., Shah, V. B., Kant, I. M. J., CALSTER, B. V., Steyerberg, E. W., & Hernandez‐Boussard, T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00832-9 · Cited by 100
Five critical quality criteria for artificial intelligence-based prediction models
van Royen, F. S.-A., Asselbergs, F. W., Alfonso, F., Vardas, P. E., & van Smeden, M. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad727 · Cited by 77
Machine Learning Models for Blood Glucose Level Prediction in Patients With Diabetes Mellitus: Systematic Review and Network Meta-Analysis
Liu, K., Li, L., Ma, Y., Jiang, J., Liu, Z., Ye, Z., Liu, S., Pu, C., Chen, C., & Wan, Y. (2023) · JMIR Medical Informatics · doi:10.2196/47833 · Cited by 77
External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis
Rockenschaub, P., Akay, E. M., Carlisle, B. G., Hilbert, A., Wendland, J., Meyer-Eschenbach, F., Näher, A.-F., Frey, D., & Madai, V. I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02830-7 · Cited by 58
Don't be misled: 3 misconceptions about external validation of clinical prediction models
la Roi-Teeuw, H. M., van Royen, F. S.-A., de Hond, A., Zahra, A., de Vries, S., Bartels, R., Carriero, A., van Doorn, S., Dunias, Z. S., Kant, I. M. J., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111387 · Cited by 57
Artificial intelligence–based image analysis in clinical testing: lessons from cervical cancer screening
Egemen, D., Perkins, R. B., Cheung, L. C., Befano, B., Rodríguez, A. C., Desai, K. T., Lemay, A., Ahmed, S. R., Antani, S., Jerónimo, J., et al. (2023) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djad202 · Cited by 55
Deep Learning in Glaucoma Detection and Progression Prediction: A Systematic Review and Meta-Analysis
Ling, X. C., Chen, H. S.-L., Yeh, P., Cheng, Y.-C., Huang, C.-Y., Shen, S., & Lee, Y. (2025) · Biomedicines · doi:10.3390/biomedicines13020420 · Cited by 65
Sustainable deployment of clinical prediction tools—a 360° approach to model maintenance
Davis, S. E., Embí, P. J., & Matheny, M. E. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae036 · Cited by 46
Externally validated prediction models for pre‐eclampsia: systematic review and meta‐analysis
Tiruneh, S. A., Vu, T. T. T., Moran, L., Callander, E. J., Allotey, J., Thangaratinam, S., Rolnik, D. L., Teede, H. J., Wang, R., & Enticott, J. (2023) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27490 · Cited by 51
Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings
Vaid, A., Sawant, A. S., Suárez‐Fariñas, M., Lee, J., Kaul, S. K., Kovatch, P., Freeman, R., Jiang, J. J., Jayaraman, P., Fayad, Z. A., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m23-0949 · Cited by 50
ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42
AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges
Okada, Y., Mertens, M., Liu, N., Lam, S. S. W., & Ong, M. E. H. (2023) · Resuscitation Plus · doi:10.1016/j.resplu.2023.100435 · Cited by 42
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 47
Why do probabilistic clinical models fail to transport between sites
Lasko, T. A., Strobl, E. V., & Stead, W. W. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01037-4 · Cited by 41
Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals
Everson, J., Nong, P., & Richwine, C. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49463 · Cited by 33
Prognosticating the outcome of intensive care in older patients—a narrative review
Beil, M., Moreno, R. P., Fronczek, J., Kogan, Y., Moreno, R. P., Flaatten, H. K., Guidet, B., de Lange, D. W., Leaver, S. K., Nachshon, A., et al. (2024) · Annals of Intensive Care · doi:10.1186/s13613-024-01330-1 · Cited by 29
Deep learning for electrocardiogram interpretation: Bench to bedside
Schots, B. B. S., Pizarro, C. S., Arends, B. K. O., Oerlemans, M. I. F. J., Ahmetagić, D., van der Harst, P., & van Es, R. (2025) · European Journal of Clinical Investigation · doi:10.1111/eci.70002 · Cited by 30
Clinical data mining: challenges, opportunities, and recommendations for translational applications
Qiao, H., Chen, Y., Qian, C., & Guo, Y. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05005-0 · Cited by 32
Pragmatic Approaches to the Evaluation and Monitoring of Artificial Intelligence in Health Care: A Science Advisory From the American Heart Association
Jain, S. S., Goto, S., Hall, J. L., Khan, S. S., MacRae, C. A., Ofori, C., Pegus, C., Pencina, M., Peterson, E. D., Schwamm, L. H., et al. (2025) · Circulation · doi:10.1161/cir.0000000000001400 · Cited by 23
External validation of models for predicting cumulative live birth over multiple complete cycles of IVF treatment
Ratna, M. B., Bhattacharya, S., & McLernon, D. J. (2023) · Human Reproduction · doi:10.1093/humrep/dead165 · Cited by 23
A methodological systematic review of validation and performance of sepsis real-time prediction models
Wang, Z., Wang, W., Sun, C., Li, J., Xie, S., Xu, J., Zou, K., Jin, Y., Yan, S., Liao, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01587-1 · Cited by 31
Laboratory Data as a Potential Source of Bias in Healthcare Artificial Intelligence and Machine Learning Models
Luu, H. S. (2024) · Annals of Laboratory Medicine · doi:10.3343/alm.2024.0323 · Cited by 27
Prediction of Morbidity and Mortality After Esophagectomy: A Systematic Review
Amerongen, M. P. V. N., de Grooth, H., Veerman, G., Ziesemer, K. A., Henegouwen, M. I. V. B., & Tuinman, P. R. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-024-14997-4 · Cited by 22
Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model
Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.0181 · Cited by 21
Leveraging artificial intelligence for perioperative cancer risk assessment of oral potentially malignant disorders
Adeoye, J., & Su, Y. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000979 · Cited by 21
Predictive models of Alzheimer’s disease dementia risk in older adults with mild cognitive impairment: a systematic review and critical appraisal
Wang, X., Zhou, S., Ye, N., Li, Y., Zhou, P., Gao, C., & Hu, H. Z. (2024) · BMC Geriatrics · doi:10.1186/s12877-024-05044-8 · Cited by 22
Updating methods for artificial intelligence–based clinical prediction models: a scoping review
Meijerink, L., Dunias, Z. S., Leeuwenberg, A. M., de Hond, A., Jenkins, D. A., Martin, G. P., Sperrin, M., Peek, N. B., Spijker, R., Hooft, L., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111636 · Cited by 19
Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals
Cabanillas Silva, P., Sun, H., Rezk, M., Roccaro-Waldmeyer, D. M., Fliegenschmidt, J., Hulde, N., von Dossow, V., Meesseman, L., Depraetere, K., Stieg, J., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51409 · Cited by 20
A multimodal deep learning model for cervical pre-cancers and cancers prediction: Development and internal validation study
Madathil, S. A., Dhouib, M., Lelong, Q., Bourassine, A., & Monsonégo, J. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109710 · Cited by 20
Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities
González, R., Saha, A., Campbell, C. J. V., Nejat, P., Lokker, C., & Norgan, A. P. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100347 · Cited by 24
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19
The Role of Artificial Intelligence in Chronic Liver Diseases and Liver Transplantation
Spann, A., Strauss, A. T., Davis, S. E., & Bhat, M. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.05.012 · Cited by 21
Detecting changes in the performance of a clinical machine learning tool over time
Schinkel, M., Boerman, A. W., Paranjape, K., Wiersinga, W. J., & Nanayakkara, P. W. B. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104823 · Cited by 19
Performance of externally validated machine learning models based on histopathology images for the diagnosis, classification, prognosis, or treatment outcome prediction in female breast cancer: A systematic review
Gonzalez, R., Nejat, P., Saha, A., Campbell, C. J. V., Norgan, A. P., & Lokker, C. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100348 · Cited by 18
Artificial intelligence to unlock real‐world evidence in clinical oncology: A primer on recent advances
Bryant, A. K., Zamora‐Resendiz, R., Dai, X., Morrow, D., Lin, Y., Jungles, K. M., Rae, J. M., Tate, A., Pearson, A. N., Jiang, R., et al. (2024) · Cancer Medicine · doi:10.1002/cam4.7253 · Cited by 20
Evaluating dental AI research papers: Key considerations for editors and reviewers
Uribe, S. E., Hamdan, M. H., Valente, N. A., Yamaguchi, S., Umer, F., Tichý, A., Pauwels, R., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105867 · Cited by 16
In humble defense of unexplainable black box prediction models in healthcare
van Royen, F. S.-A., Weerts, H., de Hond, A., Geersing, G., Rutten, F. H., Moons, K. G. M., & van Smeden, M. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112013 · Cited by 18
Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas
Suh, Y. J., Han, K., Kwon, Y., Kim, H., Lee, S., Hwang, S. H., Kim, M. H., Shin, H. J., Lee, C. Y., & Shim, H. S. (2024) · Yonsei Medical Journal · doi:10.3349/ymj.2023.0368 · Cited by 15
External validation of SCORE2-Diabetes in The Netherlands across various socioeconomic levels in native-Dutch and non-Dutch populations
Alfaraj, S., Kist, J. M., Groenwold, R. H. H., Spruit, M. R., Mook-Kanamori, D. O., & Vos, R. C. (2024) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwae354 · Cited by 15
The validation of prediction models deserves more recognition
Mansmann, U., & Ön, B. I. (2025) · BMC Medicine · doi:10.1186/s12916-025-03994-3 · Cited by 16
Updating M6 pregnancy of unknown location risk‐prediction model including evaluation of clinical factors
Kyriacou, C., Ledger, A., Bobdiwala, S., Ayim, F., Kirk, E., Abughazza, O., Guha, S., Vathanan, V., Gould, D., Timmerman, D., et al. (2023) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.27515 · Cited by 15
Clinical prediction models to diagnose neonatal sepsis in low-income and middle-income countries: a scoping review
Neal, S. R., Sturrock, S., Musorowegomo, D., Gannon, H., Zaman, M., Cortina‐Borja, M., Doaré, K. L., Heys, M., Chimhini, G., & Fitzgerald, F. (2025) · BMJ Global Health · doi:10.1136/bmjgh-2024-017582 · Cited by 14
Predicting blood transfusion following traumatic injury using machine learning models: A systematic review and narrative synthesis
Oakley, W., Tandle, S., Perkins, Z., & Marsden, M. E. R. (2024) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000004385 · Cited by 15
Development and validation of artificial intelligence models for early detection of postoperative infections (PERISCOPE): a multicentre study using electronic health record data
van der Meijden, S. L., van Boekel, A. M., Schinkelshoek, L., van Goor, H., Steyerberg, E. W., Nelissen, R. G. H. H., Mesotten, D., Geerts, B. F., de Boer, M. G. J., Arbous, M. S., et al. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101163 · Cited by 13
The diagnostic and prognostic capability of artificial intelligence in spinal cord injury: A systematic review
Gill, S. S., Ponniah, H. S., Giersztein, S., Anantharaj, R. M., Namireddy, S. R., Killilea, J., Ramsay, D. S. C., Salih, A. M., Thavarajasingam, A., Scurtu, D., et al. (2025) · Brain and Spine · doi:10.1016/j.bas.2025.104208 · Cited by 18
Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery
Landolfo, C., Ceusters, J., Valentin, L., Froyman, W., Van Gorp, T., Heremans, R., Baert, T., Wouters, R., Vankerckhoven, A., Van Rompuy, A.-S., et al. (2024) · British Journal of Cancer · doi:10.1038/s41416-024-02578-x · Cited by 14
External validation of 12 existing survival prediction models for patients with spinal metastases
Bindels, B. J. J., Kuijten, R. H., Groot, O. Q., Huele, E. H., Gal, R., de Groot, M., van der Velden, J. M., Delawi, D., Schwab, J. H., Verkooijen, H. M., et al. (2025) · The Spine Journal · doi:10.1016/j.spinee.2025.01.014 · Cited by 12
Good practices and common pitfalls of machine learning in nutrition research
Kirk, D. (2024) · Proceedings of The Nutrition Society · doi:10.1017/s0029665124007638 · Cited by 11
Personalized surveillance in colorectal cancer: Integrating circulating tumor DNA and artificial intelligence into post-treatment follow-up
Negoi, I. (2025) · World Journal of Gastroenterology · doi:10.3748/wjg.v31.i18.106670 · Cited by 11
Contesting the conventional wisdom of periodontal risk assessment
Raittio, E. J., López, R., & Bælum, V. (2024) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12942 · Cited by 11
Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage
de Winkel, J., Maas, C. C. H. M., Roozenbeek, B., Klaveren, D. V., & Lingsma, H. F. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02280-9 · Cited by 10
Diagnostic framework to validate clinical machine learning models locally on temporally stamped data
Schuessler, M., Fleming, S. L., Meyer, S., Seto, T., & Hernandez‐Boussard, T. (2025) · Communications Medicine · doi:10.1038/s43856-025-00965-w · Cited by 13
The Enemies of Reliable and Useful Clinical Prediction Models: A Review of Statistical and Scientific Challenges
CALSTER, B. V., van Smeden, M., van Amsterdam, W. A. C., Coemans, M., Wynants, L., & Steyerberg, E. W. (2025) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042324-123749 · Cited by 14
Radiomics and artificial intelligence for risk stratification of pulmonary nodules: Ready for primetime?
Kim, R. Y. (2024) · Cancer Biomarkers · doi:10.3233/cbm-230360 · Cited by 12
Arnicolide D: a multi-targeted anticancer sesquiterpene lactone—preclinical efficacy and mechanistic insights
Shetty, M. S., Peña‐Corona, S. I., Borbolla‐Jiménez, F. V., Kaverikana, R., Shetty, S., Shet, V. B., Almarhoon, Z. M., Călina, D. C., Leyva‐Gómez, G., & Sharifi‐Rad, J. (2024) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-024-03095-7 · Cited by 9
Comprehensive Symptom Prediction in Inpatients With Acute Psychiatric Disorders Using Wearable-Based Deep Learning Models: Development and Validation Study
Hong, M., Kang, R.-R., Yang, J. H., Rhee, S. J., Lee, H., Kim, Y.-G., Lee, K. Y., Kim, H., Lee, Y. S., Youn, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/65994 · Cited by 9
Soluble Urokinase-Type Plasminogen Activator Receptor and Inflammatory Biomarker Response with Prognostic Significance after Acute Neuronal Injury – a Prospective Cohort Study
Sajanti, A., Hellström, S., Bennett, C., Srinath, A., Jhaveri, A., Cao, Y. J., Takala, R. S. K., Frantzén, J., Koskimäki, F., Falter, J., et al. (2024) · Inflammation · doi:10.1007/s10753-024-02185-1 · Cited by 9
The urgency of centering safety-net organizations in AI governance
Nong, P., Maurer, E. W., & Dwivedi, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01479-4 · Cited by 11
Feedback loops in intensive care unit prognostic models: an under-recognised threat to clinical validity
Balcarcel, D. R., Mehta, S. D., Dixon, C. G., Woods-Hill, C. Z., Goligher, E. C., van Amsterdam, W. A. C., & Yehya, N. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100880 · Cited by 9
Identifying older adults at increased risk of medication-related readmission to hospital within 30 days of discharge: development and validation of a risk assessment tool
Glans, M., Kempen, T. G. H., Jakobsson, U., Ekstam, A. K., Bondesson, Å., & Midlöv, P. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-070559 · Cited by 9
Ethical Pitfalls in AI‐Based Predictive Models in Surgery
Hmido, S. B., Rahim, H. A., Keller, B. P., Daams, F. D., Schakel, M., Goslings, J. C., van Dijkum, E. J. M. N., Rainey, S., Geert Kazemier, M., Bak, M., et al. (2025) · World Journal of Surgery · doi:10.1002/wjs.70080 · Cited by 8
Individualized Prediction of Platelet Transfusion Outcomes in Preterm Infants With Severe Thrombocytopenia
van der Staaij, H., Prosepe, I., Caram‐Deelder, C., Keogh, R. H., Deschmann, E., Dame, C., Onland, W., Prins, S. A., Cassel, F., d’Haens, E. J., et al. (2025) · JAMA · doi:10.1001/jama.2025.14194 · Cited by 8
Caveats in Using Abnormality/Probability Scores from Artificial Intelligence Algorithms: Neither True Probability nor Level of Trustworthiness
Park, S. H., & Hwang, E. J. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2024.0144 · Cited by 8
Machine learning for predicting duration of surgery and length of stay: A literature review on joint arthroplasty
Nejad, M. C., Matthiesen, R. V., Dukovska‐Popovska, I., Jakobsen, T. L., & Johansen, J. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105631 · Cited by 7
Predicting periprosthetic joint infection: external validation of preoperative prediction models
Yoon, S. I., Jutte, P. C., Soriano, À., de Sousa, R. J. G., Zijlstra, W. P., & Wouthuyzen‐Bakker, M. (2024) · Journal of Bone and Joint Infection · doi:10.5194/jbji-9-231-2024 · Cited by 7
Development and validation of an explainable machine learning and nomogram model for early detection and risk stratification of polycystic ovary syndrome: a multicenter study
Yao, B., Yu, X., Zhang, Y., Chen, J., Zhu, X., Zhang, C., & Jijun, T. (2025) · Frontiers in Endocrinology · doi:10.3389/fendo.2025.1719631 · Cited by 10
The role of artificial intelligence in life-sustaining treatment decisions: current state and future considerations
Wernly, B., Guidet, B., & Beil, M. (2024) · Intensive Care Medicine · doi:10.1007/s00134-024-07738-4 · Cited by 8
Clinical Validity of a Machine Learning Decision Support System for Early Detection of Hepatitis B Virus: A Binational External Validation Study
Ajuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6
Developing a clinical prediction model to modify empirical antibiotics for non-typhoidal Salmonella bloodstream infection in children under-five in the Democratic Republic of Congo
Tack, B., Vita, D., Mbuyamba, J., Ntangu, E., Vuvu, H., Kahindo, I., Ngina, J., Luyindula, A., Nama, N., Mputu, T., et al. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-024-10319-x · Cited by 7
AP-GRIP evaluation framework for data-driven train delay prediction models: systematic literature review
Tiong, K. Y., Ma, Z., & Palmqvist, C.-W. (2025) · European Transport Research Review · doi:10.1186/s12544-024-00704-7 · Cited by 6
Multi‐Centre External Validation of a Nomogram for 10‐Year Periodontal Tooth Loss Prediction
Serroni, M., Ravidà, A., Santamaria, P., Dias, D. R., Cheng, G., Shaikh, S., Natanzi, N. D., Saleh, M. H. A., Nibali, L., & Troiano, G. (2025) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.14143 · Cited by 6
Developing and Validating Predictive Models for Adverse Drug Reactions using Electronic Health Records (EHRs): A Narrative Review
Nwokedi, V. U., Odedele, M. K., Yusuff, T. A., Anderson, I. A., Akanbi, O. O., Agu, C. P., & Omoike, A. O. (2025) · Epidemiology and Health Data Insights · doi:10.63946/ehdi/17369 · Cited by 7
Using routinely collected data for research purposes: challenges and mitigation strategies
Hoffmann, S., Morris, T. P., Herrmann, M., Heinze, G., Wynants, L., CALSTER, B. V., Bischl, B., Schmid, M., Shaw, P. A., Mathes, T., et al. (2026) · BMJ · doi:10.1136/bmj-2025-087812 · Cited by 8
Morphometry-based radiomics for predicting therapeutic response in patients with gliomas following radiotherapy
Sherminie, L. P. G., Jayatilake, M. L., Hewavithana, B., Weerakoon, B. S., & Vijithananda, S. M. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1139902 · Cited by 6
Modest performance of text mining to extract health outcomes may be almost sufficient for high-quality prognostic model development
Grotenhuis, Z., Mosteiro, P. J., & Leeuwenberg, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108014 · Cited by 8
Personalized decision-making for aneurysm treatment of aneurysmal subarachnoid hemorrhage: development and validation of a clinical prediction tool
de Winkel, J., Roozenbeek, B., Dijkland, S. A., Dammers, R., van Doormaal, P., van der Jagt, M., Klaveren, D. V., Dippel, D. W. J., & Lingsma, H. F. (2024) · BMC Neurology · doi:10.1186/s12883-024-03546-x · Cited by 6
International multi-institutional external validation of preoperative risk scores for 30-day in-hospital mortality in paediatric patients
Tangel, V. E., Hoeks, S. E., Stolker, R. J., Brown, S. E. S., Pryor, K. O., de Graaff, J. C., Pace, N. L., Domino, K. B., Muehlschlegel, J. D., Kheterpal, S., et al. (2024) · British Journal of Anaesthesia · doi:10.1016/j.bja.2024.09.003 · Cited by 6
Balancing accuracy and Interpretability: An R package assessing complex relationships beyond the Cox model and applications to clinical prediction
Shamsutdinova, D. D., Stamate, D., & Stahl, D. R. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105700 · Cited by 6
Risk prediction tools for pressure injury occurrence: an umbrella review of systematic reviews reporting model development and validation methods
Hillier, B., Scandrett, K., Coombe, A., Hernandez‐Boussard, T., Steyerberg, E. W., Takwoingi, Y., Veličković, V. M., & Dinnes, J. (2025) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00182-4 · Cited by 8
Dynamic Updating of Psychosis Prediction Models in Individuals at Ultra-High Risk of Psychosis
Hartmann, S., Dwyer, D., Scott, I., Wannan, C., Nguyen, J., Lin, A. P., Middeldorp, C. M., Wood, S. J., Yung, A. R., McGorry, P. D., et al. (2025) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.03.006 · Cited by 6
Glaucoma Detection from Fundus Retinal Images Using Deep Feature Fusion and Residual Attention Mechanisms
Obayes, M. Q., Nassrullah, A. H. A., Abas, A., Hameed, M., Alomairi, A. O. W., Nayyef, H. A., & Ahmed, S. R. (2025) · International Conference on Cyber Resilience (ICCR) · doi:10.1109/iccr67387.2025.11292358 · Cited by 5
Planned but ever published? A retrospective analysis of clinical prediction model studies registered on clinicaltrials.gov since 2000
White, N. M., Parsons, R., Borg, D. N., Collins, P. G. S., & Barnett, A. G. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111433 · Cited by 5
Clinical prediction models for in vitro fertilization outcomes: a systematic review, meta-analysis, and external validation
Tian, C. H., Liu, L., Huang, Y. F., Yang, H. J., Lai, Y. Y., Li, C., Gan, D., & Yang, J. (2025) · Human Reproduction · doi:10.1093/humrep/deaf013 · Cited by 5
Declining Clinical Utility of Tools for Predicting Sentinel Lymph Node Biopsy Status: A Single Institution Experience from 2000 to 2021
Drebin, H. M., Kurtansky, N. R., Hosein, S., Nadelmann, E. R., Moy, A. P., Ariyan, C. E., Bello, D. M., Brady, M. S., Coit, D. G., Marchetti, M. A., et al. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-024-16698-4 · Cited by 5
A comparison of regression models for static and dynamic prediction of a prognostic outcome during admission in electronic health care recordsPreprint
Gao, S., Albu, E., Putter, H., Stijnen, P., Rademakers, F. E., Cossey, V., Debaveye, Y. A., Janssens, C., CALSTER, B. V., & Wynants, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4384973/v1 · Cited by 5
Machine learning and deep learning predictive models for prognosis in patients with atrial fibrillation: a systematic review and meta-analysis
Wang, X., Bampton, T. J., Dharmaprani, D., Mahajan, R., & Palmer, L. J. (2025) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000154 · Cited by 4
Analytical performance and calibration strategies of low-cost particulate matter sensors for indoor and workplace monitoring—a review
Feng, Z., Zheng, L., & Liu, D. (2025) · Analytical Methods · doi:10.1039/d5ay01554e · Cited by 4
Hybrid feature-selection and diversity-guided stacking framework for interpretable ensemble learning: Application to COVID-19 mortality prediction
Mohtasham, F., Nazari, S. S. H., Pourhoseingholi, M. A., Kavousi, K., & Zali, M. R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0341198 · Cited by 4

Authors: Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M.
Affiliations: KU Leuven (BE)
Maastricht University (NL)
ORCID: Van Calster, Ben
Steyerberg, Ewout W.
Wynants, Laure
van Smeden, Maarten
Venue: BMC Medicine
Publisher: Springer Science and Business Media LLC
Published: 2023-02-24
Volume: 21
Issue: 1
Pages: 70-70
ISSN: 1741-7015 (ISSN-L)
DOI: 10.1186/s12916-023-02779-w
PMC: PMC9951847
OpenAlex: W4321782827
Cited by: 286
References: 56
Citations by year: 2026: 99 · 2025: 91 · 2024: 67 · 2023: 29
FWCI: 58.07
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0 (tdm)
creativecommons.org/licenses/by/4.0 (vor)
Locations: BMC Medicine (publishedVersion, cc-by)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
Research Publications (Maastricht University) (publishedVersion, cc-by)
Leiden Repository (Leiden University) (submittedVersion, cc-by)
Lirias (KU Leuven) (submittedVersion, other-oa)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders: Fonds Wetenschappelijk Onderzoek (G097322N)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords: clinical prediction model, model validation, external validation, internal validation, model updating, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

There is no such thing as a validated prediction model

Abstract

Overview

Integrity signals

Figures

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks