Cited by 666Statistical Methods in EpidemiologyStatistical Methods and InferenceStatistical Methods in Clinical Trials

The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

Open access

Collins, Gary S., Ogundimu, Emmanuel O., Altman, Douglas G.·2016·Statistics in Medicine

Open this paper in CoChat — free account Open PDF

Abstract

After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c ‐index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1002/sim.6787

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Evaluation of Factors Related to Late Recurrence - Later than 10 Years after the Initial Treatment - in Primary Breast Cancer
Nishimura, R., Osako, T., Nishiyama, Y., Tashima, R., Nakano, M., Fujisue, M., Toyozumi, Y., & Arima, N. (2013) · Oncology · doi:10.1159/000353099 · Cited by 13,239
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citation
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,319
Self-citation

Flagged first, then by citations

Show all 56 references

1 of the 56 references couldn’t be matched to a known paper.

Sorted by citations

Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Sample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K. G. M., de Groot, J. A. H., Collins, P. G. S., Altman, D. G., Eijkemans, M. J. C., & Reitsma, J. B. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 820
Self-citation
External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges
Riley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585
Self-citation
Prognosis for patients with amyotrophic lateral sclerosis: development and validation of a personalised prediction model
Westeneng, H., Debray, T. P. A., Visser, A. E., van Eijk, R. P. A., Rooney, J. P. K., Calvo, A., Opie-Martin, S., McDermott, C. J., Thompson, A. G., Pinto, S., et al. (2018) · The Lancet Neurology · doi:10.1016/s1474-4422(18)30089-9 · Cited by 544
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
Sample size determination: A practical guide for health researchers
Althubaiti, A. (2022) · Journal of general and family medicine · doi:10.1002/jgf2.600 · Cited by 437
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology
Gerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P., & Collins, P. G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 415
American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine
Kattan, M. W., Hess, K. R., Amin, M. B., Lü, Y., Moons, K. G. M., Gershenwald, J. E., Gimotty, P. A., Guinney, J., Halabi, S., Lazar, A. J., et al. (2016) · CA A Cancer Journal for Clinicians · doi:10.3322/caac.21339 · Cited by 334
Association between periodontal diseases and cardiovascular diseases, diabetes and respiratory diseases: Consensus report of the Joint Workshop by the European Federation of Periodontology ( EFP ) and the European arm of the World Organization of Family Doctors ( WONCA Europe)
Herrera, D., Sanz, M., Shapira, L., Brotons, C., Chapple, I., Frese, T., Graziani, F., Hobbs, R. F., Huck, O., Hummers, E., et al. (2023) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13807 · Cited by 315
Risk prediction models for selection of lung cancer screening candidates: A retrospective validation study
ten Haaf, K., Jeon, J., Tammemägi, M. C., Han, S. S., Kong, C. Y., Plevritis, S. K., Feuer, E. J., de Koning, H. J., Steyerberg, E. W., & Meza, R. (2017) · PLoS Medicine · doi:10.1371/journal.pmed.1002277 · Cited by 286
The importance of being external. methodological insights for the external validation of machine learning models in medicine
Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106288 · Cited by 284
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 271
The IASLC Lung Cancer Staging Project: Methodology and Validation Used in the Development of Proposals for Revision of the Stage Classification of NSCLC in the Forthcoming (Eighth) Edition of the TNM Classification of Lung Cancer
Detterbeck, F., Chansky, K., Groome, P., Bolejack, V., Crowley, J. J., Shemanski, L., Kennedy, C., Krasnik, M., Peake, M. D., Rami-Porta, R., et al. (2016) · Journal of Thoracic Oncology · doi:10.1016/j.jtho.2016.06.028 · Cited by 253
Development and Validation of a Clinically Based Risk Calculator for the Transdiagnostic Prediction of Psychosis
Fusar-Poli, P., Rutigliano, G., Stahl, D., Davies, C., Bonoldi, I., Reilly, T., & McGuire, P. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0284 · Cited by 246
Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges
Meehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219
Prognostic models for breast cancer: a systematic review
Phung, M. T., Tin, S. T., & Elwood, M. (2019) · BMC Cancer · doi:10.1186/s12885-019-5442-6 · Cited by 214
Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study
Riley, R. D., Snell, K. I. E., Archer, L., Ensor, J., Debray, T. P. A., CALSTER, B. V., van Smeden, M., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074821 · Cited by 213
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study
Gupta, R. K., Marks, M., Samuels, T. H. A., Luintel, A., Rampling, T., Chowdhury, H., Quartagno, M., Nair, A., Lipman, M. C. I., Abubakar, I. I., et al. (2020) · European Respiratory Journal · doi:10.1183/13993003.03498-2020 · Cited by 197
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197
Updated Alternative Fistula Risk Score (ua-FRS) to Include Minimally Invasive Pancreatoduodenectomy
Mungroop, T. H., Klompmaker, S., Wellner, U. F., Steyerberg, E. W., Coratti, A., D’Hondt, M. A., Pastena, M. D., Dokmak, S., Khatkov, I. E., Saint‐Marc, O., et al. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000003234 · Cited by 195
Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model
Collins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6986 · Cited by 185
Self-citation
An examination of American College of Surgeons NSQIP surgical risk calculator accuracy.
Cohen, M. E., Liu, Y., Ko, C. Y., & Hall, B. L. (2017) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2016.12.057 · Cited by 176
Poor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 173
Prediction model of bleeding after endoscopic submucosal dissection for early gastric cancer: BEST-J score
Hatta, W., Tsuji, Y., Yoshio, T., Kakushima, N., Hoteya, S., Doyama, H., Nagami, Y., Hikichi, T., Kobayashi, M., Morita, Y., et al. (2020) · Gut · doi:10.1136/gutjnl-2019-319926 · Cited by 175
Derivation and validation of a novel risk score for safe discharge after acute lower gastrointestinal bleeding: a modelling study
Oakland, K., Jairath, V., Uberoi, R., Guy, R., Ayaru, L., Mortensen, N., Murphy, M. F., & Collins, P. G. S. (2017) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(17)30150-4 · Cited by 164
How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods
Han, K., Song, K., & Choi, B. W. (2016) · Korean Journal of Radiology · doi:10.3348/kjr.2016.17.3.339 · Cited by 161
Validation in prediction research: the waste by data splitting
Steyerberg, E. W. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.07.010 · Cited by 157
Development and External Validation of a Prognostic Multivariable Model on Admission for Hospitalized Patients with COVID-19Preprint
Xie, J., Hungerford, D., Chen, H., Abrams, S. T., Li, S., Wang, G., Wang, Y., Kang, H., Bonnett, L. J., Zheng, R., et al. (2020) · SSRN Electronic Journal · doi:10.1101/2020.03.28.20045997 · Cited by 162
Development and Validation of a Scoring System to Predict Outcomes of Vedolizumab Treatment in Patients With Crohn’s Disease
Dulai, P. S., Boland, B. S., Singh, S., Chaudrey, K., Koliani‐Pace, J. L., Kochhar, G. S., Parikh, M. P., Shmidt, E., Hartke, J. G., Chilukuri, P., et al. (2018) · Gastroenterology · doi:10.1053/j.gastro.2018.05.039 · Cited by 149
Meta-Analysis Comparing Established Risk Prediction Models (EuroSCORE II, STS Score, and ACEF Score) for Perioperative Mortality During Cardiac Surgery
Sullivan, P. G., Wallach, J. D., & Ioannidis, J. P. A. (2016) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2016.08.024 · Cited by 144
Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142
Comparison of Multivariable Logistic Regression and Other Machine Learning Algorithms for Prognostic Prediction Studies in Pregnancy Care: Systematic Review and Meta-Analysis
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 148
Prediction of Persistent Post-Concussion Symptoms after Mild Traumatic Brain Injury
Cnossen, M. C., van der Naalt, J., Spikman, J. M., Nieboer, D., Yue, J. K., Winkler, E. A., Manley, G. T., von Steinbüchel, N., Polinder, S., Steyerberg, E. W., et al. (2018) · Journal of Neurotrauma · doi:10.1089/neu.2017.5486 · Cited by 153
The Kidney Failure Risk Equation for prediction of end stage renal disease in UK primary care: An external validation and clinical impact projection cohort study
Major, R., Shepherd, D. J., Medcalf, J. F., Xu, G., Gray, L. J., & Brunskill, N. J. (2019) · PLoS Medicine · doi:10.1371/journal.pmed.1002955 · Cited by 135
Embryo selection with artificial intelligence: how to evaluate and compare methods?
Kragh, M. F., & Karstoft, H. (2021) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-021-02254-6 · Cited by 133
Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective
Luijken, K., Groenwold, R. H. H., CALSTER, B. V., Steyerberg, E. W., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8183 · Cited by 139
External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125
Performance of Firth-and logF-type penalized methods in risk prediction for small or sparse binary data
Rahman, M. S., & Sultana, M. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0313-9 · Cited by 120
Comparison of Prehospital Scales for Predicting Large Anterior Vessel Occlusion in the Ambulance Setting
Nguyen, T., van den Wijngaard, I. R., Bosch, J., van Belle, E., van Zwet, E. W., Vermeulen, T., Duijndam, D., Koster, G. T., de Schryver, E. L. L. M., Kloos, L. M. H., et al. (2020) · JAMA Neurology · doi:10.1001/jamaneurol.2020.4418 · Cited by 117
Minimum sample size calculations for external validation of a clinical prediction model with a time‐to‐event outcome
Riley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116
Towards the best kidney failure prediction tool: a systematic review and selection aid
Ramspek, C. L., de Jong, Y., Dekker, F. W., & van Diepen, M. (2019) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfz018 · Cited by 108
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Validation of prediction models in the presence of competing risks: a guide through modern methods
van Geloven, N., Giardiello, D., Bonneville, E. F., Teece, L., Ramspek, C. L., van Smeden, M., Snell, K. I. E., van Calster, B., Pohar-Perme, M., Riley, R. D., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069249 · Cited by 106
Transdiagnostic Risk Calculator for the Automatic Detection of Individuals at Risk and the Prediction of Psychosis: Second Replication in an Independent National Health Service Trust
Fusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., McGuire, P., & Osborn, D. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sby070 · Cited by 95
Cardiovascular risk prediction in type 2 diabetes: a comparison of 22 risk scores in primary care settings
Dziopa, K., Asselbergs, F. W., Gratton, J., Chaturvedi, N., & Schmidt, A. F. (2022) · Diabetologia · doi:10.1007/s00125-021-05640-y · Cited by 106
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 99
Using natural language processing on electronic health records to enhance detection and prediction of psychosis risk.
Irving, J., Patel, R., Oliver, D., Colling, C., Pritchard, M., Broadbent, M., Baldwin, H., Stahl, D., Stewart, R., & Fusar-Poli, P. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa126 · Cited by 92
Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J., & Rijnbeek, P. R. (2024) · Journal Of Big Data · doi:10.1186/s40537-023-00857-7 · Cited by 91
Transparent reporting of multivariable prediction models developed or validated using clustered data: TRIPOD-Cluster checklist
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071018 · Cited by 101
Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation
Meertens, L. J. E., van Montfort, P., Scheepers, H. C. J., van Kuijk, S. M. J., Aardenburg, R., Langenveld, J., van Dooren, I. M. A., Zwaan, I. M., Spaanderman, M. E. A., & Smits, L. J. M. (2018) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13358 · Cited by 91
Prognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 92
Blood vessel segmentation in retinal fundus images using Gabor filters, fractional derivatives, and Expectation Maximization
Aguirre–Ramos, H., Avina‐Cervantes, J. G., Cruz-Aceves, I., Ruiz-Pinales, J., & Ledesma, S. (2018) · Applied Mathematics and Computation · doi:10.1016/j.amc.2018.07.057 · Cited by 85
Systematic review of prognostic models for recurrent venous thromboembolism (VTE) post-treatment of first unprovoked VTE
Ensor, J., Riley, R. D., Moore, D., Snell, K. I. E., Bayliss, S., & Fitzmaurice, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011190 · Cited by 84
Delirium prediction in the intensive care unit: comparison of two delirium prediction models
Wassenaar, A., Schoonhoven, L., Devlin, J. W., van Haren, F. M. P., Slooter, A. J. C., Jorens, P. G., van der Jagt, M., Simons, K. S., Egerod, I., Burry, L. D., et al. (2018) · Critical Care · doi:10.1186/s13054-018-2037-6 · Cited by 84
Arrhythmic risk prediction in arrhythmogenic right ventricular cardiomyopathy: external validation of the arrhythmogenic right ventricular cardiomyopathy risk calculator
Jordà, P., Bosman, L. P., Gasperetti, A., Mazzanti, A., Gourraud, J., Davies, B., Frederiksen, T. C., Weidmann, Z. M., Marco, A. D., Roberts, J. D., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac289 · Cited by 86
Just How Confident Can We Be in Predicting Sports Injuries? A Systematic Review of the Methodological Conduct and Performance of Existing Musculoskeletal Injury Prediction Models in Sport
Bullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, P. G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 92
Trends in the conduct and reporting of clinical prediction model development and validation: a systematic review
Yang, C. B., Kors, J. A., Ioannou, S., John, L. H., Markus, A. F., Rekkas, A., de Ridder, M. A. J., Seinen, T. M., Williams, R. D., & Rijnbeek, P. R. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac002 · Cited by 81
External Validation of the Oakland Score to Assess Safe Hospital Discharge Among Adult Patients With Acute Lower Gastrointestinal Bleeding in the US
Oakland, K., Kothiwale, S., Forehand, T., Jackson, E. S., Bucknall, C. A., Sey, M. S. L., Singh, S., Jairath, V., & Perlin, J. B. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.9630 · Cited by 71
Development and validation of a multivariate predictive model for rheumatoid arthritis mortality using a machine learning approach
Lezcano, J. M., Salazar, F., León, L., Toledano, E., Jover, J. Á., Fernández‐Gutiérrez, B., Soudah, E., González‐Álvaro, I., Abásolo, L., & Rodríguez‐Rodríguez, L. (2017) · Scientific Reports · doi:10.1038/s41598-017-10558-w · Cited by 73
SCORE2 cardiovascular risk prediction models in an ethnic and socioeconomic diverse population in the Netherlands: an external validation study
Kist, J. M., Vos, R. C., Mairuhu, A. T. A., Struijs, J. N., van Peet, P. G., Vos, H. M. M., van Os, H. J. A., Beishuizen, E. D., Sijpkens, Y. W. J., Faiq, M. A., et al. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.101862 · Cited by 69
Automatic Triage of 12‐Lead ECGs Using Deep Convolutional Neural Networks
van de Leur, R. R., Blom, L. J., Gavves, E., Hof, I. E., van der Heijden, J. F., Clappers, N., Doevendans, P. A., Hassink, R. J., & van Es, R. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.015138 · Cited by 70
Sample Size Analysis for Machine Learning Clinical Validation Studies
Goldenholz, D. M., Sun, H., Ganglberger, W., & Westover, M. B. (2023) · Biomedicines · doi:10.3390/biomedicines11030685 · Cited by 73
Sensitivity of the Kaiser Permanente early-onset sepsis calculator: A systematic review and meta-analysis
Pettinger, K. J., Mayers, K., McKechnie, L., & Phillips, B. (2019) · EClinicalMedicine · doi:10.1016/j.eclinm.2019.11.020 · Cited by 67
Performance of Cardiovascular Disease Risk Scores in People Diagnosed With Type 2 Diabetes: External Validation Using Data From the National Scottish Diabetes Register
Read, S. H., van Diepen, M., Colhoun, H. M., Halbesma, N., Lindsay, R. S., McKnight, J. A., McAllister, D., Pearson, E. R., Petrie, J. R., Philip, S., et al. (2018) · Diabetes Care · doi:10.2337/dc18-0578 · Cited by 66
A comparison of injury scoring systems in predicting burn mortality.
Halgas, B. J., Bay, R. C., & Foster, K. N. (2018) · Europe PMC (PubMed Central) · Cited by 66
Correction: The Kidney Failure Risk Equation for prediction of end stage renal disease in UK primary care: An external validation and clinical impact projection cohort study
Major, R., Shepherd, D. J., Medcalf, J. F., Xu, G., Gray, L. J., & Brunskill, N. J. (2020) · PLoS Medicine · doi:10.1371/journal.pmed.1003313 · Cited by 66
External validation of the Revised Cardiac Risk Index and National Surgical Quality Improvement Program Myocardial Infarction and Cardiac Arrest calculator in noncardiac vascular surgery
Fronczek, J., Polok, K., Devereaux, P. J., Górka, J., Archbold, A., Biccard, B. M., Duceppe, E., Manach, Y. L., Sessler, D. I., Duchińska, M., et al. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.05.029 · Cited by 63
PRECISE-DAPT score for bleeding risk prediction in patients on dual or single antiplatelet regimens: insights from the GLOBAL LEADERS and GLASSY
Gragnano, F., Heg, D., Franzone, A., McFadden, E. P., Leonardi, S., Piccolo, R., Vranckx, P., Branca, M., Serruys, P. W. J. C., Benit, E., et al. (2020) · European Heart Journal - Cardiovascular Pharmacotherapy · doi:10.1093/ehjcvp/pvaa106 · Cited by 62
Predicting Myocardial Injury and Other Cardiac Complications After Elective Noncardiac Surgery with the Revised Cardiac Risk Index: The VISION Study
Roshanov, P. S., Sessler, D. I., Chow, C. K., Garg, A. X., Walsh, M. W., Lam, N. N., Hildebrand, A. M., Biccard, B. M., Acedillo, R. R., MacNeil, S. D., et al. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.03.015 · Cited by 66
Clinical prediction in defined populations: a simulation study investigating when and how to aggregate existing models
Martin, G. P., Mamas, M. A., Peek, N. B., Buchan, I. E., & Sperrin, M. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0277-1 · Cited by 60
Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 64
Hypotension Prediction Index with non-invasive continuous arterial pressure waveforms (ClearSight): clinical performance in Gynaecologic Oncologic Surgery
Frassanito, L., Giuri, P. P., Vassalli, F., Piersanti, A., Longo, A., Zanfini, B. A., Catarci, S., Fagotti, A., Scambia, G., & Draisci, G. (2021) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-021-00763-4 · Cited by 58
Black Box Prediction Methods in Sports Medicine Deserve a Red Card for Reckless Practice: A Change of Tactics is Needed to Advance Athlete Care
Bullock, G. S., Hughes, T., Arundale, A. J. H., Ward, P., Collins, P. G. S., & Kluzek, S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01655-6 · Cited by 61
Analysis of lung cancer risk model (PLCOM2012 and LLPv2) performance in a community-based lung cancer screening programme
Lebrett, M. B., Balata, H., Evison, M., Colligan, D., Duerden, R., Elton, P., Greaves, M., Howells, J., Irion, K., Karunaratne, D., et al. (2020) · Thorax · doi:10.1136/thoraxjnl-2020-214626 · Cited by 57
Neural network and logistic regression diagnostic prediction models for giant cell arteritis: development and validation
Ing, E. B., Miller, N. R., Nguyen, A. M., Su, W., Bursztyn, L. L. C. D., Poole, M., Kansal, V., Toren, A., Albreiki, D., Mouhanna, J. G., et al. (2019) · Clinical ophthalmology · doi:10.2147/opth.s193460 · Cited by 57
Estimation of required sample size for external validation of risk models for binary outcomes
Pavlou, M., Chen, Q., Omar, R., Seaman, S. R., Steyerberg, E. W., White, I. R., & Ambler, G. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211007522 · Cited by 58
Machine learning to predict early recurrence after oesophageal cancer surgery
Rahman, S., Walker, R., Lloyd, M. A., Grace, B. L., van Boxel, G. I., Kingma, B. F., Ruurda, J. P. H., van Hillegersberg, R., Harris, S., Parsons, S. L., et al. (2020) · British journal of surgery · doi:10.1002/bjs.11461 · Cited by 56
The new simplified MARIA score applies beyond clinical trials: A suitable clinical practice tool for Crohn’s disease that parallels a simple endoscopic index and fecal calprotectin
Roseira, J., Ventosa, A. R., de Sousa, H. T., & Brito, J. B. (2020) · United European Gastroenterology Journal · doi:10.1177/2050640620943089 · Cited by 57
Metabolomic and genomic prediction of common diseases in 700,217 participants in three national biobanks
Nightingale Health Biobank Collaborative Group, Barrett, J. C., Esko, T., Fischer, K., Jostins-Dean, L., Jousilahti, P., Julkunen, H., Jääskeläinen, T., Kangas, A. J., Kerimov, N., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-54357-0 · Cited by 74
Prediction models for the mortality risk in chronic dialysis patients: a systematic review and independent external validation study
Ramspek, C. L., Voskamp, P. W. M., van Ittersum, F. J., Krediet, R. T., Dekker, F. W., & van Diepen, M. (2017) · Clinical Epidemiology · doi:10.2147/clep.s139748 · Cited by 56
Predicting Chemotherapy Toxicity in Older Patients with Cancer: A Multicenter Prospective Study
Feliú, J., Jiménez-Munárriz, B., Basterretxea, L., Paredero, I., Llabrés, E., Rebollo, M. A., Losada, B., Espinosa, E., Gironés, R., Custodio, A., et al. (2020) · The Oncologist · doi:10.1634/theoncologist.2019-0701 · Cited by 50
Derivation and Validation of a Biomarker-Based Clinical Algorithm to Rule Out Sepsis From Noninfectious Systemic Inflammatory Response Syndrome at Emergency Department Admission: A Multicenter Prospective Study*
Mearelli, F., Fiotti, N., Giansante, C., Casarsa, C., Orso, D., Helmersen, M. D., Altamura, N., Ruscio, M., Castello, L. M., Colonetti, E., et al. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003206 · Cited by 50
Development and external validation of DISPAIR fistula risk score for clinically relevant postoperative pancreatic fistula risk after distal pancreatectomy
Bonsdorff, A., Ghorbani, P., Helanterä, I., Tarvainen, T., Kontio, T., Belfrage, H., Sirén, J., Kokkola, A., Sparrelid, E., & Sallinen, V. J. (2022) · British journal of surgery · doi:10.1093/bjs/znac266 · Cited by 49
Show and tell: A critical review on robustness and uncertainty for a more responsible medical AI
Marconi, L., & Cabitza, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105970 · Cited by 58
Development and external validation of the Psychosis Metabolic Risk Calculator (PsyMetRiC): a cardiometabolic risk prediction algorithm for young people with psychosis
Perry, B. I., Osimo, E. F., Upthegrove, R., Mallikarjun, P. K., Yorke, J., Štochl, J., Pérez, J., Zammit, S., Howes, O. D., Jones, P. B., et al. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00114-0 · Cited by 50
Prediction modelling in the early detection of neonatal sepsis
Sahu, P., Stanly, E. A. R., Lewis, L., Prabhu, K., Rao, M., & Kunhikatta, V. (2022) · World Journal of Pediatrics · doi:10.1007/s12519-021-00505-1 · Cited by 56
Assessment of predictive performance of caries risk assessment models based on a systematic review and meta-analysis
Su, N., Lagerweij, M. D., & van der Heijden, G. J. M. G. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103664 · Cited by 49
Validation and updating of risk models based on multinomial logistic regression
CALSTER, B. V., Hoorde, K. V., Vergouwe, Y., Bobdiwala, S., Condous, G. S., Kirk, E., Bourne, T., & Steyerberg, E. W. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-016-0002-x · Cited by 47
A novel risk calculator to predict outcome after surgery for symptomatic spinal metastases; use of a large prospective patient database to personalise surgical management
Choi, D., Pavlou, M., Omar, R., Arts, M. P., Balabaud, L., Buchowski, J. M., Bünger, C. E., Chung, C. K., Coppes, M. H., Depreitere, B., et al. (2018) · European Journal of Cancer · doi:10.1016/j.ejca.2018.11.011 · Cited by 47
Assessment and comparison of the max-ICH score and ICH score by external validation
Schmidt, F., Liotta, E. M., Prabhakaran, S., Naidech, A. M., & Maas, M. B. (2018) · Neurology · doi:10.1212/wnl.0000000000006117 · Cited by 45
A practical solution to estimate the sample size required for clinical prediction models generated from observational research on data
Baeza-Delgado, C., Alberich, L. C., Sierra, J. M. C., Veiga-Canuto, D., Heras, B. M. D. L., Raza, B., & Martí‐Bonmatí, L. (2022) · European Radiology Experimental · doi:10.1186/s41747-022-00276-y · Cited by 45
Discrimination and calibration properties of the hypotension probability indicator during cardiac and vascular surgery
Ranucci, M., Barile, L., Ambrogi, F., & Pistuddi, V. (2019) · Minerva Anestesiologica · doi:10.23736/s0375-9393.18.12620-4 · Cited by 45
Prediction models for development of retinopathy in people with type 2 diabetes: systematic review and external validation in a Dutch primary care setting
van der Heijden, A. A. W. A., Nijpels, G., Badloe, F. M. S., Lovejoy, H. L., Peelen, L. M., Feenstra, T., Moons, K. G. M., Slieker, R. C., Herings, R. M. C., Elders, P. J. M., et al. (2020) · Diabetologia · doi:10.1007/s00125-020-05134-3 · Cited by 48

Authors: Collins, G. S., Ogundimu, E. O., & Altman, D. G.
Affiliations: Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre University of Oxford Windmill Road Oxford OX3 7LD U.K.
University of Oxford (GB)
ORCID: Collins, Gary S.
Ogundimu, Emmanuel O.
Altman, Douglas G.
Venue: Statistics in Medicine
Publisher: Wiley
Published: 2015-11-09
Volume: 35
Issue: 2
Pages: 214-226
ISSN: 0277-6715 (ISSN-L)
1097-0258
DOI: 10.1002/sim.6787
PMC: PMC4738418
OpenAlex: W2122892545
Cited by: 665
References: 56
Citations by year: 2026: 51 · 2025: 87 · 2024: 77 · 2023: 60 · 2022: 89 · 2021: 90 · 2020: 74 · 2019: 51 · 2018: 38 · 2017: 24 · 2016: 23
FWCI: 25.28
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/ (vor)
doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
Locations: Statistics in Medicine (publishedVersion, cc-by)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
Durham Research Online (Durham University) (publishedVersion, other-oa)
Durham Research Online (Durham University) (acceptedVersion, other-oa)
Durham Research Online (Durham University) (submittedVersion, cc-by)
Northumbria Research Link (Northumbria University) (submittedVersion)
University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Funders: Medical Research Council (G1100513)
Medical Research Council Prognosis Research Strategy (PROGRESS) Partnership (G0902393/99558)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords: external validation, sample size, resampling, concordance index, calibration, D-statistic, event counts, model performance estimation
MeSH terms: Cardiovascular Diseases, Diabetes Mellitus, Type 2, Humans, Prognosis, Risk Factors, Models, Statistical, Multivariate Analysis, Databases, Factual, Sample Size, Validation Studies as Topic, Biostatistics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks