The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Sample size considerations for the external validation of a multivariable prognostic model: a resampling study
Open accessAbstract
After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c ‐index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Collins et al. [10] recommended that externally validating a prognostic model requires a minimum of 100 events and 100 non-events.”
From The External Validation of a Multivariable Prediction Model for Recurrent Pelvic Organ Prolapse After Native Tissue Repair: A Prospective Cohort Study (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredIn the citing paper
“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Nishimura, R., Osako, T., Nishiyama, Y., Tashima, R., Nakano, M., Fujisue, M., Toyozumi, Y., & Arima, N. (2013) · Oncology · doi:10.1159/000353099 · Cited by 13,239
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,319
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508
Self-citationPredicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., Minhas, R., Sheikh, A., & Brindle, P. (2008) · BMJ · doi:10.1136/bmj.39609.449676.25 · Cited by 1,459 · cited 3×
“9 million person years of observation) contributing 96709 cardiovascular events from the QRESEARCH database [20].”
BackgroundMethodPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195
Self-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
External validation of a Cox prognostic model: principles and methods
Royston, P. J., & Altman, D. G. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-33 · Cited by 911
Self-citationThe design of simulation studies in medical statistics
Burton, A., Altman, D. G., Royston, P. J., & HOLDER, R. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2673 · Cited by 755
Self-citationCollins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729
Self-citationVergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636
Debray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542
Gerds, T., & Schumacher, M. (2006) · Biometrical journal. Biometrische Zeitschrift · doi:10.1002/bimj.200610301 · Cited by 541
Collins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 473
A new measure of prognostic separation in survival data
Royston, P., & Sauerbrei, W. (2004) · Statistics in Medicine · doi:10.1002/sim.1621 · Cited by 450 · cited 5×
“We examined two R(2)-type measures [40,41] (explained variation [32] and explained randomness”
BackgroundMethodInfluentialHippisley‐Cox, J., Coupland, C. A. C., Robson, J. P., Sheikh, A., & Brindle, P. M. (2009) · BMJ · doi:10.1136/bmj.b880 · Cited by 373
Systematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P. J., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 351
Explained Variation for Survival Models
Royston, P. (2006) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0600600105 · Cited by 319
An updated catalog of prostate cancer predictive tools
Shariat, S. F., Karakiewicz, P. I., Roehrborn, C. G., & Kattan, M. W. (2008) · Cancer · doi:10.1002/cncr.23908 · Cited by 264
Collins, P. G. S., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c2442 · Cited by 250
Self-citationCollins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 245
Self-citationVergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243
Kooperberg, C. L., Stone, C. J., & Truong, Y. K.-N. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476491 · Cited by 180
Explained randomness in proportional hazards models
O’Quigley, J., Xu, R., & Stare, J. (2004) · Statistics in Medicine · doi:10.1002/sim.1946 · Cited by 152
A comparison of imputation methods in a longitudinal randomized clinical trial
Tang, L., Song, J., Belin, T. R., & Unützer, J. (2005) · Statistics in Medicine · doi:10.1002/sim.2099 · Cited by 124
Assessment of survival prediction models based on microarray data
Schumacher, M., Binder, H., & Gerds, T. (2007) · Bioinform. · doi:10.1093/bioinformatics/btm232 · Cited by 114
Peek, N., Arts, D. G. T., Bosman, R. J., van der Voort, P. H. J., & de Keizer, N. F. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.08.011 · Cited by 92
Choodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2011) · Statistics in Medicine · doi:10.1002/sim.4242 · Cited by 91
Barriers to Routine Risk-Score Use for Healthy Primary Care Patients
Müller‐Riemenschneider, F. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.66 · Cited by 87
External validation of QDSCORE® for predicting the 10‐year risk of developing Type 2 diabetes
Collins, P. G. S., & Altman, D. G. (2011) · Diabetic Medicine · doi:10.1111/j.1464-5491.2011.03237.x · Cited by 61
Self-citationCollins, G. S., & Altman, D. (2012) · British Journal of Cancer · doi:10.1038/bjc.2012.266 · Cited by 60
Self-citationBrusselaers, N., Juhász, I., Erdei, I., Monstrey, S., & Blot, S. I. (2009) · Burns · doi:10.1016/j.burns.2008.12.017 · Cited by 42
Collins, P. G. S., & Altman, D. G. (2012) · Heart · doi:10.1136/heartjnl-2012-302014 · Cited by 33
Self-citationChamogeorgakis, T. P., Toumpoulis, I. K., Tomos, P., Ieromonachos, C., Angouras, D. C., Georgiannakis, E., Michail, P., & Rokkas, C. K. (2009) · Interactive Cardiovascular and Thoracic Surgery · doi:10.1510/icvts.2008.201178 · Cited by 32
Identifying suspected breast cancer: development and validation of a clinical prediction rule
McCowan, C., Donnan, P. T., Dewar, J. A., Thompson, A. M., & Fahey, T. P. (2011) · British Journal of General Practice · doi:10.3399/bjgp11x572391 · Cited by 31
Collins, P. G. S., & Altman, D. G. (2012) · European Journal of Cancer · doi:10.1016/j.ejca.2012.10.023 · Cited by 27
Self-citationCollins, P. G. S., & Altman, D. G. (2012) · British Journal of General Practice · doi:10.3399/bjgp12x636065 · Cited by 26
Self-citationChoodari‐Oskooei, B., Royston, P. J., & Parmar, M. (2012) · Statistics in Medicine · doi:10.1002/sim.5460 · Cited by 25
Collins, P. G. S., & Altman, D. G. (2012) · European Journal of Cancer Care · doi:10.1111/ecc.12015 · Cited by 20
Self-citationCollins, P. G. S., & Altman, D. G. (2012) · Cancer Epidemiology · doi:10.1016/j.canep.2012.11.005 · Cited by 16
Self-citation- Self-citation
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Flagged first, then by citations
1 of the 56 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Metadata
- Authors
- Collins, G. S., Ogundimu, E. O., & Altman, D. G.
- Affiliations
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre University of Oxford Windmill Road Oxford OX3 7LD U.K.
- University of Oxford (GB)
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2015-11-09
- Volume
- 35
- Issue
- 2
- Pages
- 214-226
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC4738418
- OpenAlex
- W2122892545
- Cited by
- 665
- References
- 56
- Citations by year
- 2026: 51 · 2025: 87 · 2024: 77 · 2023: 60 · 2022: 89 · 2021: 90 · 2020: 74 · 2019: 51 · 2018: 38 · 2017: 24 · 2016: 23
- FWCI
- 25.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Statistics in Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Durham Research Online (Durham University) (publishedVersion, other-oa)
- Durham Research Online (Durham University) (acceptedVersion, other-oa)
- Durham Research Online (Durham University) (submittedVersion, cc-by)
- Northumbria Research Link (Northumbria University) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Medical Research Council (G1100513)
- Medical Research Council Prognosis Research Strategy (PROGRESS) Partnership (G0902393/99558)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- external validation, sample size, resampling, concordance index, calibration, D-statistic, event counts, model performance estimation
- MeSH terms
- Cardiovascular Diseases, Diabetes Mellitus, Type 2, Humans, Prognosis, Risk Factors, Models, Statistical, Multivariate Analysis, Databases, Factual, Sample Size, Validation Studies as Topic, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
