The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Sample size considerations for the external validation of a multivariable prognostic model: a resampling study
Open accessAbstract
After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c ‐index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Collins et al. [10] recommended that externally validating a prognostic model requires a minimum of 100 events and 100 non-events.”
From The External Validation of a Multivariable Prediction Model for Recurrent Pelvic Organ Prolapse After Native Tissue Repair: A Prospective Cohort Study (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredIn the citing paper
“Previous recommendations were to include at least 100 to 200 individuals in the smallest outcome category.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Nishimura, R., Osako, T., Nishiyama, Y., Tashima, R., Nakano, M., Fujisue, M., Toyozumi, Y., & Arima, N. (2013) · Oncology · doi:10.1159/000353099 · Cited by 13,239
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,319
Self-citation
Flagged first, then by citations
1 of the 56 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 601–666 of 666
Zhong, L., Xu, Z., Liu, X., Wang, D., Tong, Q.-Y., Tian, X., Xiong, J., Hou, X., & Bai, T. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.120580
Zhou, Y., Zheng, Y., Li, C., Sharma, A., & Tang, L. (2025) · Epidemiologic Methods · doi:10.1515/em-2025-0012
Safety and effectiveness of the Kaiser Permanente early-onset neonatal sepsis calculator in Qatar
Vellamgot, A. P., Salim, S. T., Salameh, K., Thazhe, S. B. K., Elikkottil, A., Habboub, L., Pattuvalappil, R., & Joseph, B. E. (2025) · Frontiers in Pediatrics · doi:10.3389/fped.2025.1637914
Toward transparent and rigorous validation of a prediction score
Yen, C. (2025) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2025.108250
Authors’ reply “Toward transparent and rigorous validation of a prediction score”
Liu, R., Ru, X., & Ye, Q. (2025) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2025.108251
Senent‐Valero, M., Librero, J., & Pastor‐Valero, M. (2021) · Figshare · doi:10.6084/m9.figshare.17132896
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.26660464
OKADA, Y., Matsuyama, T., Hayashida, K., Takauji, S., Kanda, J., & Yokobori, S. (2022) · Figshare · doi:10.6084/m9.figshare.19905856
Adam, J. A., Poel, E., van Eck-Smit, B. L., Mom, C. H., Stalpers, L. J. A., Laan, J. J., Kidd, E. A., Stoker, J., & Bipat, S. (2024) · Figshare · doi:10.6084/m9.figshare.25678529
Han, K. S., Kim, S. J., Lee, E. J., Shin, J. H., Lee, J. S., & Lee, S. W. (2021) · Figshare · doi:10.6084/m9.figshare.13605569
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.26660470
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.24940484
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.26660473
Senent‐Valero, M., Librero, J., & Pastor‐Valero, M. (2021) · Figshare · doi:10.6084/m9.figshare.17132887
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.24940986
Luijken, K., Song, J., & Groenwold, R. H. H. (2022) · Figshare · doi:10.6084/m9.figshare.19533915
Dardenne, N., Locquet, M., Diep, A. N., Gilbert, A., Delrez, S., Beaudart, C., Brabant, C., Ghuysen, A., Donneau, A.-F., & Bruyère, O. (2022) · Figshare · doi:10.6084/m9.figshare.19767815
Senent‐Valero, M., Librero, J., & Pastor‐Valero, M. (2021) · Figshare · doi:10.6084/m9.figshare.17132893
Amacher, S. A., Blatter, R., Briel, M., Appenzeller‐Herzog, C., Bohren, C., Becker, C., Beck, K., Gross, S., Tisljar, K., Sutter, R., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26552957
Yang, C. B., Fridgeirsson, E. A., Kors, J. A., Reps, J. M., & Rijnbeek, P. R. (2024) · Figshare · doi:10.6084/m9.figshare.26660467
Williams, R. D., Markus, A. F., Yang, C. B., Duarte‐Salles, T., DuVall, S., Falconer, T., Jonnagaddala, J., Kim, C., Rho, Y., Williams, A. E., et al. (2022) · Figshare · doi:10.6084/m9.figshare.19094134
He, W., Fu, X., & Chen, S. (2024) · Figshare · doi:10.6084/m9.figshare.26619469
Senent‐Valero, M., Librero, J., & Pastor‐Valero, M. (2021) · Figshare · doi:10.6084/m9.figshare.17132884
Yamashita, Y., Inoue, G., Nozaki, Y., Kitajima, R., Matsubara, K., Horii, T., Mohri, J., Atsuda, K., & Matsubara, H. (2021) · Figshare · doi:10.6084/m9.figshare.17083637
Luijken, K., Song, J., & Groenwold, R. H. H. (2022) · Figshare · doi:10.6084/m9.figshare.19533909
Wang, P., Deng, J., & Wu, S. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2025.1579873
Therán-León, J. S., García, C. F., Pinzón, H. T., Strauch-Díaz, C. I., & Arenas-Moreno, C. E. (2026) · Research Square · doi:10.21203/rs.3.rs-8674390/v1
Hincapié, C., van der Aart, T. J., van Wijk, R. J., Föhse, K., Hoogerwerf, J. J., Jaimes, F., Koeze, J., Linder, A., van Londen, M., ter Maaten, J. C., et al. (2026) · Journal of Infection and Public Health · doi:10.1016/j.jiph.2026.103170
Mousse, A. H., Allavena, C., Gagneux‐Brunon, A., Raffi, F., Hocqueloux, L., Genet, C., Pascard, M., Jolly, D., Bani‐Sadr, F., Kanagaratnam, L., et al. (2026) · HIV Medicine · doi:10.1111/hiv.70207
Troeman, D. P. R., Harbarth, S., Kluytmans, J. A. J. W., van Werkhoven, C. H., the ASPIRE-SSI Study Team, Malhotra‐Kumar, S., Timbermont, L., Vlaeminck, J., Vilken, T., Xavier, B. B., et al. (2026) · Antimicrobial Resistance and Infection Control · doi:10.1186/s13756-026-01712-z
Qiao, F., Xue, X., Yu, H., Cai, Y., Tian, D., Wang, Y., Liu, Q., & Li, Q. (2026) · Scientific Reports · doi:10.1038/s41598-026-41953-x
Zhao, K.-H., Liu, J., Chen, Y.-S., Yi, W.-T., Liu, J.-J., & Zeng, Y. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1777061
Du, Q., Wang, Z., Yang, L., Liu, Y. H., Cao, J., Xia, M., & Yang, X. (2026) · Frontiers in Pediatrics · doi:10.3389/fped.2026.1770475
Ding, J., Chiu, J. C., Moon, S., Ren, Y., Turner, D., Shoval, G., Niv, Y., & Berwian, I. M. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1774560
Fazel, S., Heathcote, L., Farouki, L., Senior, J., Perry, A., Fanshawe, T. R., & Shaw, J. J. (2025) · Health Technology Assessment · doi:10.3310/gjsf1605
Li, X., Shen, Q., Liu, Z., Zhai, Q., & Zhang, X. (2026) · The Heart Surgery Forum · doi:10.31083/hsf50244
Validating an Oropharyngeal Cancer Risk Prediction Model with Data from a Canadian Population
Ghanati, H., Nicolau, B. F., Ying, Y., Koushik, A., & Madathil, S. (2026) · Journal of The Canadian Dental Association · doi:10.66515/jcda.92.q1
Self‐harm in individuals with substance use disorders: Predictive factors and risk model
Yu, R., Tracy, D. K., Sinclair, J., Brikell, I., Lichtenstein, P., Larsson, H., & Fazel, S. (2026) · Addiction · doi:10.1111/add.70465
de Groot, T. M., Gonzalez, M. R., van der Linden, L. R., Skornja, I., Groot, O. Q., Schwab, J. H., Doornberg, J. N., Jutte, P. C., & Lozano‐Calderón, S. A. (2026) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000003955 · Cited by 1
Sey, M. S. L., Renteln, D. A. V., Kohansal, A., van Zanten, S. V., Singh, H., Edmonds, M., Wiemer, H., Zhou, F., Pedicelli, A., Keshlaf, S., et al. (2026) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2026.05.014
Garrouste, V., Guérineau, A., Guénézan, J., Moumneh, T., Narcisse, S., Martin, P. L., Giovannetti, O., Bouvier, F., & Jouffroy, R. (2026) · Prehospital Emergency Care · doi:10.1080/10903127.2026.2677853
Eryanto, H., Marsofiyati, M., Adha, M. A., Manik, T., Ali, Y., & Sumardi, N. A. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.103054
Parry-Jones, T., Kostopoulos, G., Fragkos, K. C., Cohen, R., Roberts, G. P., Bond‐Smith, G., & Eddama, M. M. R. (2026) · International Journal of Surgery · doi:10.1097/js9.0000000000005433
Guillen-Ramirez, H., Lucas, K. L., Wintsch, Y. M., Blatter, T. U., Triep, K., Endrich, O., & Beldi, G. (2026) · Journal of the American Medical Informatics Association · doi:10.64898/2026.06.15.26355656
External validation of the VALE scoring system for hemorrhage risk in pediatric AVM patients
Kashefiolasl, S., Porto, L., Lachner, K., Merker, M., Schrewe, R., Schubert‐Bast, S., Keil, F., Prinz, V., Finger, T., & Czabanka, M. (2026) · Neurosurgical Review · doi:10.1007/s10143-026-04366-y
Chen, M. Y., Liu, H., Jin, L., Feng, X., Dai, B., Wang, F., Wang, Q., Chen, Y., Yi, M., Jia, B., et al. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1804485
Seifu, B. L., Belsti, Y., Yimer, N. B., Goldstein, R. F., Enticott, J., & Teede, H. J. (2026) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.70300
Canizalez‐Román, A., Llanes-Cardenas, R., Martínez‐Villa, F. A., & Izabal, D. Y. A. (2026) · Clinical & Translational Oncology · doi:10.1007/s12094-026-04490-7
Lee, M. J., Murray, A., Smith, C. J., Abdi, Z., Duncan, A., Dwyer, E., Gompels, M., Henderson, M., Johnson, M. A., Lander, F., et al. (2026) · HIV Medicine · doi:10.1111/hiv.70282
Pharmacogenomics of antipsychotic-induced weight gain: A systematic review
Kronenbuerger, M., Yoshida, K., Li, P. Y., Tavakoli, E., Magarbeh, L., Elsheikh, S. S. M., Gorbovskaya, I., Zai, C. C. H., Fleischmann, R., Zai, G., et al. (2026) · Schizophrenia Research · doi:10.1016/j.schres.2026.07.013
Karimzadeh, A., Fütterer, C., Hansen, K., Coban, Ö., Hein, S., Nekolla, S. G., Heck, M., Tauber, R., Haller, B., Gafita, A., et al. (2026) · Journal of Nuclear Medicine · doi:10.2967/jnumed.125.270944
Shen, M., Wang, Y., Wu, Z., Chen, L., Wu, S., Pan, H., & Xu, B. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1830769
Liu, R., Peng, B., Qi, K., Zhang, M., & Kuang, Y. (2026) · Spectrochimica Acta Part A Molecular and Biomolecular Spectroscopy · doi:10.1016/j.saa.2026.128518
Archer, L., Relton, S. D., Akbari, A., Best, K., Bestwick, R., Bucknall, M., Conroy, S., Hattle, M., Hollinghurst, J., Howdon, D., et al. (2025) · Health Technology Assessment · doi:10.3310/gjac1008
Rieck, C., Eisentraut, L., Penava, P., & Buettner, R. (2026) · IEEE Access · doi:10.1109/access.2026.3723687
Tracy, A., Fazla, F., Salluh, J. I. F., Haniffa, R., Uddin, R. A. M. E., Aryal, D., Brinkman, S., Burghi, G., Doherty, Z. B., Dongelmans, D. A., et al. (2026) · Critical Care and Resuscitation · doi:10.1016/j.ccrj.2026.100206
Alvis, B. D., Schmeckpeper, J., Rali, A. S., Huston, J., Tsai, S., Amancherla, K. V., Armstrong, D., Gupta, R., Mede, A. I., Whitfield, J. S., et al. (2026) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2026.08.010
Rajkovic, L. A., Veerbeek-Preuss, S., Bärtschi, S., Ottiger, B., Nyffeler, T., & Veerbeek, J. M. (2026) · Neurorehabilitation and neural repair · doi:10.1177/15459683261477077
Ito, A., Kamiya, M., Hayata, E., Nagasaki, S., & Nakata, M. (2026) · AJOG Global Reports · doi:10.1016/j.xagr.2026.100694
He, M., Zhang, Y., Xiao, J., Yang, M., Wen, Y., & Huang, D. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1926236
Negru, T., Stana, L. G., Stanga, L., Dejeu, D., Ilie, A. C., & Pantea, Ș. (2026) · Medical Sciences · doi:10.3390/medsci14060632
Wang, Y.-P., Wang, W., Chen, J., Luo, J.-D., Xuan, J., He, X., Liu, W.-M., Wei, X.-J., Wu, Q.-Y., Zhang, Y.-X., et al. (2026) · Gastric Cancer · doi:10.1007/s10120-026-01806-y
Karalunas, S. L., Faraone, S. V., Mooney, M. A., Weiskopf, N. G., Doyle, A. E., Kurtz‐Nelson, E. C., Keehn, R. M., Sibley, M. H., Wozniak, J. R., Zaback, T., et al. (2026) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2026.10.006
Metadata
- Authors
- Collins, G. S., Ogundimu, E. O., & Altman, D. G.
- Affiliations
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre University of Oxford Windmill Road Oxford OX3 7LD U.K.
- University of Oxford (GB)
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2015-11-09
- Volume
- 35
- Issue
- 2
- Pages
- 214-226
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC4738418
- OpenAlex
- W2122892545
- Cited by
- 665
- References
- 56
- Citations by year
- 2026: 51 · 2025: 87 · 2024: 77 · 2023: 60 · 2022: 89 · 2021: 90 · 2020: 74 · 2019: 51 · 2018: 38 · 2017: 24 · 2016: 23
- FWCI
- 25.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Statistics in Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Durham Research Online (Durham University) (publishedVersion, other-oa)
- Durham Research Online (Durham University) (acceptedVersion, other-oa)
- Durham Research Online (Durham University) (submittedVersion, cc-by)
- Northumbria Research Link (Northumbria University) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Medical Research Council (G1100513)
- Medical Research Council Prognosis Research Strategy (PROGRESS) Partnership (G0902393/99558)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- external validation, sample size, resampling, concordance index, calibration, D-statistic, event counts, model performance estimation
- MeSH terms
- Cardiovascular Diseases, Diabetes Mellitus, Type 2, Humans, Prognosis, Risk Factors, Models, Statistical, Multivariate Analysis, Databases, Factual, Sample Size, Validation Studies as Topic, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review