Cited by 1,409Statistical Methods in EpidemiologyStatistical Methods and InferenceStatistical Methods in Clinical Trials

Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes

Open access
Riley, Richard D, Snell, Kym IE, Ensor, Joie, Burke, Danielle L, Harrell Jr, Frank E, Moons, Karel GM, Collins, Gary S·2019·Statistics in Medicine

Abstract

When designing a study to develop a new prediction model with binary or time‐to‐event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants ( n ) and outcome events ( E ) relative to the number of predictor parameters ( p ) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥ 0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R 2 , and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p , and require prespecification of the model's anticipated Cox‐Snell R 2 , which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/sim.7992

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The model development lifecycle includes internal validation, external testing, calibration assessment, and monitoring to detect performance drift over time.

    “The model development lifecycle includes internal validation, external testing, calibration assessment, and monitoring to detect performance drift over time [43].”

    From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [43]

    The section on prespecifying R² discusses external validation as a way to estimate performance in new individuals, and the Discussion mentions calibration in new data. The paper focuses on sample-size criteria for model development; the passages do not present the claimed lifecycle or address monitoring for performance drift over time. Page numbers were not available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • Adequate sample size is important for prediction model development, especially when outcomes are rare or many predictors are used, because insufficient sample size increases overfitting risk.

    “Adequate sample size is important for prediction model development, especially when outcomes are rare or many predictors are used, because insufficient sample size increases overfitting risk [43].”
  • Reliable detection of moderate molecular effects in Alzheimer's disease requires at least 500 paired omics and imaging samples, yet most studies fall short of this benchmark.

    “For instance, reliable detection of moderate molecular effects in AD requires at least 500 paired omics and imaging samples, yet most studies fall short of this benchmark423.”

    From Artificial intelligence-driven multi-omics approaches in Alzheimer's disease: Progress, challenges, and future directions (2025) · cites this paper as [423]

    The section on precise estimates of predictor effects says sample-size needs depend on the effect and study characteristics; its worked example concerns venous thromboembolism, not Alzheimer’s disease. Searches for Alzheimer’s disease, paired omics and imaging, and a 500-sample threshold found no passage supporting that benchmark or the assertion that most studies fall short. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 61 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. [33]Mittlboeck M, Heinzl H. Pseudo R‐squared measures of generalized linear models. In: Proceedings of the 1st European Workshop on the Assessment of Diagnostic Performance; 2004; Milan, Italy.
  2. [49]Royston P, Lambert PC. Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model. Boca Raton, Fl: CRC Press; 2011.

Cited by

Sorted by citations

Authors
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S.
Affiliations
  • Keele University (GB)
  • Department of Biostatistics Vanderbilt University School of Medicine Nashville Tennessee
  • University Medical Center Utrecht (NL)
  • Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences University of Oxford Oxford UK
Corresponding author
Riley, Richard D
Venue
Statistics in Medicine
Publisher
Wiley
Published
2018-10-24
Volume
38
Issue
7
Pages
1276-1296
ISSN
  • 0277-6715 (ISSN-L)
  • 1097-0258
OpenAlex
W2898559532
Cited by
1,409
References
61
Citations by year
2026: 483 · 2025: 250 · 2024: 186 · 2023: 134 · 2022: 120 · 2021: 123 · 2020: 93 · 2019: 16 · 2018: 3
FWCI
49.17
Type
journal article
Language
en
Open access
Locations
  • Statistics in Medicine (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by-nc)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
Funders
  • National Institute for Health Research
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (9120.8004, 918.10.615)
  • National Center for Advancing Translational Sciences (UL1 TR002243)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
minimum sample size, binary outcomes, time-to-event outcomes, overfitting, shrinkage factor, prognostic model, diagnostic model, Chagas disease, recurrent venous thromboembolism
MeSH terms
Computer Simulation, Humans, Regression Analysis, Time, Multivariate Analysis, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review