Calculating the sample size required for developing a clinical prediction model

Riley, Richard D; Ensor, Joie; Snell, Kym I E; Harrell, Frank E; Martin, Glen P; Reitsma, Johannes B; Moons, Karel G M; Collins, Gary; van Smeden, Maarten

doi:10.1136/bmj.m441

Cited by 2,917Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsStatistical Methods in Clinical Trials

Calculating the sample size required for developing a clinical prediction model

Open access

Riley, Richard D, Ensor, Joie, Snell, Kym I E, Harrell, Frank E, Martin, Glen P, Reitsma, Johannes B, Moons, Karel G M, Collins, Gary +1 more·2020·BMJ

DOI 10.1136/bmj.m441

Open this paper in CoChat — free account Open PDF

Abstract

Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1136/bmj.m441

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The sample size needed to reduce model overfitting was calculated assuming an expected shrinkage rate of 10%, 30 variables, and R2cs 0.3, yielding 740 cases.
“In step three, the sample size needed to reduce model overfitting is calculated, assuming an expected shrinkage rate of 10%, with an expected number of variables P to be included in the final predictive model of 30: S is the expected shrinkage factor, and the noise ratio evaluation index R2cs is 0.3; the calculated sample size is 740 cases.”
From Machine learning-driven prediction model for successful weaning of patients from mechanical ventilation in ICU (2026) · cites this paper as Riley [13]
The paper’s Step 3 section (page not available in our copy) supports calculating sample size to limit overfitting using 10% expected shrinkage, the number of candidate predictor parameters, and anticipated Cox–Snell R². It does not report the claimed calculation of 740 cases for 30 parameters and R²cs of 0.3; its worked examples use different inputs. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Genomic and molecular data support integration of tumour biology and genetic risk into decision-making, particularly in precision medicine applications.
“Genomic and molecular data support integration of tumour biology and genetic risk into decision-making, particularly in precision medicine applications [42].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [42]
The Abstract passage says clinical prediction models can use family history and biomarkers to inform diagnosis and prognosis. It does not address genomic or molecular data, tumour biology, genetic risk, or precision-medicine decisions; searches for those terms found no supporting passage in our copy. (LLM, conf=0.74)
Experimental — accuracy not yet measured
The sample size calculated using Riley and colleagues' guidance is the minimum for a standard statistical model.
“The sample size calculated this way is the minimum for a standard statistical model.”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [50]
Sample size calculations indicated a minimum sample of 2082 patients, which is larger than the available sample, raising concerns about possible overfitting.
“Sample size calculations50 indicated a minimum sample of 2082 patients, which is larger than the available sample, raising concerns about possible overfitting issues (step 6).”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [50]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957 · see citing works
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · see citing works
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
Vittinghoff, E., & McCulloch, C. E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwk052 · Cited by 4,359 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984 · see citing works
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
The Analysis of Binary Data.
Plackett, R. L., & Cox, D. R. (1971) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344247 · Cited by 2,273 · see citing works
Cardiovascular disease risk profiles
Anderson, K. M., Odell, P. M., Wilson, P. W. F., & Kannel, W. B. (1991) · American Heart Journal · doi:10.1016/0002-8703(91)90861-b · Cited by 2,163 · see citing works
Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates
Peduzzi, P., Concato, J., Feinstein, A. R., & Holford, T. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00048-8 · Cited by 2,154 · see citing works
Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study
Hippisley-Cox, J., Coupland, C., & Brindle, P. (2017) · BMJ · doi:10.1136/bmj.j2099 · Cited by 1,703 · see citing works
Variable selection – A review and recommendations for the practicing statistician
Heinze, G., Wallisch, C., & Dunkler, D. (2018) · Biometrical Journal · doi:10.1002/bimj.201700067 · Cited by 1,673 · see citing works
PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Self-citation
Derivation of a Simple Clinical Model to Categorize Patients Probability of Pulmonary Embolism: Increasing the Models Utility with the SimpliRED D-dimer
Anderson, D., Rodger, M., Ginsberg, J., Kearon, C., Gent, M., Turpie, A., Bormanis, J., Weitz, J., Chamberlain, M., Bowie, D., et al. (2000) · Thrombosis and Haemostasis · doi:10.1055/s-0037-1613830 · Cited by 1,651 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citation
Clinical Prediction Models
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Self-citation
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · see citing works
Value of assessment of pretest probability of deep-vein thrombosis in clinical management
Wells, P. S., Anderson, D. R., Bormanis, J., Guy, F., Mitchell, M. J., Gray, L. C., Clement, C., Robinson, K. S., & Lewandowski, B. J. (1997) · The Lancet · doi:10.1016/s0140-6736(97)08140-3 · Cited by 1,209 · see citing works
The Nottingham prognostic index in primary breast cancer
Galea, M. H., Blamey, R. W., Elston, C. W., & Ellis, I. O. (1992) · Breast Cancer Research and Treatment · doi:10.1007/bf01840834 · Cited by 1,071 · see citing works
The number of subjects per variable required in linear regression analyses
Austin, P. C., & Steyerberg, E. W. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.014 · Cited by 1,067 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Sparse data bias: a problem hiding in plain sight
Greenland, S., Mansournia, M. A., & Altman, D. G. (2016) · BMJ · doi:10.1136/bmj.i1981 · Cited by 898 · see citing works
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints
van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-137 · Cited by 809 · see citing works
Importance of events per independent variable in proportional hazards analysis I. Background, goals, and general strategy
Concato, J., Peduzzi, P. N., Holford, T. R., & Feinstein, A. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00510-2 · Cited by 797 · see citing works
Sample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K., de Groot, J. D., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 776 · see citing works
A prognostic index in primary breast cancer.
Haybittle, J., Blamey, R., Elston, C., Johnson, J., Doyle, P., Campbell, F., Nicholson, R. I., & Griffiths, K. (1982) · British Journal of Cancer · doi:10.1038/bjc.1982.62 · Cited by 758 · see citing works
A guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J. A., Collins, P. G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 754 · see citing works
How to develop a more accurate risk prediction model when there are few events
Pavlou, M., Ambler, G., Seaman, S. R., Guttmann, O. P., Elliot, P. M., King, M., & Omar, R. (2015) · BMJ · doi:10.1136/bmj.h3868 · Cited by 668 · see citing works
Sample size considerations for the external validation of a multivariable prognostic model: A resampling study
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665 · see citing works
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · see citing works
Self-citation
Substantial effective sample sizes were required for external validation studies of predictive logistic regression models
Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · see citing works
A bootstrap resampling procedure for model building: Application to the cox regression model
Sauerbrei, W., & Schumacher, M. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111607 · Cited by 601 · see citing works
External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges
Riley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
No rationale for 1 variable per 10 events criterion for binary logistic regression analysis
van Smeden, M., de Groot, J. D., Moons, K., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0267-3 · Cited by 563 · see citing works
Validation and updating of predictive logistic regression models: a study on sample size and shrinkage
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Adequate sample size for developing prediction models is not simply related to events per variable
Ogundimu, E. O., Altman, D. G., & Collins, G. S. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.02.031 · Cited by 546 · see citing works
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · see citing works
Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · see citing works
A new measure of prognostic separation in survival data
Royston, P., & Sauerbrei, W. (2004) · Statistics in Medicine · doi:10.1002/sim.1621 · Cited by 450 · see citing works
Clinical risk prediction for pre-eclampsia in nulliparous women: development of model in international prospective cohort
North, R. A., McCowan, L. M. E., Dekker, G., Poston, L., Chan, E. H. Y., Stewart, A. W., Black, M. A., Taylor, R. S., Walker, J. J., Baker, P. N., et al. (2011) · BMJ · doi:10.1136/bmj.d1875 · Cited by 412 · see citing works
Sample-Size Calculations for the Cox Proportional Hazards Regression Model with Nonbinary Covariates
Hsieh, F. Y., & Lavori, P. W. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00104-5 · Cited by 374 · see citing works
Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes
Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Moons, K. G. M., & Collins, P. G. S. (2018) · Statistics in Medicine · doi:10.1002/sim.7993 · Cited by 372 · see citing works
Explained Variation for Survival Models
Royston, P. (2006) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0600600105 · Cited by 319 · see citing works
Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure
Courvoisier, D. S., Combescure, C., Agoritsas, T., Gayet‐Ageron, A., & Perneger, T. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.012 · Cited by 318 · see citing works
Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant.
Kelley, K., & Maxwell, S. E. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.305 · Cited by 310 · see citing works
Prognosis Research in Health Care
(2019) · Oxford University Press eBooks · doi:10.1093/med/9780198796619.001.0001 · Cited by 221 · see citing works
A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data
Wynants, L., Bouwmeester, W., Moons, K. G. M., Moerbeek, M., Timmerman, D., Huffel, S. V., CALSTER, B. V., & Vergouwe, Y. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.02.002 · Cited by 178 · see citing works
An Overview of Variance Inflation Factors for Sample-Size Calculation
Hsieh, F. Y., Lavori, P. W., Cohen, H. J., & Feussner, J. R. (2003) · Evaluation & the Health Professions · doi:10.1177/0163278703255230 · Cited by 173 · see citing works
A closed testing procedure to select an appropriate method for updating prediction models
Vergouwe, Y., Nieboer, D., Oostenbrink, R., Debray, T. P. A., Murray, G. D., Kattan, M. W., Koffijberg, H., Moons, K. G. M., & Steyerberg, E. W. (2016) · Statistics in Medicine · doi:10.1002/sim.7179 · Cited by 159 · see citing works
Sample size planning for the coefficient of variation from the accuracy in parameter estimation approach
Kelley, K. (2007) · Behavior Research Methods · doi:10.3758/bf03192966 · Cited by 152 · see citing works
Explained randomness in proportional hazards models
O’Quigley, J., Xu, R., & Stare, J. (2004) · Statistics in Medicine · doi:10.1002/sim.1946 · Cited by 152 · see citing works
Sample size considerations and predictive performance of multinomial logistic prediction models
de Jong, V. D., Eijkemans, M., van Calster, B., Timmerman, D., Moons, K., Steyerberg, E., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8063 · Cited by 140 · see citing works
Stability Investigations of Multivariable Regression Models Derived from Low- and High-Dimensional Data
Sauerbrei, W., Boulesteix, A., & Binder, H. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2011.629890 · Cited by 134 · see citing works
Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use
Debray, T. P. A., Riley, R. D., Rovers, M. M., Reitsma, J. B., Moons, K. G. M., & Cochrane IPD Meta-analysis Methods group (2015) · PLoS Medicine · doi:10.1371/journal.pmed.1001886 · Cited by 130 · see citing works
Using regression models for prediction: shrinkage and regression to the mean
Copas, J. B. (1997) · Statistical Methods in Medical Research · doi:10.1177/096228029700600206 · Cited by 128 · see citing works
Stability of multivariable fractional polynomial models with selection of variables and transformations: a bootstrap investigation
Royston, P. J., & Sauerbrei, W. (2003) · Statistics in Medicine · doi:10.1002/sim.1310 · Cited by 121 · see citing works
Risk prediction with machine learning and regression methods
Steyerberg, E. W., van der Ploeg, T., & CALSTER, B. V. (2014) · Biometrical Journal · doi:10.1002/bimj.201300297 · Cited by 89 · see citing works
Shrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy
van Houwelingen, H. C. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00154 · Cited by 86 · see citing works
Systematic review of prognostic models for recurrent venous thromboembolism (VTE) post-treatment of first unprovoked VTE
Ensor, J., Riley, R. D., Moore, D., Snell, K. I. E., Bayliss, S., & Fitzmaurice, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011190 · Cited by 84 · see citing works
Increasing statistical power without increasing sample size.
McClelland, G. H. (2000) · American Psychologist · doi:10.1037/0003-066x.55.8.963 · Cited by 81 · see citing works
Meta‐analysis and aggregation of multiple published prediction models
Debray, T. P. A., Koffijberg, H., Nieboer, D., Vergouwe, Y., Steyerberg, E. W., & Moons, K. G. M. (2014) · Statistics in Medicine · doi:10.1002/sim.6080 · Cited by 73 · see citing works
Aggregating published prediction models with individual participant data: a comparison of different approaches
Debray, T. P. A., Koffijberg, H., Vergouwe, Y., Moons, K., & Steyerberg, E. (2012) · Statistics in Medicine · doi:10.1002/sim.5412 · Cited by 66 · see citing works
Development and validation of a prediction model for fat mass in children and adolescents: meta-analysis using individual participant data
Hudda, M. T., Fewtrell, M. S., Haroun, D., Lum, S., Williams, J. E., Wells, J. C. K., Riley, R. D., Owen, C. G., Cook, D. G., Rudnicka, A. R., et al. (2019) · BMJ · doi:10.1136/bmj.l4293 · Cited by 65 · see citing works
Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015
Wessler, B. S., Paulus, J. K., Lundquist, C. M., Ajlan, M., Natto, Z. S., Janes, W., Jethmalani, N., Raman, G., Lutz, J. S., & Kent, D. M. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0021-2 · Cited by 57 · see citing works
Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluation
Ensor, J., Riley, R. D., Jowett, S. M., Monahan, M., Snell, K. I. E., Bayliss, S. E., Moore, D., Fitzmaurice, D. A., & on behalf of the PIT-STOP collaborative group (2016) · Health Technology Assessment · doi:10.3310/hta20120 · Cited by 40 · see citing works
Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data
Jinks, R. C., Royston, P., & Parmar, M. K. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0078-y · Cited by 32 · see citing works
Prognosis research ideally should measure time-varying predictors at their intended moment of use
Whittle, R., Royle, K., Jordan, K. P., Riley, R. D., Mallen, C. D., & Peat, G. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-016-0006-6 · Cited by 27 · see citing works
Incorporating published univariable associations in diagnostic and prognostic modeling
Debray, T. P. A., Koffijberg, H., Lu, D., Vergouwe, Y., Steyerberg, E. W., & Moons, K. G. M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-121 · Cited by 16 · see citing works
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4
Untitled work
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
A simple method of sample size calculation for linear and logistic regression
(1998) · doi:10.1002/(sici)1097-0258(19980730)17:14

Flagged first, then by citations

2 of the 80 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

Riley RD, van der Windt D, Croft P, etal , eds. Prognosis Research in Healthcare: Concepts, Methods and Impact. Oxford University Press, 2019.
McFadden D. Conditional logit analysis of qualitative choice behavior. In: Zarembka P, ed. Frontiers in Econometrics New York. Academic Press, 1974: 104-42.
Schmoor C, Sauerbrei W, Schumacher M. Sample size considerations for the evaluation of prognostic factors in survival analysis. Stat Med 2000;19:441-52. 10

Sorted by citations

TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · see citing works
Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Self-citation
PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069 · see citing works
Self-citation
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score
Knight, S. R., Ho, A. Y. W., Pius, R., Buchan, I. E., Carson, G., Drake, T. M., Dunning, J. W., Fairfield, C. J., Gamble, C. L., Green, C., et al. (2020) · BMJ · doi:10.1136/bmj.m3339 · Cited by 1,020 · see citing works
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
Self-citation
Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R., Archer, L., Snell, K., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L., & Collins, G. S. (2024) · British medical journal · doi:10.1136/bmj-2023-074820 · Cited by 515 · see citing works
Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review
Andaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415 · see citing works
Sample size determination: A practical guide for health researchers
Althubaiti, A. (2022) · Journal of General and Family Medicine · doi:10.1002/jgf2.600 · Cited by 481 · see citing works
TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods
(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 375 · see citing works
Association between periodontal diseases and cardiovascular diseases, diabetes and respiratory diseases: Consensus report of the Joint Workshop by the European Federation of Periodontology ( EFP ) and the European arm of the World Organization of Family Doctors ( WONCA Europe)
Herrera, D., Sanz, M., Shapira, L., Brotons, C., Chapple, I., Frese, T., Graziani, F., Hobbs, R. F., Huck, O., Hummers, E., et al. (2023) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13807 · Cited by 315 · see citing works
Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit
van de Sande, D., van Genderen, M. E., Huiskens, J., Gommers, D., & van Bommel, J. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06446-7 · Cited by 279 · see citing works
A CHecklist for statistical Assessment of Medical Papers (the CHAMP statement): explanation and elaboration
Mansournia, M. A., Collins, P. G. S., Nielsen, R. O., Nazemipour, M., Jewell, N. P., Altman, D. G., & Campbell, M. J. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-103652 · Cited by 268 · see citing works
Clinical prediction models: diagnosis versus prognosis
van Smeden, M., Reitsma, J. B., Riley, R. D., Collins, P. G. S., & Moons, K. G. M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.009 · Cited by 241 · see citing works
Enhanced diagnosis of advanced fibrosis and cirrhosis in individuals with NAFLD using FibroScan-based Agile scores
Sanyal, A. J., Foucquier, J., Younossi, Z. M., Harrison, S. A., Newsome, P. N., Chan, W., Yilmaz, Y. N., de Lédinghen, V., Costentin, C. E., Zheng, M., et al. (2022) · Journal of Hepatology · doi:10.1016/j.jhep.2022.10.034 · Cited by 228 · see citing works
Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study
Riley, R. D., Snell, K. I. E., Archer, L., Ensor, J., Debray, T. P. A., CALSTER, B. V., van Smeden, M., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074821 · Cited by 213 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
Minimum sample size for developing a multivariable prediction model using multinomial logistic regression
Pate, A., Riley, R. D., Collins, P. G. S., van Smeden, M., CALSTER, B. V., Ensor, J., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231151220 · Cited by 172 · see citing works
Generalizable machine learning for stress monitoring from wearable devices: A systematic literature review
Vos, G., Trinh, K., Sarnyai, Z., & Azghadi, M. R. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105026 · Cited by 182 · see citing works
Associations of timing of physical activity with all-cause and cause-specific mortality in a prospective cohort study
Feng, H., Yang, L., Liang, Y. Y., Ai, S., Liu, Y., Liu, Y., Jin, X., Lei, B., Wang, J., Zheng, N., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-36546-5 · Cited by 160 · see citing works
Targeted proteomics improves cardiovascular risk prediction in secondary prevention
Nurmohamed, N. S., Pereira, J. P. B., Hoogeveen, R. M., Kroon, J., Kraaijenhof, J. M., Waissi, F., Timmerman, N., Bom, M. J., Hoefer, I. E., Knaapen, P., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac055 · Cited by 148 · see citing works
Harnessing Big Data Analytics for Healthcare: A Comprehensive Review of Frameworks, Implications, Applications, and Impacts
Ahmed, A., Xi, R., Hou, M., Shah, S. A., & Hameed, S. (2023) · IEEE Access · doi:10.1109/access.2023.3323574 · Cited by 147 · see citing works
Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease
van Smeden, M., Heinze, G., CALSTER, B. V., Asselbergs, F. W., Vardas, P. E., Bruining, N., de Jaegere, P. P. T., Moore, J. H., Denaxas, S., Boulesteix, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac238 · Cited by 147 · see citing works
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149 · see citing works
Developing, implementing, and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter
van de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135 · see citing works
Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140 · see citing works
The future of General Movement Assessment: The role of computer vision and machine learning – A scoping review
Silva, N., Zhang, D., Kulvičius, T., Gail, A., Barreiros, C., Lindstaedt, S., Kraft, M., Bölte, S., Poustka, L., Nielsen‐Saines, K., et al. (2021) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2021.103854 · Cited by 137 · see citing works
Machine learning and artificial intelligence in haematology
Shouval, R., Fein, J. A., Savani, B. N., Mohty, M., & Nagler, A. (2020) · British Journal of Haematology · doi:10.1111/bjh.16915 · Cited by 128 · see citing works
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124 · see citing works
“Alternatively, the level of discrimination between different groups (e.g., the variance of groups) may require a larger number of samples to detect differences 30 .”
External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Targeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 115 · see citing works
Prognostic Factors for 30-Day Mortality in Critically Ill Patients With Coronavirus Disease 2019: An Observational Cohort Study
Ferrando-Vivas, P., Doidge, J., Thomas, K., Gould, D. W., Mouncey, P., Shankar-Hari, M., Young, J. D., Rowan, K. M., & Harrison, D. A. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004740 · Cited by 113 · see citing works
Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records
Tomašev, N., Harris, N., Baur, S., Mottram, A., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Magliulo, V., et al. (2021) · Nature Protocols · doi:10.1038/s41596-021-00513-5 · Cited by 104 · see citing works
Statistical analysis of high-dimensional biomedical data: a gentle introduction to analytical goals, common approaches and challenges
Rahnenführer, J., de Bin, R., Benner, A., Ambrogi, F., Lusa, L., Boulesteix, A., Migliavacca, E., Binder, H., Michiels, S., Sauerbrei, W., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-02858-y · Cited by 103 · see citing works
Cancer symptom experience and help-seeking behaviour during the COVID-19 pandemic in the UK: a cross-sectional population survey
Quinn-Scoggins, H. D., Cannings-John, R. L., Moriarty, Y., Whitelock, V., Whitaker, K. L., Grozeva, D., Hughes, J., Townson, J. K., Osborne, K., Goddard, M., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053095 · Cited by 107 · see citing works
Stress Monitoring Using Wearable Sensors: A Pilot Study and Stress-Predict Dataset
Iqbal, T., Simpkin, A. J., Roshan, D., Glynn, N., Killilea, J., Walsh, J. C., Molloy, G. J., Ganly, S., Ryman, H., Coen, E., et al. (2022) · Sensors · doi:10.3390/s22218135 · Cited by 106 · see citing works
Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review
Dhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, P. G. S. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02008-1 · Cited by 99 · see citing works
Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development
Bradshaw, T., Boellaard, R., Dutta, J., Jha, A. K., Jacobs, P. D., Li, Q., Liu, C., Sitek, A., Saboury, B., Scott, P. J. H., et al. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.121.262567 · Cited by 96 · see citing works
Tackling COVID-19 Through Responsible AI Innovation: Five Steps in the Right Direction
Leslie, D. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4bb9d7a7 · Cited by 95 · see citing works
Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study
Sisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231165001 · Cited by 99 · see citing works
Lung Ultrasound Prediction Model for Acute Respiratory Distress Syndrome: A Multicenter Prospective Observational Study
Smit, M. R., Hagens, L. A., Heijnen, N. F. L., Pisani, L., Cherpanath, T. G. V., Dongelmans, D. A., de Grooth, H., Pierrakos, C. N., Tuinman, P. R., Zimatore, C., et al. (2023) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202210-1882oc · Cited by 92 · see citing works
Development and validation of a prediction model for 30-day mortality in hospitalised patients with COVID-19: the COVID-19 SEIMC score
Berenguer, J., Borobia, A. M., Ryan, P., Rodríguez‐Baño, J., Bellón, J. M., Jarrín, I., Carratalà, J., Pachón, J., Carcas, A. J., Yllescas, M., et al. (2021) · Thorax · doi:10.1136/thoraxjnl-2020-216001 · Cited by 91 · see citing works
Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study
Clift, A. K., Dodwell, D., Lord, S., Petrou, S., Brady, M., Collins, G. S., & Hippisley-Cox, J. (2023) · BMJ · doi:10.1136/bmj-2022-073800 · Cited by 97 · see citing works
Prognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 89 · see citing works
Non-invasive diagnosis of deep vein thrombosis from ultrasound imaging with machine learning
Kainz, B., Heinrich‬, M. P., Makropoulos, A., Oppenheimer, J., Mandegaran, R., Sankar, S., Deane, C., Mischkewitz, S., Al-Noor, F., Rawdin, A. C., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00503-7 · Cited by 86 · see citing works
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93 · see citing works
Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis
Wannan, C., Nelson, B., Addington, J., Allott, K., Anticevic, A., Arango, C., Baker, J. T., Bearden, C. E., Billah, T., Sylvain Bouix, A., et al. (2024) · Schizophrenia Bulletin · doi:10.1093/schbul/sbae011 · Cited by 90 · see citing works
Development and validation of a prediction model for severe respiratory failure in hospitalized patients with SARS-CoV-2 infection: a multicentre cohort study (PREDI-CO study)
Bartoletti, M., Giannella, M., Scudeller, L., Tedeschi, S., Rinaldi, M., Bussini, L., Fornaro, G., Pascale, R., Pancaldi, L., Pasquini, Z. A. I., et al. (2020) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2020.08.003 · Cited by 83 · see citing works
Diagnostic Performance of Electronic Noses in Cancer Diagnoses Using Exhaled Breath
Scheepers, M. H. M. C., Al-Difaie, Z., Brandts, L., Peeters, A., van Grinsven, B., & Bouvy, N. D. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.19372 · Cited by 83 · see citing works
Multiplex immunofluorescence and single‐cell transcriptomic profiling reveal the spatial cell interaction networks in the non‐small cell lung cancer microenvironment
Peng, H., Wu, X., Liu, S., He, M., Xie, C. L., Zhong, R., Liu, J., Tang, C., Li, C., Xiong, S., et al. (2023) · Clinical and Translational Medicine · doi:10.1002/ctm2.1155 · Cited by 87 · see citing works
Dealing with continuous variables and modelling non-linear associations in healthcare data: practical guide
López‐Ayala, P., Riley, R. D., Collins, P. G. S., & Zimmermann, T. (2025) · BMJ · doi:10.1136/bmj-2024-082440 · Cited by 88 · see citing works
Burnout in nurses working in China: A national questionnaire survey
Zhang, W., Miao, R., Tang, J., Su, Q., Aung, L. H. H., Pi, H., & Sai, X. (2020) · International Journal of Nursing Practice · doi:10.1111/ijn.12908 · Cited by 82 · see citing works
Position paper of the EACVI and EANM on artificial intelligence applications in multimodality cardiovascular imaging using SPECT/CT, PET/CT, and cardiac CT
Slart, R. H. J. A., Williams, M. A., Juárez‐Orozco, L. E., Rischpler, C., Dweck, M. R., Glaudemans, A. W. J. M., Gimelli, A., Georgoulias, P., Gheysens, O. S., Gaemperli, O., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05341-z · Cited by 86 · see citing works
Application of machine learning techniques for predicting survival in ovarian cancer
Azar, A. S., Rikan, S. B., Naemi, A., Bagherzadeh, J., Pirnejad, H., Mohasefi, M. B., & Wiil, U. K. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02087-y · Cited by 81 · see citing works
Strength and Biomechanical Risk Factors for Noncontact ACL Injury in Elite Female Footballers: A Prospective Study
Collings, T., Diamond, L. E., BARRETT, R. S., Timmins, R. G., HICKEY, J. T., MOULIN, W. S. D., WILLIAMS, M. D., BEERWORTH, K. A., & Bourne, M. N. (2022) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000002908 · Cited by 80 · see citing works
Prognostic factors associated with mortality among patients receiving venovenous extracorporeal membrane oxygenation for COVID-19: a systematic review and meta-analysis
Tran, A., Fernando, S. M., Rochwerg, B. N., Barbaro, R. P., Hodgson, C. L., Munshi, L., MacLaren, G., Ramanathan, K. R., Hough, C. L., Brochard, L. J., et al. (2022) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(22)00296-x · Cited by 80 · see citing works
Development and validation of a risk factor-based system to predict short-term survival in adult hospitalized patients with COVID-19: a multicenter, retrospective, cohort study
Zhang, S., Guo, M., Duan, L., Wu, F., Hu, G., Wang, Z., Huang, Q., Liao, T., Xu, J., Ma, Y., et al. (2020) · Critical Care · doi:10.1186/s13054-020-03123-x · Cited by 79 · see citing works
A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint
Hoogland, J., IntHout, J., Belias, M., Rovers, M. M., Riley, R. D., Harrell, F. E., Moons, K. G. M., Debray, T. P. A., & Reitsma, J. B. (2021) · Statistics in Medicine · doi:10.1002/sim.9154 · Cited by 79 · see citing works
Poor handling of continuous predictors in clinical prediction models using logistic regression: a systematic review
Ma, J., Dhiman, P., Qi, C., Bullock, G. S., van Smeden, M., Riley, R. D., & Collins, P. G. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.017 · Cited by 78 · see citing works
Just How Confident Can We Be in Predicting Sports Injuries? A Systematic Review of the Methodological Conduct and Performance of Existing Musculoskeletal Injury Prediction Models in Sport
Bullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, P. G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 88 · see citing works
State of the Art of Machine Learning–Enabled Clinical Decision Support in Intensive Care Units: Literature Review
Hong, N., Liu, C., Gao, J., Han, L., Chang, F., Gong, M., & Su, L. (2022) · JMIR Medical Informatics · doi:10.2196/28781 · Cited by 76 · see citing works
Five critical quality criteria for artificial intelligence-based prediction models
van Royen, F. S.-A., Asselbergs, F. W., Alfonso, F., Vardas, P. E., & van Smeden, M. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad727 · Cited by 77 · see citing works
Prediction models for diagnosis and prognosis in Covid-19
Sperrin, M., Grant, S. W., & Peek, N. B. (2020) · BMJ · doi:10.1136/bmj.m1464 · Cited by 74 · see citing works
Appraising prediction research: a guide and meta‐review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)
de Jong, Y., Ramspek, C. L., Zoccali, C., Jager, K. J., Dekker, F. W., & van Diepen, M. (2021) · Nephrology · doi:10.1111/nep.13913 · Cited by 73 · see citing works
Predicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71 · see citing works
Self-citation
Loss of Smell and Taste Can Accurately Predict COVID-19 Infection: A Machine-Learning Approach
Callejón-Leblic, M. A., Moreno‐Luna, R., del Cuvillo, A., Reyes-Tejero, I. M., Garcia-Villaran, M. A., Santos, M. L., Maza-Solano, J. M., Jiménez, D. I. M., Palacios-García, J. M., Fernandez-Velez, C., et al. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10040570 · Cited by 72 · see citing works
Comparison of machine learning and conventional logistic regression-based prediction models for gestational diabetes in an ethnically diverse population; the Monash GDM Machine learning model
Belsti, Y., Moran, L., Du, L., Mousa, A., Silva, K. D., Enticott, J., & Teede, H. J. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105228 · Cited by 76 · see citing works
A note on estimating the Cox‐Snell R 2 from a reported C statistic ( AUROC ) to inform sample size calculations for developing a prediction model with a binary outcome
Riley, R. D., CALSTER, B. V., & Collins, P. G. S. (2020) · Statistics in Medicine · doi:10.1002/sim.8806 · Cited by 70 · see citing works
Risk factors and outcomes associated with recurrent autoimmune hepatitis following liver transplantation
Montaño‐Loza, A. J., Ronca, V., Ebadi, M., Hansen, B. E., Hirschfield, G. M., Elwir, S., Alsaed, M., Milkiewicz, P., Janik, M. K., Marschall, H., et al. (2022) · Journal of Hepatology · doi:10.1016/j.jhep.2022.01.022 · Cited by 69 · see citing works
Fully automated discrimination of Alzheimer’s disease using resting-state electroencephalography signals
Ding, Y., Chu, Y., Liu, M., Ling, Z., Wang, S., Li, X., & Li, Y. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-430 · Cited by 65 · see citing works
Attention-Based Multimodal Fusion with Contrast for Robust Clinical Prediction in the Face of Missing Modalities
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66 · see citing works
Multivariable Prediction Model for Futile Recanalization Therapies in Patients With Acute Ischemic Stroke
Meinel, T. R., Lerch, C., Fischer, U., Beyeler, M., Mujanović, A., Kurmann, C. C., Siepen, B., Scutelnic, A., Müller, M., Goeldlin, M. B., et al. (2022) · Neurology · doi:10.1212/wnl.0000000000200815 · Cited by 66 · see citing works
Development and Validation of a Predictive Model of Severe Fatigue After Breast Cancer Diagnosis: Toward a Personalized Framework in Survivorship Care
Meglio, A. D., Havas, J., Soldato, D., Presti, D., Martin, É., Pistilli, B., Menvielle, G., Dumas, A., Charles, C., Everhard, S., et al. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.21.01252 · Cited by 65 · see citing works
Machine Learning in Laboratory Medicine: Recommendations of the IFCC Working Group
Master, S. R., Badrick, T. C., Bietenbeck, A., & Haymond, S. (2023) · Clinical Chemistry · doi:10.1093/clinchem/hvad055 · Cited by 64 · see citing works
Integrating genome-wide polygenic risk scores and non-genetic risk to predict colorectal cancer diagnosis using UK Biobank data: population based cohort study
Briggs, S., Law, P., East, J. E., Wordsworth, S., Dunlop, M. G., Houlston, R. S., Hippisley‐Cox, J., & Tomlinson, I. P. M. (2022) · BMJ · doi:10.1136/bmj-2022-071707 · Cited by 64 · see citing works
Smartwatch- and smartphone-based remote assessment of brain health and detection of mild cognitive impairment
Butler, P. M., Yang, J. F., Brown, R. Z., Hobbs, M., Becker, A., Penalver-Andres, J. A., Syz, P., Muller, S., Cosne, G., Juraver, A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03475-9 · Cited by 66 · see citing works
Automated machine learning‐based model predicts postoperative delirium using readily extractable perioperative collected electronic data
Hu, X., Liu, H., Zhao, X., Sun, X., Zhou, J. H., Gao, X. H., Guan, H., Zhou, Y., Zhao, Q., Han, Y., et al. (2021) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13758 · Cited by 63 · see citing works
Developing, validating, updating and judging the impact of prognostic models for respiratory diseases
van Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 60 · see citing works
Domain-specific cognitive impairment 6 months after stroke: The value of early cognitive screening
Milosevich, E. T., Moore, M. J., Pendlebury, S. T., & Demeyere, N. (2023) · International Journal of Stroke · doi:10.1177/17474930231205787 · Cited by 62 · see citing works
Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study
Kamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., et al. (2022) · BMJ · doi:10.1136/bmj-2021-068576 · Cited by 60 · see citing works
A model to predict difficult airway alerts after videolaryngoscopy in adults with anticipated difficult airways – the VIDIAC score
Kohse, E. K., Siebert, H. K., Sasu, P. B., Loock, K., Dohrmann, T., Breitfeld, P., Barclay-Steuart, A., Stark, M., Sehner, S., Zöllner, C., et al. (2022) · Anaesthesia · doi:10.1111/anae.15841 · Cited by 60 · see citing works
Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 62 · see citing works
Clinical prediction models and the multiverse of madness
Riley, R. D., Pate, A., Dhiman, P., Archer, L., Martin, G. P., & Collins, P. G. S. (2023) · BMC Medicine · doi:10.1186/s12916-023-03212-y · Cited by 63 · see citing works
Common statistical concepts in the supervised Machine Learning arena
Rashidi, H. H., Albahra, S., Robertson, S., Tran, N. K., & Hu, B. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1130229 · Cited by 59 · see citing works
Development and validation of an international preoperative risk assessment model for postoperative delirium
Dodsworth, B. T., Reeve, K. A., Falco, L., Hueting, T. A., Sadeghirad, B., Mbuagbaw, L. C. E., Goettel, N., & Gelsomino, N. S. (2023) · Age and Ageing · doi:10.1093/ageing/afad086 · Cited by 59 · see citing works
Left ventricular functional recovery after atrial fibrillation catheter ablation in heart failure: a prediction model
Bergonti, M., Ascione, C., Marcon, L., Pambrun, T., Rocca, D. G. D., Ferrero, T. G., Pannone, L., Kühne, M., Compagnucci, P., Bonomi, A., et al. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad428 · Cited by 59 · see citing works
Risk Prediction Models for Cardiotoxicity of Chemotherapy Among Patients With Breast Cancer
Kabore, E., MacDonald, C., Kaboré, A., Didier, R. J., Arveux, P., Méda, N., Boutron‐Ruault, M., & Guénancia, C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.0569 · Cited by 60 · see citing works
Porto-sinusoidal vascular liver disorder with portal hypertension: Natural history and long-term outcome
Magaz, M., Giudicelli, H., Abraldeṣ, J. G., Nicoară-Farcău, O., Turón, F., Rajoriya, N., Goel, A., Raymenants, K., Hillaire, S., Téllez, L., et al. (2024) · Journal of Hepatology · doi:10.1016/j.jhep.2024.07.035 · Cited by 58 · see citing works
Radiomics Models for Predicting Microvascular Invasion in Hepatocellular Carcinoma: A Systematic Review and Radiomics Quality Score Assessment
Wang, Q., Li, C., Zhang, J., Hu, X., Fan, Y., Ma, K., Sparrelid, E., & Brismar, T. B. (2021) · Cancers · doi:10.3390/cancers13225864 · Cited by 57 · see citing works
Not just “big” data: Importance of sample size, measurement error, and uninformative predictors for developing prognostic models for digital interventions
McNamara, M. E., Zisser, M., Beevers, C. G., & Shumake, J. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104086 · Cited by 56 · see citing works
Machine learning-based automated sponge cytology for screening of oesophageal squamous cell carcinoma and adenocarcinoma of the oesophagogastric junction: a nationwide, multicohort, prospective study
Gao, Y., Xin, L., Han, L., Yao, B., Zhang, T., Zhou, A.-J., Huang, S., Wang, J., Feng, Y., Yao, S.-H., et al. (2023) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(23)00004-3 · Cited by 56 · see citing works
Predicting Colorectal Cancer Survival Using Time-to-Event Machine Learning: Retrospective Cohort Study
Yang, X., Qiu, H., Wang, L., & Wang, X. (2023) · Journal of Medical Internet Research · doi:10.2196/44417 · Cited by 56 · see citing works
Beyond Comparing Machine Learning and Logistic Regression in Clinical Prediction Modelling: Shifting from Model Debate to Data Quality
Hu, Y., Zhang, X., Slavin, V., Belsti, Y., Tiruneh, S. A., Callander, E. J., & Enticott, J. (2025) · Journal of Medical Internet Research · doi:10.2196/77721 · Cited by 68 · see citing works
External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19
Barish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58 · see citing works
To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets
Šinkovec, H., Heinze, G., Blagus, R., & Geroldinger, A. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01374-y · Cited by 56 · see citing works

Statistical primer: sample size considerations for developing and validating clinical prediction models
Martin, G. P., Riley, R. D., Ensor, J., & Grant, S. W. (2025) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezaf142 · Cited by 22 · see citing works
Prediction Model Studies are Not Considering Sample Size Requirements to Develop Their Model: A Systematic ReviewPreprint
Dhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, P. G. S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4416958
临床预测模型开发与验证的样本量估算 Sample Size Estimation for the Development and Validation of Clinical Prediction Models
姜英玉，谷鸿秋 (2024) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.3969/j.issn.1673-5765.2024.05.002
Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review
Dhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, P. G. S. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02008-1 · Cited by 99 · see citing works
Additional file 1 of Adaptive sample size determination for the development of clinical prediction models
Christodoulou, E., van Smeden, M., Edlinger, M., Timmerman, D., Wanitschek, M. M., Steyerberg, E. W., & CALSTER, B. V. (2021) · Figshare · doi:10.6084/m9.figshare.14256821

Authors: Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M.
Affiliations: Keele University (GB)
Vanderbilt University (US)
Manchester Academic Health Science Centre (GB)
University Medical Center Utrecht (NL)
Corresponding author: Riley, Richard D
ORCID: Riley, Richard D
Ensor, Joie
Snell, Kym I E
Harrell, Frank E
Martin, Glen P
Reitsma, Johannes B
Moons, Karel G M
Collins, Gary
Venue: BMJ
Publisher: BMJ
Published: 2020-03-18
Volume: 368
Pages: m441
ISSN: 1756-1833 (ISSN-L)
DOI: 10.1136/bmj.m441
OpenAlex: W3012413426
Cited by: 2,917
References: 80
Citations by year: 2026: 863 · 2025: 606 · 2024: 531 · 2023: 402 · 2022: 275 · 2021: 174 · 2020: 58 · 2012: 2
FWCI: 160.59
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
Locations: BMJ (publishedVersion)
PubMed (publishedVersion)
Research Explorer (The University of Manchester) (publishedVersion, other-oa)
Keele Research Repository (Keele University) (submittedVersion, cc-by)
Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
HighWire Press Open Archive (submittedVersion)
University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Funders: Georgia Clinical and Translational Science Alliance (UL1TR002243)
Cancer Research UK (27294)
National Institute for Health and Care Research (SPCR-142)
Department of Health and Social Care
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (91208004, 918.10.615, project 9120.8004)
National Institutes of Health (UL1TR002243)
NIHR School for Primary Care Research
National Center for Advancing Translational Sciences (TR002243)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords: sample size determination, clinical prediction model, model development, prediction accuracy, prognostic model, diagnostic model, small sample size
MeSH terms: Clinical Decision-Making, Forecasting, Humans, Models, Theoretical, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Calculating the sample size required for developing a clinical prediction model

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks