The DOI matches its Crossref record; 78 of the 80 references it cites were found in a registry or an index.
Calculating the sample size required for developing a clinical prediction model
Open accessAbstract
Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In step three, the sample size needed to reduce model overfitting is calculated, assuming an expected shrinkage rate of 10%, with an expected number of variables P to be included in the final predictive model of 30: S is the expected shrinkage factor, and the noise ratio evaluation index R2cs is 0.3; the calculated sample size is 740 cases.”
From Machine learning-driven prediction model for successful weaning of patients from mechanical ventilation in ICU (2026) · cites this paper as Riley [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Genomic and molecular data support integration of tumour biology and genetic risk into decision-making, particularly in precision medicine applications [42].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [42]
Experimental — accuracy not yet measuredIn the citing paper
“The sample size calculated this way is the minimum for a standard statistical model.”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [50]
In the citing paper
“Sample size calculations50 indicated a minimum sample of 2082 patients, which is larger than the available sample, raising concerns about possible overfitting issues (step 6).”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationRelaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
Vittinghoff, E., & McCulloch, C. E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwk052 · Cited by 4,359
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
Self-citation
Research works printed but not linked: 2
- 10.1177/962280218784726 · DOI
- McFadden, 1974, Frontiers in Econometrics New York., 104
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citationMoons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
Self-citationKnight, S. R., Ho, A. Y. W., Pius, R., Buchan, I. E., Carson, G., Drake, T. M., Dunning, J. W., Fairfield, C. J., Gamble, C. L., Green, C., et al. (2020) · BMJ · doi:10.1136/bmj.m3339 · Cited by 1,039
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Self-citationde Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
“For regression-based methods 48 , and a selection of machine-learning-based methods 47 , technique-speci fi c a priori sample size calculations are available, although for many model architectures and settings (e.g., semi-supervised learning, decision trees, or convolutional neural networks) no…”
BackgroundMethodDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
“Calculating sample size requirements for a specific model Riley and colleagues 50 provide helpful guidance and code 51 52 on sample size calculations.”
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
“38 Another overall measure of fit is the mean squared error of predictions, which for continuous outcomes can be obtained on external validation by calculating the mean of the squared difference between”
BackgroundAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415
Sample size determination: A practical guide for health researchers
Althubaiti, A. (2022) · Journal of general and family medicine · doi:10.1002/jgf2.600 · Cited by 437
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 427
Herrera, D., Sanz, M., Shapira, L., Brotons, C., Chapple, I., Frese, T., Graziani, F., Hobbs, R. F., Huck, O., Hummers, E., et al. (2023) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13807 · Cited by 315
van de Sande, D., van Genderen, M. E., Huiskens, J., Gommers, D., & van Bommel, J. (2021) · Intensive Care Medicine · doi:10.1007/s00134-021-06446-7 · Cited by 279
Mansournia, M. A., Collins, P. G. S., Nielsen, R. O., Nazemipour, M., Jewell, N. P., Altman, D. G., & Campbell, M. J. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-103652 · Cited by 268
Clinical prediction models: diagnosis versus prognosis
van Smeden, M., Reitsma, J. B., Riley, R. D., Collins, P. G. S., & Moons, K. G. M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.009 · Cited by 249
Sanyal, A. J., Foucquier, J., Younossi, Z. M., Harrison, S. A., Newsome, P. N., Chan, W., Yilmaz, Y. N., de Lédinghen, V., Costentin, C. E., Zheng, M., et al. (2022) · Journal of Hepatology · doi:10.1016/j.jhep.2022.10.034 · Cited by 228
Riley, R. D., Snell, K. I. E., Archer, L., Ensor, J., Debray, T. P. A., CALSTER, B. V., van Smeden, M., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074821 · Cited by 213
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211
“…a Cox–Snell R 2 of 0.08 and targeting a uniform shrinkage factor of 0.9, leading to 752 participants (53 deaths) corresponding to an EPP of 7.5 (Riley et al., 2019a, 2020) We develop two models in a random sample size of 752 participants, using: a. Logistic regression with a LASSO penalty,…”
BackgroundMethodInfluentialPate, A., Riley, R. D., Collins, P. G. S., van Smeden, M., CALSTER, B. V., Ensor, J., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231151220 · Cited by 172
“Plugging this into equation (14), and noting that n is a monotonically increasing function of SVH MN , we get the following requirement for criterion (ii) for multinomial logistic regression prediction models:”
BackgroundVos, G., Trinh, K., Sarnyai, Z., & Azghadi, M. R. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105026 · Cited by 200
Feng, H., Yang, L., Liang, Y. Y., Ai, S., Liu, Y., Liu, Y., Jin, X., Lei, B., Wang, J., Zheng, N., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-36546-5 · Cited by 166
Targeted proteomics improves cardiovascular risk prediction in secondary prevention
Nurmohamed, N. S., Pereira, J. P. B., Hoogeveen, R. M., Kroon, J., Kraaijenhof, J. M., Waissi, F., Timmerman, N., Bom, M. J., Hoefer, I. E., Knaapen, P., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac055 · Cited by 147
Ahmed, A., Xi, R., Hou, M., Shah, S. A., & Hameed, S. (2023) · IEEE Access · doi:10.1109/access.2023.3323574 · Cited by 164
“Riley et al. [98] addressed in their study to propose a new methodology for sample size calculations for new experiments.”
MethodRiley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142
Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease
van Smeden, M., Heinze, G., CALSTER, B. V., Asselbergs, F. W., Vardas, P. E., Bruining, N., de Jaegere, P. P. T., Moore, J. H., Denaxas, S., Boulesteix, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac238 · Cited by 155
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149
“There were instances of potential spin [11, 40], with relatively low AUC values under 0.”
BackgroundMethodvan de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
“Use of smaller datasets for non-regression and ensemble machine learning models is problematic and increases their risk of overfitting further due to increased flexibility and categorisation of prediction inherent to many machine learning methods [116, 117].”
BackgroundSilva, N., Zhang, D., Kulvičius, T., Gail, A., Barreiros, C., Lindstaedt, S., Kraft, M., Bölte, S., Poustka, L., Nielsen‐Saines, K., et al. (2021) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2021.103854 · Cited by 137
“If only a small dataset is available, data splitting will become challenging and additional strategies will be required (e.g., Beleites, Neugebauer, Bocklitz, Krafft, & Popp, 2013; Riley et al., 2020; Shahinfar, Meek, & Falzon, 2020).”
Machine learning and artificial intelligence in haematology
Shouval, R., Fein, J. A., Savani, B. N., Mohty, M., & Nagler, A. (2020) · British Journal of Haematology · doi:10.1111/bjh.16915 · Cited by 133
“Generally, for model development, tuning and internal validation, it is preferred to use all available data with resampling methods, such as cross-validation rather than data splitting.(50) Resampling methods maximize data use and help avoid overfitting.”
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124
“Alternatively, the level of discrimination between different groups (e.g., the variance of groups) may require a larger number of samples to detect differences 30 .”
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125
“In contrast to model development studies [7-10], relatively little research has been published on the sample size needed to externally validate a prediction model.”
ResultTargeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 121
“This internal validation is likely to give a robust estimate of the model’s performance when appropriate steps were taken during the model development to ensure the study has adequate sample size [21], that overfitting is minimised [22], that in-sample optimism is estimated precisely and corrected for [8], and that th…”
BackgroundFerrando-Vivas, P., Doidge, J., Thomas, K., Gould, D. W., Mouncey, P., Shankar-Hari, M., Young, J. D., Rowan, K. M., & Harrison, D. A. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000004740 · Cited by 113
Tomašev, N., Harris, N., Baur, S., Mottram, A., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Magliulo, V., et al. (2021) · Nature Protocols · doi:10.1038/s41596-021-00513-5 · Cited by 109
“The actual dataset size required for each speci fi c application is dependent on many factors, including the patient cohort, input feature space present and the endpoint of interest; for further details on power calculations for clinical prediction models, we refer the reader to Riley et al. 17 .”
MethodRahnenführer, J., Bin, R. D., Benner, A., Ambrogi, F., Lusa, L., Boulesteix, A., Migliavacca, E., Binder, H., Michiels, S., Sauerbrei, W., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-02858-y · Cited by 132
Quinn-Scoggins, H. D., Cannings-John, R. L., Moriarty, Y., Whitelock, V., Whitaker, K. L., Grozeva, D., Hughes, J., Townson, J. K., Osborne, K., Goddard, M., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053095 · Cited by 107
Stress Monitoring Using Wearable Sensors: A Pilot Study and Stress-Predict Dataset
Iqbal, T., Simpkin, A. J., Roshan, D., Glynn, N., Killilea, J., Walsh, J. C., Molloy, G. J., Ganly, S., Ryman, H., Coen, E., et al. (2022) · Sensors · doi:10.3390/s22218135 · Cited by 106
Dhiman, P., Ma, J., Qi, C., Bullock, G. S., Sergeant, J. C., Riley, R. D., & Collins, P. G. S. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02008-1 · Cited by 111
“In the absence of a reported R 2 and c-statistic, we used a conservative value taken as 15% of the maximum R 2 value to derive the minimum required sample size, as recommended by Riley et al. [14].”
BackgroundMethodInfluentialNuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development
Bradshaw, T., Boellaard, R., Dutta, J., Jha, A. K., Jacobs, P. D., Li, Q., Liu, C., Sitek, A., Saboury, B., Scott, P. J. H., et al. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.121.262567 · Cited by 103
“No formal guidelines exist for estimating the size of the training set, although some practical approaches have been described (10), and trial and error are therefore often necessary (11).”
BackgroundTackling COVID-19 Through Responsible AI Innovation: Five Steps in the Right Direction
Leslie, D. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4bb9d7a7 · Cited by 103
Sisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231165001 · Cited by 99
“As the number of incomplete predictors increases, so does the number of potential missing indicators eligible for inclusion in the outcome model, which could introduce issues of overfitting, 40,41 and variable selection becomes challenging.”
BackgroundSmit, M. R., Hagens, L. A., Heijnen, N. F. L., Pisani, L., Cherpanath, T. G. V., Dongelmans, D. A., de Grooth, H., Pierrakos, C. N., Tuinman, P. R., Zimatore, C., et al. (2023) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202210-1882oc · Cited by 94
Berenguer, J., Borobia, A. M., Ryan, P., Rodríguez‐Baño, J., Bellón, J. M., Jarrín, I., Carratalà, J., Pachón, J., Carcas, A. J., Yllescas, M., et al. (2021) · Thorax · doi:10.1136/thoraxjnl-2020-216001 · Cited by 90
Clift, A. K., Dodwell, D., Lord, S., Petrou, S., Brady, M., Collins, G. S., & Hippisley-Cox, J. (2023) · BMJ · doi:10.1136/bmj-2022-073800 · Cited by 97
Prognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 92
Non-Invasive Diagnosis of Deep Vein Thrombosis From Ultrasound With Machine Learning
Kainz, B., Heinrich, M. P., Makropoulos, A., Oppenheimer, J., Mandegaran, R., Sankar, S., Deane, C., Mischkewitz, S., Al-Noor, F., Rawdin, A., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-021-00503-7 · Cited by 86
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93
Wannan, C., Nelson, B., Addington, J., Allott, K., Anticevic, A., Arango, C., Baker, J. T., Bearden, C. E., Billah, T., Sylvain Bouix, A., et al. (2024) · Schizophrenia Bulletin · doi:10.1093/schbul/sbae011 · Cited by 96
“We have used a recently described method 80 for developing a clinical prediction model using a traditional likelihood-based approach (eg, logistic regression) to estimate an appropriate sample size.”
MethodBartoletti, M., Giannella, M., Scudeller, L., Tedeschi, S., Rinaldi, M., Bussini, L., Fornaro, G., Pascale, R., Pancaldi, L., Pasquini, Z. A. I., et al. (2020) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2020.08.003 · Cited by 83
Diagnostic Performance of Electronic Noses in Cancer Diagnoses Using Exhaled Breath
Scheepers, M. H. M. C., Al-Difaie, Z., Brandts, L., Peeters, A., van Grinsven, B., & Bouvy, N. D. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.19372 · Cited by 83
Peng, H., Wu, X., Liu, S., He, M., Xie, C. L., Zhong, R., Liu, J., Tang, C., Li, C., Xiong, S., et al. (2023) · Clinical and Translational Medicine · doi:10.1002/ctm2.1155 · Cited by 89
López‐Ayala, P., Riley, R. D., Collins, P. G. S., & Zimmermann, T. (2025) · BMJ · doi:10.1136/bmj-2024-082440 · Cited by 99
“21”
Burnout in nurses working in China: A national questionnaire survey
Zhang, W., Miao, R., Tang, J., Su, Q., Aung, L. H. H., Pi, H., & Sai, X. (2020) · International Journal of Nursing Practice · doi:10.1111/ijn.12908 · Cited by 82
Slart, R. H. J. A., Williams, M. A., Juárez‐Orozco, L. E., Rischpler, C., Dweck, M. R., Glaudemans, A. W. J. M., Gimelli, A., Georgoulias, P., Gheysens, O. S., Gaemperli, O., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05341-z · Cited by 88
“Sample size calculations for ML analysis, and DL in particular, can be challenging but the number of available events should be considered in the generation of prediction models [91].”
BackgroundApplication of machine learning techniques for predicting survival in ovarian cancer
Azar, A. S., Rikan, S. B., Naemi, A., Bagherzadeh, J., Pirnejad, H., Mohasefi, M. B., & Wiil, U. K. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02087-y · Cited by 90
“Therefore, we were able to get accurate outcomes with relatively few errors in the results [54].”
ResultCollings, T., Diamond, L. E., BARRETT, R. S., Timmins, R. G., HICKEY, J. T., MOULIN, W. S. D., WILLIAMS, M. D., BEERWORTH, K. A., & Bourne, M. N. (2022) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000002908 · Cited by 80
Tran, A., Fernando, S. M., Rochwerg, B. N., Barbaro, R. P., Hodgson, C. L., Munshi, L., MacLaren, G., Ramanathan, K. R., Hough, C. L., Brochard, L. J., et al. (2022) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(22)00296-x · Cited by 82
Zhang, S., Guo, M., Duan, L., Wu, F., Hu, G., Wang, Z., Huang, Q., Liao, T., Xu, J., Ma, Y., et al. (2020) · Critical Care · doi:10.1186/s13054-020-03123-x · Cited by 79
“To avoid overfitting in our model, we calculated the numbers of variables allowed to enroll in our multivariable Cox regression model based on a previous study for guidance on sample size requirements for prediction models [17].”
MethodHoogland, J., IntHout, J., Belias, M., Rovers, M. M., Riley, R. D., Harrell, F. E., Moons, K. G. M., Debray, T. P. A., & Reitsma, J. B. (2021) · Statistics in Medicine · doi:10.1002/sim.9154 · Cited by 79
“Existing guidelines on clinical prediction modeling provide a lower bound for the sample size needed in case of individualized treatment effect prediction [51, 22, 21].”
BackgroundMethodInfluentialMa, J., Dhiman, P., Qi, C., Bullock, G. S., van Smeden, M., Riley, R. D., & Collins, P. G. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.017 · Cited by 84
“While new sample size guidance is now available [13], it is highly unlikely that these studies would have met the new criteria.”
Bullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, P. G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 92
“Sample size guidance for developing clinical prediction models using regression have recently been proposed, and one model in the review followed these principles [63, 66].”
Hong, N., Liu, C., Gao, J., Han, L., Chang, F., Gong, M., & Su, L. (2022) · JMIR Medical Informatics · doi:10.2196/28781 · Cited by 76
Five critical quality criteria for artificial intelligence-based prediction models
van Royen, F. S.-A., Asselbergs, F. W., Alfonso, F., Vardas, P. E., & van Smeden, M. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad727 · Cited by 77
“26,27 However, due to the higher complexity of AI-based prediction models, the minimal required sample size may often be (much) larger, sometimes requiring data on multiple thousands of individuals, especially if the predicted outcome is rare (i.e. lower incidence or prevalence of the outcome to be…”
BackgroundPrediction models for diagnosis and prognosis in Covid-19
Sperrin, M., Grant, S. W., & Peek, N. B. (2020) · BMJ · doi:10.1136/bmj.m1464 · Cited by 74
de Jong, Y., Ramspek, C. L., Zoccali, C., Jager, K. J., Dekker, F. W., & van Diepen, M. (2021) · Nephrology · doi:10.1111/nep.13913 · Cited by 74
“New and more accurate methods for sample size calculation in prediction model development studies have been proposed recently.(11) Finally, shrinkage of the model's coefficients, or internal validation (i.”
MethodPredicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71
“…and ‘ good ’ Backs-Dermott et al (2010) 32 Berlanga et al (1999) 33 Johansson et al (2015) 34 Judd et al (2016) 35 Klein et al (2018) 29 Mocking et al (2021) 36 Pintor et al (2009) 38 Ruhe et al (2019) 37 Van Loo et al (2015) 30 Van Loo et al (2018) 26 Van Loo et al (2020) 25 Wang et al 31”
Self-citationLoss of Smell and Taste Can Accurately Predict COVID-19 Infection: A Machine-Learning Approach
Callejón-Leblic, M. A., Moreno‐Luna, R., del Cuvillo, A., Reyes-Tejero, I. M., Garcia-Villaran, M. A., Santos, M. L., Maza-Solano, J. M., Jiménez, D. I. M., Palacios-García, J. M., Fernandez-Velez, C., et al. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10040570 · Cited by 75
Belsti, Y., Moran, L., Du, L., Mousa, A., Silva, K. D., Enticott, J., & Teede, H. J. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105228 · Cited by 78
Riley, R. D., CALSTER, B. V., & Collins, P. G. S. (2020) · Statistics in Medicine · doi:10.1002/sim.8806 · Cited by 71
Montaño‐Loza, A. J., Ronca, V., Ebadi, M., Hansen, B. E., Hirschfield, G. M., Elwir, S., Alsaed, M., Milkiewicz, P., Janik, M. K., Marschall, H., et al. (2022) · Journal of Hepatology · doi:10.1016/j.jhep.2022.01.022 · Cited by 69
“Although pre-screening of variables based on univariable p values is a commonly accepted approach and has been adopted in our large multicenter study, this strategy might lead to overestimation and risks for optimism bias.(40) Variables with p value equal or less than 0.”
MethodDing, Y., Chu, Y., Liu, M., Ling, Z., Wang, S., Li, X., & Li, Y. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-430 · Cited by 65
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66
Meinel, T. R., Lerch, C., Fischer, U., Beyeler, M., Mujanović, A., Kurmann, C. C., Siepen, B., Scutelnic, A., Müller, M., Goeldlin, M. B., et al. (2022) · Neurology · doi:10.1212/wnl.0000000000200815 · Cited by 66
“25.(32) This resulted in a final sample size of 1,230 patients.”
BackgroundMeglio, A. D., Havas, J., Soldato, D., Presti, D., Martin, É., Pistilli, B., Menvielle, G., Dumas, A., Charles, C., Everhard, S., et al. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.21.01252 · Cited by 65
Machine Learning in Laboratory Medicine: Recommendations of the IFCC Working Group
Master, S. R., Badrick, T. C., Bietenbeck, A., & Haymond, S. (2023) · Clinical Chemistry · doi:10.1093/clinchem/hvad055 · Cited by 64
Briggs, S., Law, P., East, J. E., Wordsworth, S., Dunlop, M. G., Houlston, R. S., Hippisley‐Cox, J., & Tomlinson, I. P. M. (2022) · BMJ · doi:10.1136/bmj-2022-071707 · Cited by 72
Butler, P. M., Yang, J. F., Brown, R. Z., Hobbs, M., Becker, A., Penalver-Andres, J. A., Syz, P., Muller, S., Cosne, G., Juraver, A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03475-9 · Cited by 66
“With a conservative assumed model AUROC of 0.7, the number of MCI (or MCI converter for the prognostic model) to be included in the study was n = 360 based on Hanley, and n = 564 based on Riley.”
InfluentialHu, X., Liu, H., Zhao, X., Sun, X., Zhou, J. H., Gao, X. H., Guan, H., Zhou, Y., Zhao, Q., Han, Y., et al. (2021) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13758 · Cited by 63
van Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 62
Domain-specific cognitive impairment 6 months after stroke: The value of early cognitive screening
Milosevich, E. T., Moore, M. J., Pendlebury, S. T., & Demeyere, N. (2023) · International Journal of Stroke · doi:10.1177/17474930231205787 · Cited by 62
“The general rule of at least 10 events required per candidate predictor parameter (EPP) was applied.(21) Demographic covariates included age, sex, and years of education.”
BackgroundKamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., et al. (2022) · BMJ · doi:10.1136/bmj-2021-068576 · Cited by 60
Kohse, E. K., Siebert, H. K., Sasu, P. B., Loock, K., Dohrmann, T., Breitfeld, P., Barclay-Steuart, A., Stark, M., Sehner, S., Zöllner, C., et al. (2022) · Anaesthesia · doi:10.1111/anae.15841 · Cited by 61
“5 and 16 candidate predictors to be appropriate [27].”
BackgroundDebray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 64
Clinical prediction models and the multiverse of madness
Riley, R. D., Pate, A., Dhiman, P., Archer, L., Martin, G. P., & Collins, P. G. S. (2023) · BMC Medicine · doi:10.1186/s12916-023-03212-y · Cited by 68
Common statistical concepts in the supervised Machine Learning arena
Rashidi, H. H., Albahra, S., Robertson, S., Tran, N. K., & Hu, B. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1130229 · Cited by 72
“However, for regression-based supervised models, some recent studies developed guidelines and statistical formulas for calculating the sample sizes based on speci fi c prediction metrics through internal validation (28, 29, 31).”
MethodDodsworth, B. T., Reeve, K. A., Falco, L., Hueting, T. A., Sadeghirad, B., Mbuagbaw, L. C. E., Goettel, N., & Gelsomino, N. S. (2023) · Age and Ageing · doi:10.1093/ageing/afad086 · Cited by 64
“The performance was similar across algorithms and logistic regression was deemed the most suitable algorithm considering the amount of data available [23], the final model was built with logistic regression with an L1 penalty (LASSO).”
MethodBergonti, M., Ascione, C., Marcon, L., Pambrun, T., Rocca, D. G. D., Ferrero, T. G., Pannone, L., Kühne, M., Compagnucci, P., Bonomi, A., et al. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad428 · Cited by 61
Risk Prediction Models for Cardiotoxicity of Chemotherapy Among Patients With Breast Cancer
Kabore, E., MacDonald, C., Kaboré, A., Didier, R. J., Arveux, P., Méda, N., Boutron‐Ruault, M., & Guénancia, C. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.0569 · Cited by 63
Magaz, M., Giudicelli, H., Abraldeṣ, J. G., Nicoară-Farcău, O., Turón, F., Rajoriya, N., Goel, A., Raymenants, K., Hillaire, S., Téllez, L., et al. (2024) · Journal of Hepatology · doi:10.1016/j.jhep.2024.07.035 · Cited by 58
Wang, Q., Li, C., Zhang, J., Hu, X., Fan, Y., Ma, K., Sparrelid, E., & Brismar, T. B. (2021) · Cancers · doi:10.3390/cancers13225864 · Cited by 59
“However, one of them validated their model in only 18 patients, which is not a sufficiently large validation cohort according to the “10-EPV” principle (at least 10 events per variable) [16,46].”
McNamara, M. E., Zisser, M., Beevers, C. G., & Shumake, J. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104086 · Cited by 55
Gao, Y., Xin, L., Han, L., Yao, B., Zhang, T., Zhou, A.-J., Huang, S., Wang, J., Feng, Y., Yao, S.-H., et al. (2023) · The Lancet. Gastroenterology & hepatology · doi:10.1016/s2468-1253(23)00004-3 · Cited by 60
Yang, X., Qiu, H., Wang, L., & Wang, X. (2023) · Journal of Medical Internet Research · doi:10.2196/44417 · Cited by 65
“A sample size calculation was performed using the “pmsampsize” [32] package in R for the CPH model, and the total required sample size was 555 patients.”
BackgroundMethodHu, Y., Zhang, X., Slavin, V., Belsti, Y., Tiruneh, S. A., Callander, E. J., & Enticott, J. (2025) · Journal of Medical Internet Research · doi:10.2196/77721 · Cited by 70
“Adherence to the minimum sample size recommendations is one way to mitigate this issue [9,10].”
Barish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58
To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets
Šinkovec, H., Heinze, G., Blagus, R., & Geroldinger, A. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01374-y · Cited by 64
“However, one should not expect that penalization can overcome the problem of insufficient sample sizes when developing prediction models [9].”
Background
Metadata
- Authors
- Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M.
- Affiliations
- Keele University (GB)
- Vanderbilt University (US)
- Manchester Academic Health Science Centre (GB)
- University Medical Center Utrecht (NL)
- Corresponding author
- Riley, Richard D
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2020-03-18
- Volume
- 368
- Pages
- m441
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W3012413426
- Cited by
- 2,917
- References
- 80
- Citations by year
- 2026: 863 · 2025: 606 · 2024: 531 · 2023: 402 · 2022: 275 · 2021: 174 · 2020: 58
- FWCI
- 160.59
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Georgia Clinical and Translational Science Alliance (UL1TR002243)
- Cancer Research UK (27294)
- National Institute for Health and Care Research (SPCR-142)
- Department of Health and Social Care
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (91208004, 918.10.615, project 9120.8004)
- National Institutes of Health (UL1TR002243)
- NIHR School for Primary Care Research
- National Center for Advancing Translational Sciences (TR002243)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- sample size determination, clinical prediction model, model development, prediction accuracy, prognostic model, diagnostic model, small sample size
- MeSH terms
- Clinical Decision-Making, Forecasting, Humans, Models, Theoretical, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
