Stability of clinical prediction models developed using statistical or machine learning methods
Open accessAbstract
Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Calibration is an active area of research focusing on aspects such as strong calibration, quantifying the degree of miscalibration, and uncertainty.
“For example, calibration is an active area of research focusing on aspects such as strong calibration, quantifying the degree of miscalibration, and uncertainty.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [115]
Riley and Collins described how to assess the stability of clinical prediction models during the model development phase using a bootstrap approach.
“Riley and Collins described how to assess the stability of clinical prediction models during the model development phase using a bootstrap approach.94”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [94]
Some machine learning approaches might require additional development steps to ensure calibration.
“Moreover, some machine learning approaches might require additional development steps to ensure calibration.94 121 122”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Robust Locally Weighted Regression and Smoothing Scatterplots
Cleveland, W. S. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481038 · Cited by 11,594 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
The GUSTO Investigators (1993) · New England Journal of Medicine · doi:10.1056/nejm199309023291001 · Cited by 4,042 · see citing works
European system for cardiac operative risk evaluation (EuroSCORE)
Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/s1010-7940(99)00134-7 · Cited by 3,362 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984 · see citing works
Self-citationNashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs043 · Cited by 2,971 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1× · see citing works
“However, if a model exhibits instability in predictions then any meaningful attempt to “explain” the model, especially at the individual level, is futile (Ghassemi et al., 2021).”
BackgroundVariable selection – A review and recommendations for the practicing statistician
Heinze, G., Wallisch, C., & Dunkler, D. (2018) · Biometrical Journal · doi:10.1002/bimj.201700067 · Cited by 1,673 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationThe random forest algorithm for statistical learning
Schonlau, M., & Zou, R. Y. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20909688 · Cited by 1,418 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Self-citationVickers, A. J., Cronin, A. M., Elkin, E. B., & Gönen, M. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-53 · Cited by 1,377 · see citing works
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156 · see citing works
Heuristics of instability and stabilization in model selection
Breiman, L. (1996) · The Annals of Statistics · doi:10.1214/aos/1032181158 · Cited by 1,144 · see citing works
The Nottingham prognostic index in primary breast cancer
Galea, M. H., Blamey, R. W., Elston, C. W., & Ellis, I. O. (1992) · Breast Cancer Research and Treatment · doi:10.1007/bf01840834 · Cited by 1,071 · see citing works
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Collins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852 · see citing works
Self-citationReilly, B. M., & Evans, A. T. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-3-200602070-00009 · Cited by 794 · see citing works
Sample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K., de Groot, J. D., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 776 · see citing works
Self-citationA prognostic index in primary breast cancer.
Haybittle, J., Blamey, R., Elston, C., Johnson, J., Doyle, P., Campbell, F., Nicholson, R. I., & Griffiths, K. (1982) · British Journal of Cancer · doi:10.1038/bjc.1982.62 · Cited by 758 · see citing works
Predictive value of statistical models.
Van Houwelingen, J. C., & Le Cessie, S. (1990) · Statistics in Medicine · doi:10.1002/sim.4780091109 · Cited by 621 · see citing works
Andaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415 · see citing works
Self-citationMinimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Self-citationMinimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes
Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Moons, K. G. M., & Collins, P. G. S. (2018) · Statistics in Medicine · doi:10.1002/sim.7993 · Cited by 372 · see citing works
Self-citationBootstrap investigation of the stability of a cox regression model
Altman, D. G., & Andersen, P. K. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080702 · Cited by 311 · see citing works
Prognosis Research in Health Care
(2019) · Oxford University Press eBooks · doi:10.1093/med/9780198796619.001.0001 · Cited by 221 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Self-citationRiley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142 · see citing works
Self-citationDhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140 · see citing works
Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision Making
Localio, A. R., & Goodman, S. N. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-4-201208210-00014 · Cited by 139 · see citing works
Sauerbrei, W., Boulesteix, A., & Binder, H. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2011.629890 · Cited by 134 · see citing works
Royston, P. J., & Sauerbrei, W. (2003) · Statistics in Medicine · doi:10.1002/sim.1310 · Cited by 121 · see citing works
Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120 · see citing works
Riley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116 · see citing works
Self-citationSubsampling Versus Bootstrapping in Resampling-Based Model Selection for Multivariable Regression
Bin, R. D., Janitza, S., Sauerbrei, W., et al. (2015) · Biometrics · doi:10.1111/biom.12381 · Cited by 105 · see citing works
Strategies for comparing treatments on a binary response with multi-centre data
Agresti, A., & Hartzel, J. S. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8 · Cited by 102 · see citing works
Stabilizing the lasso against cross-validation variability
Roberts, S., & Nowak, G. (2013) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2013.09.008 · Cited by 90 · see citing works
Shrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy
van Houwelingen, H. C. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00154 · Cited by 86 · see citing works
Ensor, J., Riley, R. D., Moore, D., Snell, K. I. E., Bayliss, S., & Fitzmaurice, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011190 · Cited by 84 · see citing works
Self-citationBootstrap Assessment of the Stability of Multivariable Models
Royston, P. J., & Sauerbrei, W. (2009) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900403 · Cited by 70 · see citing works
Riley, R. D., CALSTER, B. V., & Collins, P. G. S. (2020) · Statistics in Medicine · doi:10.1002/sim.8806 · Cited by 70 · see citing works
Self-citationCalibrating random forests for probability estimation
Dankowski, T., & Ziegler, A. (2016) · Statistics in Medicine · doi:10.1002/sim.6959 · Cited by 65 · see citing works
On stability issues in deriving multivariable regression models
Sauerbrei, W., Buchholz, A., Boulesteix, A., & Binder, H. (2014) · Biometrical Journal · doi:10.1002/bimj.201300222 · Cited by 63 · see citing works
To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets
Šinkovec, H., Heinze, G., Blagus, R., & Geroldinger, A. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01374-y · Cited by 56 · see citing works
Wallisch, C., Dunkler, D., Rauch, G., et al. (2020) · Statistics in Medicine · doi:10.1002/sim.8779 · Cited by 52 · see citing works
Cross-Validation, Shrinkage and Variable Selection in Linear Regression Revisited
van Houwelingen, H. C., & Sauerbrei, W. (2013) · Open Journal of Statistics · doi:10.4236/ojs.2013.32011 · Cited by 50 · see citing works
Martin, G. P., Riley, R. D., Collins, P. G. S., & Sperrin, M. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211046388 · Cited by 40 · see citing works
Self-citationLeeuwenberg, A. M., van Smeden, M., Langendijk, J. A., van der Schaaf, A., Mauer, M. E. L., Moons, K. G. M., Reitsma, J. B., & Schuit, E. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-021-00115-5 · Cited by 39 · see citing works
Pate, A., Emsley, R., Sperrin, M., Martin, G. P., & van Staa, T. P. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00082-3 · Cited by 36 · see citing works
On Algorithmic Fairness in Medical Practice
Grote, T., & Keeling, G. (2022) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s0963180121000839 · Cited by 32 · see citing works
Janitza, S., Binder, H., & Boulesteix, A. (2015) · Biometrical Journal · doi:10.1002/bimj.201400246 · Cited by 29 · see citing works
On the choice and influence of the number of boosting steps for high-dimensional linear Cox-models
Seibold, H., Bernau, C., Boulesteix, A., et al. (2016) · Computational statistics (Zeitschrift) · doi:10.1007/s00180-017-0773-8 · Cited by 29 · see citing works
Hennig, C., & Sauerbrei, W. (2019) · Advances in Data Analysis and Classification · doi:10.1007/s11634-018-00351-6 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Platt, J. (2000). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers. Cambridge: MIT Press.
- Riley, R. D., van der Windt, D., Croft, P., & Moons, K. G. M. (Eds.). (2019). Prognosis research in healthcare: Concepts, methods and impact. Oxford University Press.
Cited by
Sorted by citations
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
Self-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationSisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231165001 · Cited by 99 · see citing works
Arina, P., Kaczorek, M. R., Hofmaenner, D. A., Pisciotta, W., Refinetti, P., Singer, M., Mazomenos, E. B., & Whittle, J. (2024) · Anesthesiology · doi:10.1097/aln.0000000000004764 · Cited by 81 · see citing works
Clinical prediction models using machine learning in oncology: challenges and recommendations
Collins, G. S., Chester-Jones, M., Gerry, S., Ma, J., Matos, J., Sehjal, J., Tsegaye, B., & Dhiman, P. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000914 · Cited by 27 · see citing works
Self-citationSitthiprawiat, P., Wittayachamnankul, B., Sirikul, W., & Laohavisudhi, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-17180-1 · Cited by 12 · see citing works
- Authors
- Riley, R. D., & Collins, G. S.
- Affiliations
- University of Birmingham (GB)
- Centre for Statistics in Medicine Nuffield Department of Orthopaedics Rheumatology and Musculoskeletal Sciences University of Oxford Oxford UK
- Corresponding author
- Riley, Richard D.
- Venue
- Biometrical Journal
- Publisher
- Wiley
- Published
- 2023-07-19
- Volume
- 65
- Issue
- 8
- Pages
- e2200302-e2200302
- ISSN
- 0323-3847 (ISSN-L)
- 1521-4036
- PMC
- PMC10952221
- OpenAlex
- W4384663192
- Cited by
- 211
- References
- 61
- Citations by year
- 2026: 98 · 2025: 62 · 2024: 39 · 2023: 12
- FWCI
- 43.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Biometrical Journal (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Cancer Research UK (C49297/A27294)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- clinical prediction model, model stability, calibration error, bootstrapping, risk of bias, model validation
- MeSH terms
- Machine Learning, Computer Simulation, Humans, Prognosis, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
