A review of statistical updating methods for clinical prediction models
Open accessAbstract
A clinical prediction model is a tool for predicting healthcare outcomes, usually within a specific population and context. A common approach is to develop a new clinical prediction model for each population and context; however, this wastes potentially useful historical information. A better approach is to update or incorporate the existing clinical prediction models already developed for use in similar contexts or populations. In addition, clinical prediction models commonly become miscalibrated over time, and need replacing or updating. In this article, we review a range of approaches for re-using and updating clinical prediction models; these fall in into three main categories: simple coefficient updating, combining multiple previous clinical prediction models in a meta-model and dynamic updating of models. We evaluated the performance (discrimination and calibration) of the different strategies using data on mortality following cardiac surgery in the United Kingdom: We found that no single strategy performed sufficiently well to be used to the exclusion of the others. In conclusion, useful tools exist for updating existing clinical prediction models to a new population or context, and these should be implemented rather than developing a new clinical prediction model from scratch, using a breadth of complementary statistical methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Statistical updating methods are available for regression-based models.
“Statistical updating methods are available for regression-based models89,90.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [89]
All statistical updating strategies improved calibration performance of the clinical prediction model.
“0.97). All updating strategies im-”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
European system for cardiac operative risk evaluation (EuroSCORE)
Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/s1010-7940(99)00134-7 · Cited by 3,362 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
Nashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs043 · Cited by 2,971 · see citing works
Using Bayesian model averaging to calibrate forecast ensembles.
Raftery, A. E., Gneiting, T., Balabdaoui, F., & Polakowski, M. (2005) · Monthly Weather Review · doi:10.1175/mwr2906.1 · Cited by 1,915 · see citing works
Goodness of fit tests for the multiple logistic regression model
Hosmer, D., & Lemesbow, S. (1980) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928008827941 · Cited by 1,780 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · see citing works
The Society of Thoracic Surgeons 2008 Cardiac Surgery Risk Models: Part 2—Isolated Valve Surgery
O’Brien, S. M., Shahian, D. M., Filardo, G., Ferraris, V. A., Haan, C. K., Rich, J. B., Normand, S. T., DeLong, E. R., Shewan, C. M., Dokholyan, R. S., et al. (2009) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2009.05.056 · Cited by 1,259 · see citing works
Shahian, D. M., O’Brien, S. M., Filardo, G., Ferraris, V. A., Haan, C. K., Rich, J. B., Normand, S. T., DeLong, E. R., Shewan, C. M., Dokholyan, R. S., et al. (2009) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2009.05.053 · Cited by 1,256 · see citing works
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
Roques, F. (2003) · European Heart Journal · doi:10.1016/s0195-668x(02)00799-6 · Cited by 1,144 · see citing works
Prognosis Research Strategy (PROGRESS) 2: Prognostic Factor Research
Riley, R. D., Hayden, J. A., Steyerberg, E. W., Moons, K. G. M., Abrams, K., Kyzas, P. A., Malats, N., Briggs, A., Schroter, S., Altman, D. G., et al. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001380 · Cited by 823 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation and Updating
Mann, C. J. H. (2009) · Kybernetes · doi:10.1108/k.2009.06738fae.002 · Cited by 794 · see citing works
Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes
Hemingway, H., Croft, P. R., Perel, P., Hayden, J. A., Abrams, K. R., Timmis, A., Briggs, A. H., Udumyan, R., Moons, K. G. M., Steyerberg, E. W., et al. (2013) · BMJ · doi:10.1136/bmj.e5595 · Cited by 662 · see citing works
Multiple Imputation with Diagnostics (mi) inR: Opening Windows into the Black Box
Su, Y., Gelman, A., Hill, J., & Yajima, M. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i02 · Cited by 645 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · see citing works
Prognosis research strategy (PROGRESS) 4: Stratified medicine research
Hingorani, A. D., van der Windt, D., Riley, R. D., Abrams, K. R., Moons, K. G. M., Steyerberg, E. W., Schroter, S., Sauerbrei, W., Altman, D. G., Hemingway, H., et al. (2013) · BMJ · doi:10.1136/bmj.e5793 · Cited by 514 · see citing works
Raftery, A. E., Kárný, M., & Ettler, P. (2010) · Technometrics · doi:10.1198/tech.2009.08104 · Cited by 445 · see citing works
Shahian, D. M., O’Brien, S. M., Filardo, G., Ferraris, V. A., Haan, C. K., Rich, J. B., Normand, S. T., DeLong, E. R., Shewan, C. M., Dokholyan, R. S., et al. (2009) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2009.05.055 · Cited by 445 · see citing works
Updating methods improved the performance of a clinical prediction model in new patients
Janssen, K. J. M., Moons, K. G. M., Kalkman, C. J., Grobbee, D. E., & Vergouwe, Y. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.018 · Cited by 381 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Monitoring the results of cardiac surgery by variable life-adjusted display
Lovegrove, J., Valencia, O., Treasure, T., et al. (1997) · The Lancet · doi:10.1016/s0140-6736(97)06507-0 · Cited by 286 · see citing works
Debray, T. P. A., Moons, K. G. M., Ahmed, I., Koffijberg, H., & Riley, R. D. (2013) · Statistics in Medicine · doi:10.1002/sim.5732 · Cited by 215 · see citing works
Control chart methods for monitoring cardiac surgical performance and their interpretation
Rogers, C. A., Reeves, B. C., Caputo, M., et al. (2004) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2004.03.011 · Cited by 190 · see citing works
Confidence Intervals for the Area Under the ROC Curve
Cortes, C., & Mohri, M. (2004) · neural information processing systems · Cited by 158 · see citing works
Hickey, G. L., Grant, S. W., Murphy, G. J., Bhabra, M., Pagano, D., McAllister, K., Buchan, I., & Bridgewater, B. (2013) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs584 · Cited by 154 · see citing works
Self-citationClinical prediction rules: What are they and what do they tell us?
Beattie, P., & Nelson, R. M. (2006) · Australian Journal of Physiotherapy · doi:10.1016/s0004-9514(06)70024-1 · Cited by 139 · see citing works
A simple method to adjust clinical prediction models to local circumstances
Janssen, K. J. M., Vergouwe, Y., Kalkman, C. J., Grobbee, D. E., & Moons, K. G. M. (2009) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-009-9041-x · Cited by 126 · see citing works
Qin, G., & Hotilovac, L. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280207087173 · Cited by 126 · see citing works
Clinical registries: governance, management, analysis and applications
Hickey, G. L., Grant, S. W., Cosgriff, R., Dimarakis, I., Pagano, D., Kappetein, A. P., & Bridgewater, B. (2013) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezt018 · Cited by 125 · see citing works
Self-citationThe Risk of Severe Postoperative Pain: Modification and Validation of a Clinical Prediction Rule
Janssen, K. J. M., Kalkman, C. J., Grobbee, D. E., Bonsel, G. J., Moons, K. G. M., & Vergouwe, Y. (2008) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e31818227da · Cited by 110 · see citing works
Grant, S. W., Hickey, G. L., Dimarakis, I., Trivedi, U., Bryan, A. J., Treasure, T., Cooper, G. J., Pagano, D., Buchan, I. E., & Bridgewater, B. (2012) · Heart · doi:10.1136/heartjnl-2012-302483 · Cited by 107 · see citing works
Self-citationDynamic Logistic Regression and Dynamic Model Averaging for Binary Classification
McCormick, T. H., Raftery, A., Madigan, D., & Burd, R. S. (2011) · Biometrics · doi:10.1111/j.1541-0420.2011.01645.x · Cited by 88 · see citing works
Using Society of Thoracic Surgeons Risk Models for Risk-Adjusting Cardiac Surgery Results
Jin, R., Furnary, A. P., Fine, S. C., et al. (2010) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2009.10.078 · Cited by 75 · see citing works
Construction, validation and updating of a prognostic model for kidney graft survival
van Houwelingen, H. C., & Thorogood, J. (1995) · Statistics in Medicine · doi:10.1002/sim.4780141806 · Cited by 74 · see citing works
Meta‐analysis and aggregation of multiple published prediction models
Debray, T. P. A., Koffijberg, H., Nieboer, D., Vergouwe, Y., Steyerberg, E. W., & Moons, K. G. M. (2014) · Statistics in Medicine · doi:10.1002/sim.6080 · Cited by 73 · see citing works
Adaptation of Clinical Prediction Models for Application in Local Settings
Kappen, T. H., Vergouwe, Y., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., & Moons, K. G. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12439755 · Cited by 70 · see citing works
Debray, T. P. A., Koffijberg, H., Vergouwe, Y., Moons, K., & Steyerberg, E. (2012) · Statistics in Medicine · doi:10.1002/sim.5412 · Cited by 66 · see citing works
Dynamic Prediction Modeling Approaches for Cardiac Surgery
Hickey, G., Grant, S., Caiado, C., Kendall, S., Dunning, J., Poullis, M., Buchan, I. E., & Bridgewater, B. (2013) · Circulation. Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.111.000012 · Cited by 55 · see citing works
Self-citationStrobl, A. N., Vickers, A. J., CALSTER, B. V., Steyerberg, E. W., Leach, R. J., Thompson, I. M., & Ankerst, D. P. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.001 · Cited by 46 · see citing works
Court hears how drug giant Merck tried to "neutralise" and "discredit" doctors critical of Vioxx
Moynihan, R. N. (2009) · BMJ · doi:10.1136/bmj.b1432 · Cited by 40 · see citing works
Simple dichotomous updating methods improved the validity of polytomous prediction models
Hoorde, K. V., Vergouwe, Y., Timmerman, D., Huffel, S. V., Steyerberg, E. W., & CALSTER, B. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.04.014 · Cited by 11 · see citing works
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Research works printed but not linked: 1
- Van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med. 2000 Dec 30;19(24):3401-15.
Cited by
Sorted by citations
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Interpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503 · see citing works
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Damen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250 · see citing works
“Updating the common slope When externally validating a prediction model, adapting the beta coefficients of the model using a single correction factor, to proportionally adjust for changes in predictor outcome associations [20].”
MethodMeehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219 · see citing works
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131 · see citing works
Self-citationBadidi, E. (2023) · Future Internet · doi:10.3390/fi15110370 · Cited by 113 · see citing works
Guo, L. L., Pfohl, S. R., Fries, J., Posada, J., Fleming, S. L., Aftandilian, C., Shah, N., & Sung, L. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735184 · Cited by 96 · see citing works
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33 · see citing works
Levy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34646-2 · Cited by 14 · see citing works
Prognostic models for COVID-19 needed updating to warrant transportability over time and space
van Klaveren, D., Zanos, T. P., Nelson, J., Levy, T. J., Park, J. G., Retel Helmrich, I. R. A., Rietjens, J. A. C., Basile, M. J., Hajizadeh, N., Lingsma, H. F., et al. (2022) · BMC Medicine · doi:10.1186/s12916-022-02651-3 · Cited by 5 · see citing works
- Authors
- Su, T.-L., Jaki, T., Hickey, G. L., Buchan, I., & Sperrin, M.
- Affiliations
- University of Manchester (GB)
- Lancaster University (GB)
- University of Liverpool (GB)
- Corresponding author
- Su, Ting-Li
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2016-07-26
- Volume
- 27
- Issue
- 1
- Pages
- 185-197
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2498993605
- Cited by
- 147
- References
- 54
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 15 · 2023: 11 · 2022: 36 · 2021: 20 · 2020: 12 · 2019: 9 · 2018: 12 · 2017: 6 · 2016: 1
- FWCI
- 9.44
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.lancs.ac.uk
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Funders
- National Institute for Health and Care Research (CDF-2010-03-16, CDF-2010-03-32, RMOFS 2012-09)
- Medical Research Council (MR/K006665/1, NIHR-CDF-2010-03-32, MC_PC_13042)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- clinical prediction model, model updating, model recalibration, meta-model, dynamic updating, model calibration, discrimination (statistics), mortality prediction, cardiac surgery outcomes
- MeSH terms
- Forecasting, United Kingdom, Cardiac Surgical Procedures, Humans, Registries, Regression Analysis, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
