Instability of the AUROC of clinical prediction models
Open accessAbstract
ABSTRACT Background External validations are essential to assess the performance of a clinical prediction model (CPM) before deployment. Apart from model misspecification, also differences in patient population, the standard of care, predictor definitions, and other factors influence a model's discriminative ability, as commonly quantified by the AUC (or c‐statistic). We aimed to quantify the variation in AUCs across sets of external validation studies and propose ways to adjust expectations of a model's performance in a new setting. Methods The Tufts‐PACE CPM Registry holds a collection of CPMs for prognosis in cardiovascular disease. We analyzed the AUC estimates of 469 CPMs with at least one external validation. Combined, these CPMs had a total of 1603 external validations reported in the literature. For each CPM and its associated set of validation studies, we performed a random‐effects meta‐analysis to estimate the between‐study standard deviation among the AUCs. Since the majority of these meta‐analyses have only a handful of validations, this leads to very poor estimates of . So, instead of focusing on a single CPM, we estimated a log‐normal distribution of across all 469 CPMs. We then used this distribution as an empirical prior. We used cross‐validation to compare this empirical Bayesian approach with frequentist fixed and random‐effects meta‐analyses. Results The 469 CPMs included in our study had a median of 2 external validations with an IQR of [1–3]. The estimated distribution of had a mean of 0.055 and a standard deviation of 0.015. If = 0.05, then the 95% prediction interval for the AUC in a new setting has a width of at least 0.1, no matter how many validations have been done. When there are fewer than 5 validations, which is typically the case, the usual frequentist methods grossly underestimate the uncertainty about the AUC in a new setting. Accounting for in a Bayesian approach achieved near nominal coverage. Conclusion Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Meta-analysis and meta-regression methods can be used to quantify and understand heterogeneity in performance across external validation studies.
“Meta-analysis and meta-regression methods can be used to quantify and understand heterogeneity in performance across external validation studies.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [102]
Heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [102]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Conducting Meta-Analyses in <i>R</i> with the <b>metafor</b> Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · see citing works
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Self-citationPlea for routinely presenting prediction intervals in meta-analysis.
IntHout, J., Ioannidis, J. P. A., Rovers, M. M., & Goeman, J. J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010247 · Cited by 2,264 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Self-citationPrediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · see citing works
Self-citationVickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Self-citationAssessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
Roques, F. (2003) · European Heart Journal · doi:10.1016/s0195-668x(02)00799-6 · Cited by 1,144 · see citing works
Baseline Risk of Major Bleeding in Non–ST-Segment–Elevation Myocardial Infarction
Subherwal, S., Bach, R. G., Chen, A. Y., Gage, B. F., Rao, S. V., Newby, L. K., Wang, T. Y., Gibler, W. B., Ohman, E. M., Roe, M. T., et al. (2009) · Circulation · doi:10.1161/circulationaha.108.828541 · Cited by 1,007 · see citing works
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · see citing works
A general parametric approach to the meta‐analysis of randomized clinical trials
Whitehead, A., & Whitehead, J. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101105 · Cited by 750 · see citing works
Debray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597 · see citing works
Self-citationSteyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552 · see citing works
Self-citationGenders, T. S. S., Steyerberg, E. W., Alkadhi, H., et al. (2011) · European Heart Journal · doi:10.1093/eurheartj/ehr014 · Cited by 518 · see citing works
Self-citationInterpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503 · see citing works
Self-citationSiontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 462 · see citing works
A simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · cited 1× · see citing works
“Here, we use the default REML method, but in the supplement, we also consider the method of Sidik and Jonkman [17] which tends to behave most differently from REML among the remaining 11 methods.”
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286 · see citing works
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Self-citationDijkland, S. A., Foks, K. A., Polinder, S., Dippel, D. W. J., Maas, A. I. R., Lingsma, H. F., & Steyerberg, E. W. (2019) · Journal of Neurotrauma · doi:10.1089/neu.2019.6401 · Cited by 164 · see citing works
Self-citationTargeted validation: validating clinical prediction models in their intended population and setting
Sperrin, M., Riley, R. D., Collins, P. G. S., & Martin, G. P. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00136-8 · Cited by 115 · see citing works
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83 · see citing works
Self-citationAbu-Assi, E., Gracía-Acuña, J. M., González, I. F., Peña, C., Díz, P., & González‐Juanatey, J. R. (2010) · Circulation · doi:10.1161/circulationaha.109.925594 · Cited by 71 · see citing works
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 60 · see citing works
Self-citationde Hond, A., Kant, I. M. J., Fornasa, M., Cinà, G., Elbers, P., Thoral, P. J., Arbous, M. S., & Steyerberg, E. W. (2022) · Critical Care Medicine · doi:10.1097/ccm.0000000000005758 · Cited by 45 · see citing works
Self-citationbaggr: Bayesian Aggregate Treatment Effects
Więcek, W., & Meager, R. (2019) · doi:10.32614/cran.package.baggr · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [2]F. E. Harrell, Regression Modeling Strategies (Switzerland: Springer International Publishing, 2015).
Cited by
Sorted by citations
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citation
- Authors
- van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W.
- Affiliations
- Department of Biomedical Data Sciences Leiden University Medical Center Leiden The Netherlands
- Leiden University Medical Center (NL)
- Department of Public Health Erasmus University Medical Center Rotterdam Netherlands
- Tufts Medical Center (US)
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2025-02-08
- Volume
- 44
- Issue
- 5
- Pages
- e70011-e70011
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC11806515
- OpenAlex
- W4407295867
- Cited by
- 19
- References
- 30
- Citations by year
- 2026: 12 · 2025: 7
- FWCI
- 16.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Statistics in Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by-nc-nd)
- Leiden Repository (Leiden University) (submittedVersion, cc-by-nc-nd)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- National Institute on Aging (K23AG055667)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical prediction model, area under the receiver operating characteristic, external validation, study heterogeneity, forecast uncertainty, cardiovascular prognosis, random-effects meta-analysis, cross-validation, model misspecification
- MeSH terms
- Bayes Theorem, Cardiovascular Diseases, Humans, Prognosis, Registries, ROC Curve, Models, Statistical, Area Under Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
