Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. Methods: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Results: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. Conclusions: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Amiodarone or an Implantable Cardioverter–Defibrillator for Congestive Heart Failure
Bardy, G. H., Lee, K. L., Mark, D. B., Poole, J. E., Packer, D. L., Boineau, R., Domanski, M., Troutman, C., Anderson, J., Johnson, G., et al. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa043399 · Cited by 6,552
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
The ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group (2002) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.288.23.2981 · Cited by 5,664
2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citationThe effect of digoxin on mortality and morbidity in patients with heart failure
(1997) · New England Journal of Medicine · doi:10.1056/nejm199702203360801 · Cited by 3,016
Spironolactone for heart failure with preserved ejection fraction
Pitt, B., Pfeffer, M. A., Assmann, S. F., Boineau, R., Anand, I. S., Claggett, B., Clausell, N., Desai, A. S., Diaz, R., Fleg, J. L., et al. (2014) · New England Journal of Medicine · doi:10.1056/nejmoa1313731 · Cited by 2,809
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Self-citationO'Connor (2009) · doi:10.1001/jama.2009.454 · Cited by 2,173
Probability of stroke: a risk profile from the Framingham Study.
Wolf, P. A., D'Agostino, R. B., Belanger, A. J., & Kannel, W. B. (1991) · Stroke · doi:10.1161/01.str.22.3.312 · Cited by 1,782
The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56)
STEVENS, R. J., KOTHARI, V., ADLER, A. I., & STRATTON, I. M. (2001) · Clinical Science · doi:10.1042/cs20000335 · Cited by 1,084
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855
Ridker, P. M., Paynter, N. P., Rifai, N., Gaziano, J. M., & Cook, N. R. (2008) · Circulation · doi:10.1161/circulationaha.108.814251 · Cited by 841
Konstam, M. A., Neaton, J. D., Dickstein, K., Drexler, H., Komajda, M., Martinez, F. A., Riegger, G. A., Malbecq, W., Smith, R. D., Guptha, S., et al. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61913-9 · Cited by 646
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 3×
“Models were updated using data from each paired validation dataset in a sequential fashion: (1) by updating the model intercept using the observed outcome rate in the validation cohort (recalibration-in-the-large); (2) by updating the intercept and rescaling all the model coefficients by the calibration slope; and (3)…”
Self-citationPeterson, P. N., Rumsfeld, J. S., Liang, L., Albert, N. M., Hernandez, A. F., Peterson, E. D., Fonarow, G. C., & Masoudi, F. A. (2010) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.109.854877 · Cited by 568
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552
Self-citationO'Connor, C. M., Abraham, W. T., Albert, N. M., Clare, R., Gattis Stough, W., Gheorghiade, M., Greenberg, B. H., Yancy, C. W., Young, J. B., & Fonarow, G. C. (2008) · American Heart Journal · doi:10.1016/j.ahj.2008.04.030 · Cited by 430
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371
Regulation of predictive analytics in medicine
Parikh, R. B., Obermeyer, Z., & Navathe, A. S. (2019) · Science · doi:10.1126/science.aaw0029 · Cited by 223
Ferrario, M., Chiodini, P., Chambless, L. E., Cesana, G., Vanuzzo, D., Panico, S., Sega, R., Pilotto, L., Palmieri, L., & Giampaoli, S. (2005) · International Journal of Epidemiology · doi:10.1093/ije/dyh405 · Cited by 214
Prediction of First Events of Coronary Heart Disease and Stroke With Consideration of Adiposity
Wilson, P. W. F., Bozeman, S. R., Burton, T. M., Hoaglin, D. C., Ben-Joseph, R., & Pashos, C. L. (2008) · Circulation · doi:10.1161/circulationaha.108.772962 · Cited by 174
Risk Prediction of Cardiovascular Disease in Type 2 Diabetes
Cederholm, J., Eeg-Olofsson, K., Eliasson, B., Zethelius, B., Nilsson, P. M., Gudbjörnsdottir, S., & Register, O. B. O. T. S. N. D. (2008) · Diabetes Care · doi:10.2337/dc08-0662 · Cited by 173
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83
Self-citationArima, H., Yonemoto, K., Doi, Y., Ninomiya, T., Hata, J., Tanizaki, Y., Fukuhara, M., Matsumura, K., Iida, M., & Kiyohara, Y. (2009) · Hypertension Research · doi:10.1038/hr.2009.161 · Cited by 66
Yatsuya, H., Iso, H., Yamagishi, K., Kokubo, Y., Saito, I., Suzuki, K., Sawada, N., Inoue, M., & Tsugane, S. (2013) · Stroke · doi:10.1161/strokeaha.111.677534 · Cited by 56
A new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48
Self-citation
Cited by
Interpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503
Self-citationThere is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
“Miscalibration can be detrimental to the medical decisions that are based on clinical prediction models [11, 12].”
BackgroundAssessing the net benefit of machine learning models in the presence of resource constraints
Singh, K., Shah, N. H., & Vickers, A. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad006 · Cited by 21
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19
Self-citationLevy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34646-2 · Cited by 14
Self-citation
Metadata
- Authors
- Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., van Klaveren, D., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., Van Calster, B., & Kent, D. M.
- Venue
- Circulation: Cardiovascular Quality and Outcomes
- Published
- 2022-04
- Volume
- 15
- Issue
- 4
- Pages
- e008487-e008487
- ISSN
- 1941-7705 (ISSN-L)
- 1941-7713
- OpenAlex
- W4220836183
- Cited by
- 55
- References
- 30
- FWCI
- 7.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.