External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature
Open accessPublished version of a preprint
Abstract
Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites and over time.
“Importantly, CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites 14,15 and over time.16,17”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationA calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Self-citationMarshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
The Problem With Composite End Points in Cardiovascular Studies
Kip, K. E., Hollabaugh, K., Marroquin, O. C., & Williams, D. O. (2008) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2007.10.034 · Cited by 348 · see citing works
The case for algorithmic stewardship for artificial intelligence and machine learning technologies
Eaneff, S., Obermeyer, Z., & Butte, A. J. (2020) · JAMA · doi:10.1001/jama.2020.9371 · Cited by 101 · see citing works
A new concordance measure for risk prediction models in external validation settings
van Klaveren, D., Gönen, M., Steyerberg, E. W., & Vergouwe, Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6997 · Cited by 48 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 34
- [1]Wessler BS Paulus J Lundquist CM Ajlan M Natto Z Janes WA Jethmalani N Raman G Lutz JS Kent DM. Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015. Diagn Progn Res. 2017; 1: 20. doi: 10.1186/s41512-017-0021-2 31093549 PMC6460840 · DOI
- [3]Collins GS Reitsma JB Altman DG Moons KG; TRIPOD Group. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group. Circulation. 2015; 131: 211– 219. doi: 10.1161/CIRCULATIONAHA.114.014508 25561516 PMC4297220 · DOI · Paper page
- [4]Steyerberg EW Vickers AJ Cook NR Gerds T Gonen M Obuchowski N Pencina MJ Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010; 21: 128– 138. doi: 10.1097/EDE.0b013e3181c30fb2 20010215 PMC3575184 · DOI · Paper page
- [5]Vergouwe Y Moons KG Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010; 172: 971– 980. doi: 10.1093/aje/kwq223 20807737 PMC2984249 · DOI · Paper page
- [6]Siontis GC Tzoulaki I Castaldi PJ Ioannidis JP. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol. 2015; 68: 25– 34. doi: 10.1016/j.jclinepi.2014.09.007 25441703 · DOI
- [7]Adibi A Sadatsafavi M Ioannidis JPA. Validation and utility testing of clinical prediction models: time to change the approach. JAMA. 2020; 324: 235– 236. doi: 10.1001/jama.2020.1230 32134437 · DOI
- [8]Damen JA Hooft L Schuit E Debray TP Collins GS Tzoulaki I Lassale CM Siontis GC Chiocchia V Roberts C. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. 2016; 353: i2416. doi: 10.1136/bmj.i2416 27184143 PMC4868251 · DOI · Paper page
- [9]Van Calster B McLernon DJ van Smeden M Wynants L Steyerberg EW; Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019; 17: 230. doi: 10.1186/s12916-019-1466-7 31842878 PMC6912996 · DOI · Paper page
- [12]Wessler BS Lundquist CM Koethe B Park JG Brown K Williamson T Ajlan M Natto Z Lutz JS Paulus JK. Clinical prediction models for valvular heart disease. J Am Heart Assoc. 2019; 8: e011972. doi: 10.1161/JAHA.119.011972 31583938 PMC6818049 · DOI
- [13]Harrell FE. Regression Modeling Strategies. 2015. Springer International Publishing
- [14]Zeger SL Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986; 42: 121– 130. 3719049
- [15]Zeger SL Liang KY Albert PS. Models for longitudinal data: a generalized estimating equation approach. Biometrics. 1988; 44: 1049– 1060. 3233245
- [16]Roques F Michel P Goldstone AR Nashef SA. The logistic EuroSCORE. Eur Heart J. 2003; 24: 881– 882. doi: 10.1016/s0195-668x(02)00799-6 12727160 · DOI
- [17]Steyerberg EW. Clinical Prediction Models. 2009. Springer New York
- [18]Nashef SA Roques F Michel P Gauducheau E Lemeshow S Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999; 16: 9– 13. doi: 10.1016/s1010-7940(99)00134-7 10456395 · DOI · Paper page
- [19]Nashef SA Roques F Sharples LD Nilsson J Smith C Goldstone AR Lockowandt U. EuroSCORE II. Eur J Cardiothorac Surg. 2012; 41: 734– 44. discussion 744. doi: 10.1093/ejcts/ezs043 22378855 · DOI · Paper page
- [20]Granger CB Goldberg RJ Dabbous O Pieper KS Eagle KA Cannon CP Van De Werf F Avezum A Goodman SG Flather MD; Global Registry of Acute Coronary Events Investigators. Predictors of hospital mortality in the global registry of acute coronary events. Arch Intern Med. 2003; 163: 2345– 2353. doi: 10.1001/archinte.163.19.2345 14581255 · DOI
- [21]O’Brien SM Shahian DM Filardo G Ferraris VA Haan CK Rich JB Normand SL DeLong ER Shewan CM Dokholyan RS; Society of Thoracic Surgeons Quality Measurement Task Force. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2–isolated valve surgery. Ann Thorac Surg. 2009; 88 1 Suppl S23– S42. doi: 10.1016/j.athoracsur.2009.05.056 19559823 · DOI
- [22]Lip GY Nieuwlaat R Pisters R Lane DA Crijns HJ. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation. Chest. 2010; 137: 263– 272. doi: 10.1378/chest.09-1584 19762550 · DOI · Paper page
- [23]Gage BF Waterman AD Shannon W Boechler M Rich MW Radford MJ. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA. 2001; 285: 2864– 2870. doi: 10.1001/jama.285.22.2864 11401607 · DOI · Paper page
- [24]Wilson PW D’Agostino RB Levy D Belanger AM Silbershatz H Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998; 97: 1837– 1847. doi: 10.1161/01.cir.97.18.1837 9603539 · DOI · Paper page
- [25]Hemphill JC 3rd Bonovich DC Besmertis L Manley GT Johnston SC. The ICH score: a simple, reliable grading scale for intracerebral hemorrhage. Stroke. 2001; 32: 891– 897. doi: 10.1161/01.str.32.4.891 11283388 · DOI
- [26]Ranucci M Castelvecchio S Menicanti L Frigiola A Pelissero G. Risk of assessing mortality risk in elective cardiac operations: age, creatinine, ejection fraction, and the law of parsimony. Circulation. 2009; 119: 3053– 3061. doi: 10.1161/CIRCULATIONAHA.108.842393 19506110 · DOI
- [28]Ban JW Stevens R Perera R. Predictors for independent external validation of cardiovascular risk clinical prediction rules: Cox proportional hazards regression analyses. Diagn Progn Res. 2018; 2: 3. doi: 10.1186/s41512-018-0025-6 31093553 PMC6460844 · DOI
- [29]Steyerberg EW Harrell FE Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016; 69: 245– 247. doi: 10.1016/j.jclinepi.2015.04.005 25981519 PMC5578404 · DOI · Paper page
- [30]Collins SP Levy PD Lindsell CJ Pang PS Storrow AB Miller CD Naftilan AJ Thohan V Abraham WT Hiestand B. The rationale for an acute heart failure syndromes clinical trials network. J Card Fail. 2009; 15: 467– 474. doi: 10.1016/j.cardfail.2008.12.013 19643356 · DOI
- [31]Karam N Bataille S Marijon E Tafflet M Benamer H Caussin C Garot P Juliard JM Pires V Boche T; e-MUST Study Investigators. Incidence, mortality, and outcome-predictors of sudden cardiac arrest complicating myocardial infarction prior to hospital admission. Circ Cardiovasc Interv. 2019; 12: e007081. doi: 10.1161/CIRCINTERVENTIONS.118.007081 30608874 · DOI
- [32]Mehta SR Wood DA Storey RF Mehran R Bainey KR Nguyen H Meeks B Di Pasquale G López-Sendón J Faxon DP; COMPLETE Trial Steering Committee and Investigators. Complete revascularization with multivessel PCI for myocardial infarction. N Engl J Med. 2019; 381: 1411– 1421. doi: 10.1056/NEJMoa1907775 31475795 · DOI
- [33]Piccolo R Bonaa KH Efthimiou O Varenne O Baldo A Urban P Kaiser C Remkes W Räber L de Belder A; Coronary Stent Trialists’ Collaboration. Drug-eluting or bare-metal stents for percutaneous coronary intervention: a systematic review and individual patient data meta-analysis of randomised clinical trials. Lancet. 2019; 393: 2503– 2510. doi: 10.1016/S0140-6736(19)30474-X 31056295 · DOI
- [35]Mehran R Rao SV Bhatt DL Gibson CM Caixeta A Eikelboom J Kaul S Wiviott SD Menon V Nikolsky E. Standardized bleeding definitions for cardiovascular clinical trials: a consensus report from the Bleeding Academic Research Consortium. Circulation. 2011; 123: 2736– 2747. doi: 10.1161/CIRCULATIONAHA.110.009449 21670242 · DOI
- [37]Yancy CW Jessup M Bozkurt B Butler J Casey DE Jr Drazner MH Fonarow GC Geraci SA Horwich T Januzzi JL. 2013 ACCF/AHA guideline for the management of heart failure: executive summary: a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. Circulation. 2013; 128: 1810– 1852. doi: 10.1161/CIR.0b013e31829e8807 23741057 · DOI · Paper page
- [38]Chew DP Hyun K Morton E Horsfall M Hillis GS Chow CK Quinn S D’Souza M Yan AT Gale CP. Objective risk assessment vs standard care for acute coronary syndromes: a randomized clinical trial. JAMA Cardiol. 2021; 6: 304– 313. doi: 10.1001/jamacardio.2020.6314 33295965 PMC7726696 · DOI
- [39]Austin PC Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med. 2019; 38: 4051– 4065. doi: 10.1002/sim.8281 31270850 PMC6771733 · DOI
- [40]Vickers AJ Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006; 26: 565– 574. doi: 10.1177/0272989X06295361 17099194 PMC2577036 · DOI · Paper page
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · see citing works
“7 8 36 The ability to critically evaluate the quality of prediction models and understand their ability to serve well in a particular setting or for a particular use case is therefore of even greater critical importance.”
BackgroundSelf-citationEvaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416 · see citing works
“7 8 36 The ability to critically evaluate the quality of prediction models and understand their ability to serve well in a particular setting or for a particular use case is therefore of even greater critical importance.”
London, A. J. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100622 · Cited by 76 · see citing works
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19 · see citing works
Self-citationLevy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34646-2 · Cited by 14 · see citing works
Prognostic models for COVID-19 needed updating to warrant transportability over time and space
van Klaveren, D., Zanos, T. P., Nelson, J., Levy, T. J., Park, J. G., Retel Helmrich, I. R. A., Rietjens, J. A. C., Basile, M. J., Hajizadeh, N., Lingsma, H. F., et al. (2022) · BMC Medicine · doi:10.1186/s12916-022-02651-3 · Cited by 5 · see citing works
Self-citationA Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
- Authors
- Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., Paulus, J. K., & Kent, D. M.
- Affiliations
- Tufts Medical Center (US)
- KU Leuven, Department of Development and Regeneration, Belgium (B.V.C.).
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.
- Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands.
- Leiden University Medical Center (NL)
- Corresponding author
- Wessler, Benjamin S.
- ORCID
- Venue
- Circulation: Cardiovascular Quality and Outcomes
- Published
- 2021-08
- Volume
- 14
- Issue
- 8
- Pages
- e007858-e007858
- ISSN
- 1941-7705 (ISSN-L)
- 1941-7713
- PMC
- PMC8366535
- OpenAlex
- W3187581061
- Cited by
- 83
- References
- 8
- Citations by year
- 2026: 21 · 2025: 15 · 2024: 13 · 2023: 14 · 2022: 14 · 2021: 6
- FWCI
- 8.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Circulation Cardiovascular Quality and Outcomes (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by-nc-nd)
- Leiden Repository (Leiden University) (submittedVersion, cc-by-nc-nd)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by-nc-nd)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc-nd)
- Funders
- Patient-Centered Outcomes Research Institute (ME-1606-35555)
- National Institutes of Health
- National Institute on Aging (K23AG055667, R03AG056447)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- external validation, model discrimination, model calibration, performance heterogeneity, overfitting, cardiovascular disease
- MeSH terms
- Cardiovascular Diseases, Humans, ROC Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
