CoChat
Sign Up
Cited by 278Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyStatistical Methods in Clinical Trials

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

Open access
Debray, Thomas PA, Damen, Johanna AAG, Riley, Richard D, Snell, Kym, Reitsma, Johannes B, Hooft, Lotty, Collins, Gary S, Moons, Karel GM·2019·Statistical Methods in Medical Research
DOI 10.1177/0962280218785504PMC PMC6728752
Open this paper in CoChat — free accountOpen PDF

Abstract

It is widely recommended that any developed—diagnostic or prognostic—prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package “metamisc”.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper provides a statistical framework for meta-analyzing the predictive performance of prediction models with binary or time-to-event outcomes, including methods to restore unreported performance estimates and to summarize discrimination and calibration. It illustrates the framework with EuroSCORE II and Framingham Risk Score meta-analyses and implements the methods in the R package “metamisc.”

Key findings

  • EuroSCORE II: summary c-statistic 0.79 (95% CI 0.76–0.81; 95% PI 0.68–0.87); summary O:E ratio 1.11 (95% CI 0.90–1.36; 95% PI 0.43–2.85); calibration slope could not be estimated.
  • Framingham Risk Score: summary c-statistic 0.69 (95% CI 0.66–0.71; 95% PI 0.59–0.77); summary O:E ratio 0.60 (95% CI 0.46–0.79; 95% PI 0.20–1.84); calibration slope 1.03 (95% CI 0.94–1.12; 95% PI 0.93–1.13) under ML Model 3.
  • Proposed empirically-based priors for between-study heterogeneity substantially improved estimation of the between-study standard deviation in small meta-analyses.
  • Bayesian estimation was recommended because frequentist methods may fail to produce reliable confidence or prediction intervals when validation studies are sparse.

Methods

  • The paper develops frequentist and Bayesian random-effects meta-analysis models for the logit c-statistic, log total O:E ratio, and calibration slope, including one-stage models and meta-regression for heterogeneity.
  • Empirical examples used 23 EuroSCORE II validations and Framingham Risk Score validations in male populations; Framingham meta-analyses included K=21 for c-statistic, K=17 for O:E ratio, and K=11 for calibration slope.
  • Missing c-statistics, standard errors, and O:E ratios were restored from reported quantities using formulas; analyses were implemented in the R package “metamisc” using “metafor” and JAGS, with Bayesian convergence assessed by potential scale reduction factor ≤1.05.

Limitations

  • The paper states that frequentist meta-analysis may suffer convergence issues and unreliable precision/heterogeneity estimates when validation studies are sparse; Hartung–Knapp intervals are too narrow with fewer than five studies, and prediction intervals have poor coverage in some settings.
  • The paper states that meta-regression has low power and may suffer ecological bias; calibration slope could not be estimated for EuroSCORE II because subgroup-specific event rates were not extracted and the slope was not directly provided.
  • Evident from the text: performance estimates are often poorly reported, requiring restoration and assumptions; time-to-event O:E analyses may require extrapolation when follow-up times differ; the empirical examples are reanalyses of existing reviews rather than new prospective validation studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/0962280218785504

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 74 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    The total number of observed and expected events was available for all 23 EuroSCORE II validations.

    “The total number of observed and expected events was available for all 23 validations.”
    Quote in the paperSubstantial claim
  • Findingown data

    Newcombe's method is fairly accurate for validation studies involving a binary or time-to-event outcome as long as the total number of observed events is sufficiently large (above 10).

    “As illustrated in Supporting Information 2, this method is fairly accurate for validation studies involving a binary or time-to-event outcome, as long as the total number of observed events is sufficiently large (above 10).”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Because the EuroSCORE II model does not consider time-to-event, estimates for total O:E ratios and their standard error could directly be derived from extracted quantities for O and E.

    “Furthermore, because the EuroSCORE II model does not consider time-to-event, estimates for the total O:E ratios and their standard error could directly be derived from extracted quantities for O and E”
    Quote in the paperSubstantial claim
  • Findingown data

    Accurate predictions of the c-statistic can be obtained for c < 0.70 by using the standard deviation of the linear predictor.

    “Results in Supporting Information 2.2 indicate that accurate predictions can be obtained for c < 0.70 by using the standard deviation of the linear predictor.”
    Quote in the paperSubstantial claim
  • Findingown data

    Estimates for the c-statistic could be obtained for all 23 EuroSCORE II validations.

    “Estimates for the c-statistic could be obtained for all 23 validations (Supporting Information 1.1).”
    Quote in the paperSubstantial claim
  • May combine two claims. 39
  • Classified as probably not a substantial claim. 24

  • May combine two claims.

    Findingown data

    For 10/24 Framingham validations, information was only available for 5 or 7.5 years of follow-up, and a sensitivity analysis extrapolated observed and expected event counts using a Poisson distribution.

    “For 10/24 validations, information was only available for 5 or 7.5 years of follow-up, and we therefore considered a sensitivity analysis where we extrapolated observed and expected event counts using a Poisson distribution.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Observed and expected event counts for different follow-up times can be extrapolated by assuming a Poisson distribution.

  • May combine two claims.

    Findingown data

    The c-statistic was reported in 19/24 Framingham validations; missing c-statistics were restored by contacting authors for two validations; for 21 estimates, 10 provided the standard error and 11 required approximation using Newcombe's method.

    “The c-statistic was only reported in 19/24 validations. In some cases, missing c-statistics could be restored by contacting the study authors (two validations). For these 21 estimates of the c-statistic, 10 studies provided the standard error and 11 studies required approximation of the standard error using Newcombe’s method”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    11 Framingham validations presented event rates for different risk strata.

    “A total of 11 validations presented event rates for different risk strata”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Omitting information on total sample size for seven studies, Model 2* could only be applied to 16 studies, yielding a summary estimate for the total O:E ratio of 1.17 (95% CI: 0.86 to 1.50).

    “Model 2* could then only be applied to 16 studies, yielding a summary estimate for the total O:E ratio of 1.17 (95% CI: 0.86 to 1.50).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the logit c-statistic and the log O:E ratio, it is unlikely that τdiscr>2 and τcal.OE>2.

    “Results in Supporting Information 4.2 demonstrate that for the logit c-statistic and the log O:E ratio, it is unlikely that τdiscr>2 and, respectively, τcal.OE>2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Although the 10-year predicted risk of CHD occurrence was usually too large (summary O:E=0.6), the calculated PI (0.2 to 1.8) indicates that predicted risks were too low for some of the validation studies.

    “Although the 10-year predicted risk of CHD occurrence was usually too large (summary O:E=0.6), the calculated PI (0.2 to 1.8) indicates that predicted risks were too low for some of the validation studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Substantial efforts are often needed to restore missing information from the primary studies and to harmonize the extracted performance statistics.

    “In particular, substantial efforts are often needed to restore missing information from the primary studies and to harmonize the extracted performance statistics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Results in Table 4 reveal substantial heterogeneity for the discrimination performance of the Framingham Risk Score, with 95% PI ranging from 0.59 to 0.77 based on the frequentist approach.

    “Results in Table 4 reveal substantial heterogeneity for the discrimination performance, with 95% PI ranging from 0.59 to 0.77 based on the frequentist approach.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The standard deviation of age substantially varied across the included validation studies (range: 2-13), further highlighting the presence of differences in patient spectrum.

    “Also the standard deviation of age substantially varied across the included validation studies (range: 2–13), further highlighting the presence of differences in patient spectrum.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The summary estimate of the calibration slope was close to 1, indicating that predictions by the Framingham Risk Score were sufficiently scaled (despite being too high on average).

    “The summary estimate of the calibration slope was close to 1, indicating that predictions by the Framingham Risk Score were sufficiently scaled (despite being too high on average).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When applying hierarchical related regressions with a binomial distribution for 16 studies and a Poisson distribution for the remaining seven studies, the meta-analysis was based on 23 studies and yielded a summary estimate of 1.10 (95% CI 0.88 to 1.32).

    “when applying hierarchical related regressions, a binomial distribution was used for aforementioned 16 studies whereas a Poisson distribution was used for the remaining seven studies. The meta-analysis is then again based on 23 studies and yielded a summary estimate of 1.10 (with a 95% CI of 0.88 to 1.32).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Although no empirical data was identified for heterogeneity on the calibration slope, individual estimates usually do not exceed [0,2] and thus it is also unlikely that τcal.slope>2.

    “Although no empirical data was identified for heterogeneity on the calibration slope, individual estimates usually do not exceed [0,2] and thus it is also unlikely that τcal.slope>2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Validation studies are often conducted using small and local data sets and rarely provide evidence about a model's potential generalizability across different settings and populations.

    “Furthermore, because validation studies are often conducted using small and local data sets,11 they rarely provide evidence about a model’s potential generalizability across different settings and populations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Interpretation of validation study results is often difficult because changes in prediction model performance are likely due to sampling error, differences in predictor effects, and/or differences in patient spectrum.

    “Unfortunately, the interpretation of validation study results is often difficult, as changes in prediction model performance are likely to occur due to sampling error, differences in predictor effects, and/or differences in patient spectrum.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source

      Not checked in the free run.

    • cites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    EuroSCORE II is a logistic regression model developed using 16,828 adult patients from 43 countries to predict mortality after cardiac surgery.

    “This logistic regression model was developed using 16,828 adult patients from 43 countries to predict mortality after cardiac surgery.3”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] Nashef SAM Roques F Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg 2012; 41: 734– 744; discussion 744–745. 22378855 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Previously discussed meta-analysis methods mainly focused on diagnostic prediction models with a binary outcome and may have limited value for prognostic models with a time-to-event outcome.

    “These methods mainly focused on (diagnostic) prediction models with a binary outcome, and may therefore have limited value when reviewing (prognostic) models with a time-to-event outcome.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    Systematic review followed by formal meta-analysis of multiple validations helps summarize overall performance across settings and reveals circumstances where a model performs suboptimally.

    “When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The total O:E ratio quantifies the averaged observed:expected ratio across the entire range of predicted risks and is strongly related to calibration-in-the-large.

    “In particular, the total O:E ratio quantifies the averaged observed:expected ratio across the entire range of predicted risks, and is strongly related to calibration-in-the-large.14”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Treating the total number of observed events as count data in a one-stage random-effects model is an alternative approach.

    “Alternatively, we may treat the total number of observed events as count data in a one-stage random-effects model43”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] Stijnen T Hamza TH Zdemir P Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat Med 2010; 29: 3046– 3067. 20827667 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Harrell's version of the c-statistic appears most widespread and recommended for time-to-event outcomes.

    “Although validation studies do not commonly report the definition and estimation method of presented c-statistics, Harrell’s version21 appears to be most widespread and recommended.22”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [21] Harrell FE Califf RM Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982; 247: 2543– 2546. 7069920 source

      Not checked in the free run.

    • cites [22] Lirette ST Aban I Quantifying predictive accuracy in survival models. J Nucl Cardiol 2015; 24: 1998– 2003. DOI:10.1007/s12350-015-0296-z. 26481370 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Blanche et al. showed that time-dependent ROC curve estimators perform similarly when censoring is independent, and are thus likely to yield comparable estimates of time-dependent concordance.

    “A recent study by Blanche et al. showed that these strategies perform similarly when censoring is independent,28 and are thus likely to yield comparable estimates of time-dependent concordance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28] Blanche P Dartigues JF Jacqmin-Gadda H Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J 2013; 55: 687– 704. 23794418 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    For survival outcomes, Oij and Nij represent quantities for specific time periods and should be adjusted when drop-out occurred or when studies evaluated different time periods.

    “For prediction models with a survival outcome, Oij and Nij represent quantities for specific (and same) time periods, and should be adjusted when drop-out occurred or when studies evaluated different time periods”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Validation studies rarely present information on the calibration intercept and slope, or different studies report estimates for different risk strata or time horizons.

    “For instance, validation studies rarely present information on the calibration intercept and slope, or different studies report estimates for different risk strata or time horizons.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source

      Not checked in the free run.

    • cites [16] Bouwmeester W Zuithoff NPA Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: 1– 12. PMC3358324 22629234 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Estimates of the total O:E ratio and its standard error should be transformed to the natural log scale prior to meta-analysis.

    “Further, as we will discuss in the next section, estimates of the total O:E ratio and its standard error should be transformed to the (natural) log scale prior to meta-analysis.30”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30] Snell KI Ensor J Debray TP, et al. Meta-analysis of prediction model performance across multiple studies: which scale helps ensure between-study normality for the C -statistic and calibration measures? Stat Meth Med Res 2017, pp. 096228021770567– 096228021770567. DOI:10.1177/0962280217705678. PMC6193210 28480827

      Not checked in the free run.

    • The authors state this as an assumption: Study-specific performance estimates are approximately normally distributed on the logit or log scale.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Random effects models usually yield larger confidence intervals and assign study weights that are more similar to one another than fixed effect meta-analysis.

    “As a result, random effects models usually yield larger confidence intervals and assign study weights that are more similar to one another than those under fixed effect meta-analysis.40”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Borenstein M Hedges LV Higgins JPT, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Meth 2010; 1: 97– 111. 26061376 source

      Not checked in the free run.

    • The authors state this as an assumption: In random effects meta-analysis, variation in predictive performance across studies may arise from chance and between-study heterogeneity.

  • May combine two claims.

    Premisecitation

    For time-to-event outcomes, concordance is not uniquely defined and several variations of the c-statistic have been proposed that adopt different strategies to account for ties or censoring and may lead to different values for c.

    “For prediction models with a time-to-event outcome, concordance is not uniquely defined and several variations of the c-statistic have been proposed.22,23 These variations typically adopt different strategies to account for ties or censoring, and may thus lead to different values for c.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [22] Lirette ST Aban I Quantifying predictive accuracy in survival models. J Nucl Cardiol 2015; 24: 1998– 2003. DOI:10.1007/s12350-015-0296-z. 26481370 source

      Not checked in the free run.

    • cites [23] Austin PC Pencinca MJ Steyerberg EW Predictive accuracy of novel risk factors and markers: a simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model. Stat Methods Med Res 2015; 26: 1053– 1077. DOI:10.1177/ 0962280214567141. 25656552 PMC5499735 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Framingham Risk Score is a Cox proportional hazards regression model developed in 1998 for predicting the 10-year risk of CHD in the general population free from CHD and not on treatment.

    “This Cox proportional hazards regression model was developed in 1998 for predicting the 10-year risk of CHD in the general population free from CHD and not on treatment”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Wilson PW D’Agostino RB Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97: 1837– 1847. 9603539 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Royston and Sauerbrei's D index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index.

    “This index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index, with Jinks et al. suggesting an equation to convert c values to D values based on empirical evidence25”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [24] Royston P Sauerbrei W A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723– 748. 14981672 source

      Not checked in the free run.

    • cites [25] Jinks RC Royston P Parmar MKB Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data. BMC Med Res Methodol 2015; 15: 82– 82. 26459415 PMC4603804 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the standard DerSimonian and Laird approach, the heterogeneity parameter τ2 is estimated separately and subsequently inserted in the equations.

    “In the standard DerSimonian and Laird approach, the heterogeneity parameter τ2 is estimated separately and subsequently inserted in the equations above.44,45”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [44] DerSimonian R Laird N Meta-analysis in clinical trials. Control Clin Trial 1986; 7: 177– 188. 3802833 source

      Not checked in the free run.

    • cites [45] Veroniki AA Jackson D Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Meth 2016; 7: 55– 79. PMC4950030 26332144 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A recent systematic review identified 23 studies examining the performance of Framingham Wilson for predicting fatal or nonfatal CHD in male populations.

    “A recent systematic review identified 23 studies examining the performance of Framingham Wilson for predicting fatal or nonfatal CHD in male populations”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Damen JAAG Pajouheshnia R Heus P, et al. Performance of the Framingham risk models and pooled cohort equations: a systematic review and meta-analysis. PLOS ONE 2018 Under review (submitted). PMC6563379 31189462

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    For binary outcome prediction models, the c-statistic is a reparameterization of Somer's D and can be derived from the distribution of the linear predictor, Cohen's effect size, or several correlation indices.

    “For instance, for prediction models with a binary outcome, the c-statistic is a reparameterization of Somer’s D,19 and can also be derived from the distribution of the linear predictor,20 Cohen’s effect size,20 or several correlation indices.21”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [19] Newson R Parameters behind nonparametric statistics: Kendalls tau, Somers D and median differences. Stata J 2002; 2: 45– 64. source

      Not checked in the free run.

    • cites [20] Austin PC Steyerberg EW Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82– 82. 22716998 PMC3528632 source

      Not checked in the free run.

    • cites [21] Harrell FE Califf RM Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982; 247: 2543– 2546. 7069920 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In meta-analysis without individual participant data, performance measures must be extracted from study publications, but they are often poorly reported.

    “In a meta-analysis without access to individual participant data (IPD), we are reliant on extracting these performance measures from study publications. However, they are often poorly reported.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source

      Not checked in the free run.

    • cites [16] Bouwmeester W Zuithoff NPA Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: 1– 12. PMC3358324 22629234 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    When missing, the standard error of the total O:E ratio can directly be derived from the total number of observed events.

    “When missing, the standard error of the total O:E ratio can directly be derived from the total number of observed events.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A recent review identified 23 validations of EuroSCORE II in patients undergoing cardiac surgery.

    “In a recent review,17 23 validations of EuroSCORE II were identified in which its predictive performance was examined in patients undergoing cardiac surgery.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Guida P Mastro F Scrascia G, et al. Performance of the European System for Cardiac Operative Risk Evaluation II: a meta-analysis of 22 studies involving 145,592 cardiac surgery procedures. J Thorac Cardiovasc Surg 2014; 148: 3049–3057.e1. 25161130 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The calibration slope could not be estimated for EuroSCORE II because subgroup-specific event rates were not extracted from the validation studies and the slope itself was not directly provided.

    “The calibration slope could not be estimated for EuroSCORE II because subgroup-specific event rates were not extracted from the validation studies and the slope itself was not directly provided.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Discrimination, and particularly calibration performance of EuroSCORE II substantially varied across studies.

    “Overall, the results suggest that, as expected, discrimination, and particularly calibration performance of EuroSCORE II substantially varied across studies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The Bayesian approach is more flexible and accounts for uncertainty more fully than a frequentist framework for linking likelihood functions.

    “the Bayesian approach is more flexible and accounts for uncertainty more fully.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Prediction intervals have poor coverage when the extent of heterogeneity is limited, when few (less than 5) studies are included, or when there are a mixture of large and small studies.

    “prediction intervals have poor coverage when the extent of heterogeneity is limited, when few (less than 5) studies are included, or when there are a mixture of large and small studies.46”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Although the average O:E ratio was close to 1, predicted risks were systematically too high or too low in certain populations as revealed by the PI.

    “Although the average O:E ratio was close to 1, predicted risks were systematically too high or too low in certain populations as revealed by the PI.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Convergence of Bayesian meta-analysis models was verified by evaluating the upper confidence limits of the potential scale reduction factor, with values ≤1.05 considered as converged.

    “Convergence of Bayesian meta-analysis models was verified by evaluating the upper confidence limits of the potential scale reduction factor (values ≤1.05 were considered as converged).60”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [60] Gelman A Rubin DB Inference from iterative simulation using multiple sequences. Stat Science 1992; 7: 457– 472. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Results from previous studies suggest that these methods are most likely to yield discordance when there is low between-study heterogeneity, when there are few studies for meta-analysis, or when studies are small or based on different sample sizes.

    “Results from previous studies suggest that these methods are most likely to yield discordance when there is low between-study heterogeneity, when there are few studies for meta-analysis, or when studies are small or based on different sample sizes.46,65,66”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source

      Not checked in the free run.

    • cites [65] Sidik K Jonkman JN A comparison of heterogeneity variance estimators in combining results of studies. Stat Med 2007; 26: 1964– 1981. 16955539 source

      Not checked in the free run.

    • cites [66] Langan D Higgins JPT Simmonds M Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Meth 2017; 8: 181– 198. DOI: 10.1002/jrsm.1198. 27060925 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Meta-regression may help to identify sources of heterogeneity and poor model transportability, but the power for this is often low.

    “Although meta-regression may help to identify sources of heterogeneity and poor model transportability, the power for this is often low.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    A 95% prediction interval provides a range for the predicted model performance in a new validation of the model.

    “This interval provides a range for the predicted model performance in a new validation of the model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Quantitative synthesis of prediction model studies may help to better understand their potential generalizability and can be achieved by applying meta-analysis methods.

    “Quantitative synthesis of prediction model studies may help to better understand their potential generalizability and can be achieved by applying meta-analysis methods.14”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.

    “Finally, because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.64”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [64] Austin PC Estimating multilevel logistic regression models when the number of clusters is low: a comparison of different statistical software procedures. Int J Biostat 2010; 6: Article 16. PMC2949382 20949128 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Bayesian methods use formal probability models to express uncertainty about parameter values.

    “Bayesian methods use formal probability models to express uncertainty about parameter values.49,50”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [49] Greenland S Bayesian perspectives for epidemiological research: I. Foundations and basic methods. Int J Epidemiol 2006; 35: 765– 775. 16446352 source

      Not checked in the free run.

    • cites [50] Spiegelhalter DJ. Evidence synthesis. In: Senn S and Barnett V (eds.) Bayesian approaches to clinical trials and health-care evaluation. Chichester, UK: John Wiley & Sons, Ltd, 2004, pp.267–303. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Heterogeneity may also arise when reported performance estimates are based on "improper" validations where certain predictors were neglected (e.g. due to missing data) or where various model parameters have been adjusted (e.g. intercept update).

    “Heterogeneity may also arise when reported performance estimates are based on “improper” validations where certain predictors were neglected (e.g. due to missing data) or where various model parameters have been adjusted (e.g. intercept update).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    We could not attribute heterogeneity in the c-statistic to differences in the distribution of the prognostic index, multicentre status, inclusion of patients before 2010, or spread of age.

    “We therefore could not attribute heterogeneity in the c-statistic to these differences.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Partlett and Riley showed that the equation for the 95% prediction interval has poor coverage in many situations and requires improvement.

    “Recently, Partlett and Riley showed that this equation has poor coverage in many situations, and requires improvement.46”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisenone given

    The Student-t distribution is chosen to account for the uncertainty in τ2.

    “The Student-t (rather than the Normal) distribution is chosen to account for the uncertainty in τ2.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    More advanced (so-called one-stage) approaches exist to directly combine individual participant data with published summary data.

    “more advanced (so-called one-stage) approaches exist to directly combine the IPD with published summary data.67”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Debray TPA Moons KGM van Valkenhoef G, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Meth 2015; 6: 239– 309. PMC5042043 26287812 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Estimates of a model's predictive performance are likely to be affected by the presence of between-study heterogeneity.

    “In addition, estimates of a model’s predictive performance are likely to be affected by the presence of between-study heterogeneity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Meta-regression may suffer from study-level confounding (also known as ecological bias) when analyzing aggregate patient (average) information.

    “In addition, meta-regression may suffer from study-level confounding (also known as ecological bias) when analyzing aggregate patient (average) information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Common sources of heterogeneity are differences in study characteristics, differences in study quality, or differences in baseline risk across the validation studies.

    “Common sources of heterogeneity are differences in study characteristics,62 differences in study quality, or differences in baseline risk across the validation studies.9,10”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [62] Ban JW Emparanza JI Urreta I, et al. Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study. PLoS ONE 2016; 11: e0145779– e0145779. 26730980 PMC4701404 source

      Not checked in the free run.

    • cites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source

      Not checked in the free run.

    • cites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Adopting a Bayesian estimation framework may help to fully propagate the uncertainty resulting from estimating the within-study standard errors and the between-study standard deviation, which in turn may improve the coverage of calculated intervals.

    “Further, adopting a Bayesian estimation framework may help to fully propagate the uncertainty resulting from estimating the within-study standard errors and the between-study standard deviation, which in turn may improve the coverage of calculated intervals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.

    “An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.9,10,26,68”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source

      Not checked in the free run.

    • cites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source

      Not checked in the free run.

    • cites [26] White IR Rapsomaniki E Emerging Risk Factors Collaboration. Covariate-adjusted measures of discrimination for survival data. Biom J 2015; 57: 592– 613. 25530064 PMC4666552 source

      Not checked in the free run.

    • cites [68] van Klaveren D Gnen M Steyerberg EW, et al. A new concordance measure for risk prediction models in external validation settings. Stat Med 2016; 35: 4136– 4152. DOI:10.1002/sim.6997. 27251001 PMC5550798 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Bayesian methods are particularly relevant when confronting sparse data, multiple comparisons, collinearity, or non-identification, and for deriving PI.

    “This is particularly relevant when confronting sparse data, multiple comparisons, collinearity, or non-identification, and for deriving PI.51”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [51] Higgins JPT Thompson SG Spiegelhalter DJ A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc 2009; 172: 137– 159. PMC2667312 19381330 source

      Not checked in the free run.

57 of 57

  • The authors state this as an assumption.

    D values can be converted to c values by making distributional assumptions about the standard deviation of the prognostic index.

    “It is also possible to convert D values to c values by making distributional assumptions with respect to the standard deviation of the prognostic index.26”

  • The authors state this as an assumption.

    Observed and expected event counts for different follow-up times can be extrapolated by assuming a Poisson distribution.

    “an alternative strategy that does not discard information is to apply extrapolation by assuming a Poisson distribution (Supporting Information 3.2).”

  • The authors state this as an assumption.

    Study-specific performance estimates are approximately normally distributed on the logit or log scale.

    “The logit transformation is applied to improve the validity of the Normality assumption, which is used to model variability around the summary c-statistic.30”

  • The authors state this as an assumption.

    In random effects meta-analysis, variation in predictive performance across studies may arise from chance and between-study heterogeneity.

    “In random effects meta-analysis models, it is assumed that variation in predictive performance measures across studies may not only appear due to chance, but is also prone to the (potential) presence of between-study heterogeneity.”

  • The paper relies on this without support.

    Validations of a specific model and existing evidence or data on predictive performance measures are often sparse.

    “validations of a specific model and thus existing evidence or data on predictive performance measures are often sparse.”

  • The authors state this as an assumption.

    Prior independence of μ and τ holds.

    “we assume prior independence of μ and τ.51”

  • The authors state this as an assumption.

    Covariate effects γ are (roughly) linear on the meta-analysis scale.

    “it assumes that covariate effects γ are (roughly) linear on the meta-analysis scale.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • O:E below 1 is described as both overprediction and underestimation

    Major

    The Framingham results interpret an O:E ratio of 0.6 as predicted risk being too large, but the meta-regression section calls O:E<1 "under-estimation." An O:E ratio below 1 means fewer events were observed than predicted, so the latter reverses the interpretation of the reported finding.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Damen JAAG, 2016, BMJ, 353
  2. Damen JAAG, 2018, PLOS ONE
  3. Snell KI, 2017, Stat Meth Med Res, 096228021770567
  4. Dorfman R, 1938, Biometric Bull, 1, 129
  5. Plummer M. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In: Proceedings of the 3rd international workshop on distributed statistical computing , Vienna, Austria, March 2003.

Cited by

Sorted by citations

Related

Authors
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G.
Affiliations
  • University Medical Center Utrecht (NL)
  • Keele University (GB)
  • Centre for Statistics in Medicine, University of Oxford, Oxford, UK
Corresponding author
Debray, Thomas PA
ORCID
Venue
Statistical Methods in Medical Research
Publisher
SAGE Publications
Published
2018-07-23
Volume
28
Issue
9
Pages
2768-2786
ISSN
  • 0962-2802 (ISSN-L)
  • 1477-0334
DOI
10.1177/0962280218785504
PMC
PMC6728752
OpenAlex
W2883189216
Semantic Scholar
6c2eb0b6f285a3b870e38916a4530aa254c8689f
Cited by
278
References
66
Citations by year
2026: 55 · 2025: 53 · 2024: 46 · 2023: 46 · 2022: 33 · 2021: 22 · 2020: 13 · 2019: 7 · 2018: 1 · 2012: 1
FWCI
2.12
Type
journal article
Language
en
Open access
Licence
Locations
  • Statistical Methods in Medical Research (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Funders
  • Netherlands Organization for Scientific Research (91617050)
  • Cochrane Methods Innovation Funds Round 2 (MTH001F)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
prediction model validation, binary outcomes, time-to-event outcomes, concordance index, calibration slope, observed-to-expected ratio, study heterogeneity, Bayesian meta-analysis, prior distributions, EuroSCORE II, Framingham risk score
MeSH terms
Systematic Reviews as Topic, Bayes Theorem, Calibration, Humans, Prognosis, Research Design, Meta-Analysis as Topic, Models, Statistical, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review