A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

In the paper

“Furthermore, because validation studies are often conducted using small and local data sets,11 they rarely provide evidence about a model’s potential generalizability across different settings and populations.”

What the sources say

• Supported

  The citation is attached to the claim that validation studies often use small and local datasets. Results > Model validation: study design reports that 48% of validation datasets had fewer than 100 outcome events; Results reports that 12 of 33 studies combining model development and validation used data from the same centre at a different time. The sentence’s subsequent inference about generalizability is not directly attributed to this citation. (LLM, conf=0.85)

  cites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source

In the paper

“In the standard DerSimonian and Laird approach, the heterogeneity parameter τ2 is estimated separately and subsequently inserted in the equations above.44,45”

What the sources say

• Not checked yetcites [44] DerSimonian R Laird N Meta-analysis in clinical trials. Control Clin Trial 1986; 7: 177– 188. 3802833 source
• Supported

  The ‘DerSimonian and Laird (DL) method’ section gives a separate, non-iterative estimate of τ² calculated from the fixed-effect Q-statistic. The ‘Models for meta-analysis’ section says the random-effects summary estimate uses weights that depend on the estimated between-study variance, supporting the description of inserting that estimate into the subsequent equations. Our copy does not show the citing paper’s ‘equations above,’ so it cannot verify their exact form. (LLM, conf=0.91)

  cites [45] Veroniki AA Jackson D Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Meth 2016; 7: 55– 79. PMC4950030 26332144 source

In the paper

“For 10/24 validations, information was only available for 5 or 7.5 years of follow-up, and we therefore considered a sensitivity analysis where we extrapolated observed and expected event counts using a Poisson distribution.”

In the paper

“The c-statistic was only reported in 19/24 validations. In some cases, missing c-statistics could be restored by contacting the study authors (two validations). For these 21 estimates of the c-statistic, 10 studies provided the standard error and 11 studies required approximation of the standard error using Newcombe’s method”

In the paper

“A total of 11 validations presented event rates for different risk strata”

In the paper

“Model 2* could then only be applied to 16 studies, yielding a summary estimate for the total O:E ratio of 1.17 (95% CI: 0.86 to 1.50).”

In the paper

“Results in Supporting Information 4.2 demonstrate that for the logit c-statistic and the log O:E ratio, it is unlikely that τdiscr>2 and, respectively, τcal.OE>2.”

In the paper

“Although the 10-year predicted risk of CHD occurrence was usually too large (summary O:E=0.6), the calculated PI (0.2 to 1.8) indicates that predicted risks were too low for some of the validation studies.”

In the paper

“In particular, substantial efforts are often needed to restore missing information from the primary studies and to harmonize the extracted performance statistics.”

In the paper

“Results in Table 4 reveal substantial heterogeneity for the discrimination performance, with 95% PI ranging from 0.59 to 0.77 based on the frequentist approach.”

In the paper

“Also the standard deviation of age substantially varied across the included validation studies (range: 2–13), further highlighting the presence of differences in patient spectrum.”

In the paper

“The summary estimate of the calibration slope was close to 1, indicating that predictions by the Framingham Risk Score were sufficiently scaled (despite being too high on average).”

In the paper

“when applying hierarchical related regressions, a binomial distribution was used for aforementioned 16 studies whereas a Poisson distribution was used for the remaining seven studies. The meta-analysis is then again based on 23 studies and yielded a summary estimate of 1.10 (with a 95% CI of 0.88 to 1.32).”

In the paper

“Although no empirical data was identified for heterogeneity on the calibration slope, individual estimates usually do not exceed [0,2] and thus it is also unlikely that τcal.slope>2.”

In the paper

“Unfortunately, the interpretation of validation study results is often difficult, as changes in prediction model performance are likely to occur due to sampling error, differences in predictor effects, and/or differences in patient spectrum.”

What the sources say

• Not checked yetcites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source
• Not checked yetcites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source

In the paper

“This logistic regression model was developed using 16,828 adult patients from 43 countries to predict mortality after cardiac surgery.3”

What the sources say

• Not checked yetcites [3] Nashef SAM Roques F Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg 2012; 41: 734– 744; discussion 744–745. 22378855 source

In the paper

“These methods mainly focused on (diagnostic) prediction models with a binary outcome, and may therefore have limited value when reviewing (prognostic) models with a time-to-event outcome.”

What the sources say

• Not checked yetcites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

In the paper

“When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment.”

In the paper

“In particular, the total O:E ratio quantifies the averaged observed:expected ratio across the entire range of predicted risks, and is strongly related to calibration-in-the-large.14”

What the sources say

• Not checked yetcites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

In the paper

“Alternatively, we may treat the total number of observed events as count data in a one-stage random-effects model43”

What the sources say

• Not checked yetcites [43] Stijnen T Hamza TH Zdemir P Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat Med 2010; 29: 3046– 3067. 20827667 source

In the paper

“Although validation studies do not commonly report the definition and estimation method of presented c-statistics, Harrell’s version21 appears to be most widespread and recommended.22”

What the sources say

• Not checked yetcites [21] Harrell FE Califf RM Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982; 247: 2543– 2546. 7069920 source
• Not checked yetcites [22] Lirette ST Aban I Quantifying predictive accuracy in survival models. J Nucl Cardiol 2015; 24: 1998– 2003. DOI:10.1007/s12350-015-0296-z. 26481370 source

In the paper

“A recent study by Blanche et al. showed that these strategies perform similarly when censoring is independent,28 and are thus likely to yield comparable estimates of time-dependent concordance.”

What the sources say

• Not checked yetcites [28] Blanche P Dartigues JF Jacqmin-Gadda H Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J 2013; 55: 687– 704. 23794418 source

In the paper

“For prediction models with a survival outcome, Oij and Nij represent quantities for specific (and same) time periods, and should be adjusted when drop-out occurred or when studies evaluated different time periods”

In the paper

“For instance, validation studies rarely present information on the calibration intercept and slope, or different studies report estimates for different risk strata or time horizons.”

What the sources say

• Not checked yetcites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source
• Not checked yetcites [16] Bouwmeester W Zuithoff NPA Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: 1– 12. PMC3358324 22629234 source

In the paper

“Further, as we will discuss in the next section, estimates of the total O:E ratio and its standard error should be transformed to the (natural) log scale prior to meta-analysis.30”

What the sources say

• Not checked yetcites [30] Snell KI Ensor J Debray TP, et al. Meta-analysis of prediction model performance across multiple studies: which scale helps ensure between-study normality for the C -statistic and calibration measures? Stat Meth Med Res 2017, pp. 096228021770567– 096228021770567. DOI:10.1177/0962280217705678. PMC6193210 28480827

In the paper

“As a result, random effects models usually yield larger confidence intervals and assign study weights that are more similar to one another than those under fixed effect meta-analysis.40”

What the sources say

• Not checked yetcites [40] Borenstein M Hedges LV Higgins JPT, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Meth 2010; 1: 97– 111. 26061376 source

In the paper

“For prediction models with a time-to-event outcome, concordance is not uniquely defined and several variations of the c-statistic have been proposed.22,23 These variations typically adopt different strategies to account for ties or censoring, and may thus lead to different values for c.”

What the sources say

• Not checked yetcites [22] Lirette ST Aban I Quantifying predictive accuracy in survival models. J Nucl Cardiol 2015; 24: 1998– 2003. DOI:10.1007/s12350-015-0296-z. 26481370 source
• Not checked yetcites [23] Austin PC Pencinca MJ Steyerberg EW Predictive accuracy of novel risk factors and markers: a simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model. Stat Methods Med Res 2015; 26: 1053– 1077. DOI:10.1177/ 0962280214567141. 25656552 PMC5499735 source

In the paper

“This Cox proportional hazards regression model was developed in 1998 for predicting the 10-year risk of CHD in the general population free from CHD and not on treatment”

What the sources say

• Not checked yetcites [4] Wilson PW D’Agostino RB Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97: 1837– 1847. 9603539 source

In the paper

“This index quantifies prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index, with Jinks et al. suggesting an equation to convert c values to D values based on empirical evidence25”

What the sources say

• Not checked yetcites [24] Royston P Sauerbrei W A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723– 748. 14981672 source
• Not checked yetcites [25] Jinks RC Royston P Parmar MKB Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data. BMC Med Res Methodol 2015; 15: 82– 82. 26459415 PMC4603804 source

In the paper

“A recent systematic review identified 23 studies examining the performance of Framingham Wilson for predicting fatal or nonfatal CHD in male populations”

What the sources say

• Not checked yetcites [18] Damen JAAG Pajouheshnia R Heus P, et al. Performance of the Framingham risk models and pooled cohort equations: a systematic review and meta-analysis. PLOS ONE 2018 Under review (submitted). PMC6563379 31189462

In the paper

“For instance, for prediction models with a binary outcome, the c-statistic is a reparameterization of Somer’s D,19 and can also be derived from the distribution of the linear predictor,20 Cohen’s effect size,20 or several correlation indices.21”

What the sources say

• Not checked yetcites [19] Newson R Parameters behind nonparametric statistics: Kendalls tau, Somers D and median differences. Stata J 2002; 2: 45– 64. source
• Not checked yetcites [20] Austin PC Steyerberg EW Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82– 82. 22716998 PMC3528632 source
• Not checked yetcites [21] Harrell FE Califf RM Pryor DB, et al. Evaluating the yield of medical tests. JAMA 1982; 247: 2543– 2546. 7069920 source

In the paper

“In a meta-analysis without access to individual participant data (IPD), we are reliant on extracting these performance measures from study publications. However, they are often poorly reported.”

What the sources say

• Not checked yetcites [11] Collins GS de Groot JA Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40– 40. 24645774 PMC3999945 source
• Not checked yetcites [16] Bouwmeester W Zuithoff NPA Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: 1– 12. PMC3358324 22629234 source

In the paper

“When missing, the standard error of the total O:E ratio can directly be derived from the total number of observed events.”

In the paper

“In a recent review,17 23 validations of EuroSCORE II were identified in which its predictive performance was examined in patients undergoing cardiac surgery.”

What the sources say

• Not checked yetcites [17] Guida P Mastro F Scrascia G, et al. Performance of the European System for Cardiac Operative Risk Evaluation II: a meta-analysis of 22 studies involving 145,592 cardiac surgery procedures. J Thorac Cardiovasc Surg 2014; 148: 3049–3057.e1. 25161130 source

In the paper

“The calibration slope could not be estimated for EuroSCORE II because subgroup-specific event rates were not extracted from the validation studies and the slope itself was not directly provided.”

In the paper

“Overall, the results suggest that, as expected, discrimination, and particularly calibration performance of EuroSCORE II substantially varied across studies.”

In the paper

“the Bayesian approach is more flexible and accounts for uncertainty more fully.”

In the paper

“prediction intervals have poor coverage when the extent of heterogeneity is limited, when few (less than 5) studies are included, or when there are a mixture of large and small studies.46”

What the sources say

• Not checked yetcites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source

In the paper

“Although the average O:E ratio was close to 1, predicted risks were systematically too high or too low in certain populations as revealed by the PI.”

In the paper

“Convergence of Bayesian meta-analysis models was verified by evaluating the upper confidence limits of the potential scale reduction factor (values ≤1.05 were considered as converged).60”

What the sources say

• Not checked yetcites [60] Gelman A Rubin DB Inference from iterative simulation using multiple sequences. Stat Science 1992; 7: 457– 472. source

In the paper

“Results from previous studies suggest that these methods are most likely to yield discordance when there is low between-study heterogeneity, when there are few studies for meta-analysis, or when studies are small or based on different sample sizes.46,65,66”

What the sources say

• Not checked yetcites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source
• Not checked yetcites [65] Sidik K Jonkman JN A comparison of heterogeneity variance estimators in combining results of studies. Stat Med 2007; 26: 1964– 1981. 16955539 source
• Not checked yetcites [66] Langan D Higgins JPT Simmonds M Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Meth 2017; 8: 181– 198. DOI: 10.1002/jrsm.1198. 27060925 source

In the paper

“Although meta-regression may help to identify sources of heterogeneity and poor model transportability, the power for this is often low.”

In the paper

“This interval provides a range for the predicted model performance in a new validation of the model.”

In the paper

“Quantitative synthesis of prediction model studies may help to better understand their potential generalizability and can be achieved by applying meta-analysis methods.14”

What the sources say

• Not checked yetcites [14] Debray TPA Damen JAAG Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460– i6460. 28057641 source

In the paper

“Finally, because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.64”

What the sources say

• Not checked yetcites [64] Austin PC Estimating multilevel logistic regression models when the number of clusters is low: a comparison of different statistical software procedures. Int J Biostat 2010; 6: Article 16. PMC2949382 20949128 source

In the paper

“Bayesian methods use formal probability models to express uncertainty about parameter values.49,50”

What the sources say

• Not checked yetcites [49] Greenland S Bayesian perspectives for epidemiological research: I. Foundations and basic methods. Int J Epidemiol 2006; 35: 765– 775. 16446352 source
• Not checked yetcites [50] Spiegelhalter DJ. Evidence synthesis. In: Senn S and Barnett V (eds.) Bayesian approaches to clinical trials and health-care evaluation. Chichester, UK: John Wiley & Sons, Ltd, 2004, pp.267–303. source

In the paper

“Heterogeneity may also arise when reported performance estimates are based on “improper” validations where certain predictors were neglected (e.g. due to missing data) or where various model parameters have been adjusted (e.g. intercept update).”

In the paper

“We therefore could not attribute heterogeneity in the c-statistic to these differences.”

In the paper

“Recently, Partlett and Riley showed that this equation has poor coverage in many situations, and requires improvement.46”

What the sources say

• Not checked yetcites [46] Partlett C Riley RD Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301– 317. 27714841 PMC5157768 source

In the paper

“The Student-t (rather than the Normal) distribution is chosen to account for the uncertainty in τ2.”

In the paper

“more advanced (so-called one-stage) approaches exist to directly combine the IPD with published summary data.67”

What the sources say

• Not checked yetcites [67] Debray TPA Moons KGM van Valkenhoef G, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Meth 2015; 6: 239– 309. PMC5042043 26287812 source

In the paper

“In addition, estimates of a model’s predictive performance are likely to be affected by the presence of between-study heterogeneity.”

In the paper

“In addition, meta-regression may suffer from study-level confounding (also known as ecological bias) when analyzing aggregate patient (average) information.”

In the paper

“Common sources of heterogeneity are differences in study characteristics,62 differences in study quality, or differences in baseline risk across the validation studies.9,10”

What the sources say

• Not checked yetcites [62] Ban JW Emparanza JI Urreta I, et al. Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study. PLoS ONE 2016; 11: e0145779– e0145779. 26730980 PMC4701404 source
• Not checked yetcites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source
• Not checked yetcites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source

In the paper

“Further, adopting a Bayesian estimation framework may help to fully propagate the uncertainty resulting from estimating the within-study standard errors and the between-study standard deviation, which in turn may improve the coverage of calculated intervals.”

In the paper

“An additional advantage of IPD is that it becomes possible to adjust discrimination performance for (variation in) subject-level covariates, and thus to gain more understanding in sources of between-study heterogeneity.9,10,26,68”

What the sources say

• Not checked yetcites [9] Debray TPA Vergouwe Y Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289. 25179855 source
• Not checked yetcites [10] Vergouwe Y Moons KGM Steyerberg EW External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980. 20807737 PMC2984249 source
• Not checked yetcites [26] White IR Rapsomaniki E Emerging Risk Factors Collaboration. Covariate-adjusted measures of discrimination for survival data. Biom J 2015; 57: 592– 613. 25530064 PMC4666552 source
• Not checked yetcites [68] van Klaveren D Gnen M Steyerberg EW, et al. A new concordance measure for risk prediction models in external validation settings. Stat Med 2016; 35: 4136– 4152. DOI:10.1002/sim.6997. 27251001 PMC5550798 source

In the paper

“This is particularly relevant when confronting sparse data, multiple comparisons, collinearity, or non-identification, and for deriving PI.51”

What the sources say

• Not checked yetcites [51] Higgins JPT Thompson SG Spiegelhalter DJ A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc 2009; 172: 137– 159. PMC2667312 19381330 source