Reporting and Methods in Clinical Prediction Research: A Systematic Review

In the paper

“Commonly missing values are related to observed participant or disease characteristics.”

What the sources say

• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
• Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
• Supported

  Section 2, “Types of missing data,” says the probability of missingness commonly depends on observed patient characteristics. It gives disease status as an example of such an observed characteristic, directly supporting the claim. (LLM, conf=0.94)

  cites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source
• Not checked yetcites [24] S Greenland, 1995, A critical look at methods for handling missing covariates in epidemiologic regression analyses., Am J Epidemiol source
• Partly supported

  The Results section ('Exploration of the missing data mechanism'; page not provided) reports differences between patients with complete and missing data in seven articles, including comparisons of patient or clinical characteristics. But only 10 of the 100 reviewed articles examined such differences, so the paper does not establish that missingness is commonly related to observed participant or disease characteristics across studies. (LLM, conf=0.85)

  cites [57] A Burton, 2004, Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines., Br J Cancer sourceExperimental — accuracy not yet measured
• Not checked yetcites [58] MH Gorelick, 2006, Bias arising from missing data in predictive models., J Clin Epidemiol source

In the paper

“Study recruitment dates were reported in 88% of the studies.”

In the paper

“11/68 studies had all cause mortality as the outcome, where such bias would not be a factor.”

In the paper

“Given this, effective sample size was sufficient in the majority of the external validation studies (9/13 models).”

In the paper

“Outcomes were well defined in 62/68 studies (Table 3).”

In the paper

“A number of studies predicted a combined endpoint (14/71).”

In the paper

“In the studies that clearly described the number of candidate predictors, the effective sample size was <10 events per variable in 50% (n = 21) of the presented models.”

In the paper

“Prediction of more than one outcome was very common in predictor finding studies, apparently because of their exploratory aim (Table S1).”

In the paper

“There were few multivariable diagnostic studies (n = 5).”

In the paper

“Description of the cohort characteristics was clear in 68/69 of the relevant prediction studies (not applicable for two non-nested case-control studies).”

In the paper

“Most multivariable prediction studies were published in the field of cardiovascular diseases (n = 24) (Table 1).”

In the paper

“The aim was usually to identify independently contributing predictors of an outcome (n = 51/71).”

In the paper

“Based on the former, as expected, the statistical power was indeed >10 events per variable in 84 (11+60+1+13) of the 124 (96+28) models fitted in these studies.”

In the paper

“Description of the candidate predictor variables was in general clear (59/68) (Table 4).”

In the paper

“Dichotomization of continuous predictors is still common practice (21/64), despite being discouraged for decades [3],[50].”

In the paper

“Prospective cohort data, either observational or randomized trial data, were most frequently used (n = 44).”

In the paper

“In the 71 publications, 135 models or sets of predictors were studied.”

In the paper

“in some studies, continuous, categorical, and time-to-event data were analyzed as binary outcomes”

In the paper

“Most studied outcomes were binary or time-to-event outcomes.”

In the paper

“We found that case-control designs were indeed used only by predictor finding studies (Table 2).”

In the paper

“Across all 71 included prediction studies, only 12 gave an explicit sample size calculation.”

In the paper

“It was difficult to distinguish, from study abstracts alone, between descriptive predictor finding studies and articles studying causality of a factor.”

In the paper

“In 51 of the 68 studies, predictor measurement was blinded for the outcome: in 44, simply because of the prospective design; in seven non-prospective studies, predictor measurement was explicitly blinded for the outcome.”

In the paper

“However, only 12 studies reported that they blinded the outcome measurement for predictor values.”

In the paper

“Participant recruitment was in general well described.”

In the paper

“Walter Bouwmeester and colleagues investigated the reporting and methods of prediction studies in 2008, in six high-impact general medical journals, and found that the majority of prediction studies do not follow current methodological recommendations.”

In the paper

“Length of follow-up was not reported in nine studies, leaving readers unable to know the time period for the predicted risks of the models.”

In the paper

“However, there was insufficient reporting of the number of candidate predictors in the vast majority of these studies, such that a proper estimation of the statistical power could not be made.”

In the paper

“Clinical setting was not reported in 4% of the studies.”

In the paper

“Inclusion criteria were reported in 64/71 studies.”

In the paper

“The majority of studies involved participants from a hospital setting (38%) or the general (healthy) population (27%).”

In the paper

“However, we found a similar number of model development studies (n = 14), and studies that aimed to evaluate an existing model (n = 6+3).”

In the paper

“R 2 and Brier score were reported in very few studies.”

In the paper

“Further, we could not assess statistical power or effective sample size for many studies because of inadequate reporting of the number of candidate predictors.”

In the paper

“Calibration was reported in only a few studies (Table 9).”

In the paper

“Occurrence and handling of missing values was not described or was unclear in 38% of all studies (Table 7).”

In the paper

“If reported, it was mostly reported by “missing values per predictor” (58%).”

In the paper

“If done, the Hosmer-Lemeshow statistic was the most often reported calibration measure.”

In the paper

“Other criteria, such as Akaike's Information Criterion or R 2, were used much less frequently.”

In the paper

“The method of selection of predictors within multivariable models was not described in 19% of the studies.”

In the paper

“Few studies reported adjusted (20%) or unadjusted (18%) results of the full model with all candidate predictors considered (Table 8).”

In the paper

“it was still used in 13% of the studies investigated here.”

In the paper

“We do acknowledge that various basic items are well described and reported in prediction studies, for example aim, participant selection, inclusion and exclusion criteria, and design.”

In the paper

“Loss to follow-up was reported in 46% of the studies where this was applicable.”

In the paper

“In contrast to randomized therapeutic trials, flow diagrams were hardly ever presented in prediction studies”

In the paper

“Most studies correctly reported the (adjusted) predictor effects derived from the final multivariable analyses.”

In the paper

“Univariable pre-selection and predictor selection in multivariable analyses based solely on p-values (with a <0.05 cut-off) was often used in predictor finding and model development studies.”

In the paper

“The majority of model development studies reported predictive performance in the development data only.”

In the paper

“As the reasons for missing values were insufficiently described in most studies that applied a complete case analysis, it was impossible to judge whether imputation methods would indeed have been appropriate.”

In the paper

“Several studies, however, did not rely exclusively on these methods for predictor selection, but rather also, as is recommended, included established predictors in their model regardless of statistical significance in their dataset.”

In the paper

“As the exact number of events per candidate predictor could almost never be assessed”

In the paper

“We observed much variation in the reporting of predictive performance measures.”

In the paper

“However, the methods used for further predictor selection during the statistical analyses were poorly reported.”

In the paper

“Only a few studies also reported results of the univariable analyses, which is often a useful comparator.”

In the paper

“Good reporting of how candidate predictors were pre-selected in our review compares favourably with other reviews [15],[20],[21],[29],[33],[35],[36],[48].”

What the sources say

• Not checked yetcites [15] A Laupacis, 1997, Clinical prediction rules. a review and suggested modifications of methodological standards., JAMA source
• Not checked yetcites [20] S Mallett, 2010, Reporting performance of prognostic models in cancer: a review., BMC Med source
• Not checked yetcites [21] S Mallett, 2010, Reporting methods in studies developing prognostic models in cancer: a review., BMC Med source

In the paper

“it was not possible to determine whether reported results were indeed subject to overfitting or optimistic predictive performances.”

In the paper

“The use of a combined endpoint will give problems if the predictor effect is in opposite directions for different outcomes included in the composite endpoint [52],[53].”

What the sources say

• Not checked yetcites [52] G Tomlinson, 2010, Composite end points in randomized trials: there is no free lunch., JAMA source
• Not checked yetcites [53] E Lim, 2008, Composite outcomes in cardiovascular research: a survey of randomized trials., Ann Intern Med source

In the paper

“Knowledge of the predictors might influence outcome assessment, resulting in a biased estimation of the predictor effects for the outcome [6],[17],[39].”

What the sources say

• Not checked yetcites [6] KG Moons, 2009, Prognosis and prognostic research: what, why, and how?, BMJ source
• Not checked yetcites [17] JG Lijmer, 1999, Empirical evidence of design-related bias in studies of diagnostic tests., JAMA source
• Not checked yetcites [39] PM Bossuyt, 2003, Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative., Ann Intern Med source

In the paper

“Use of poor methods may lead to biased results [2],[7],[10],[12],[13],[15],[17]–[32].”

What the sources say

• Not checked yetcites [2] DG Altman, 2007, Prognostic models: a methodological framework and review of models for breast cancer.
• Not checked yetcites [7] P Royston, 2009, Prognosis and prognostic research: developing a prognostic model., BMJ source
• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source

In the paper

“However, selective reporting of outcomes (and predictors) is often a risk [51].”

What the sources say

• Not checked yetcites [51] JJ Kirkham, 2010, The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews., BMJ source

In the paper

“Quantifying the impact of a prediction model on participant outcome requires a comparative study design [9],[48].”

What the sources say

• Not checked yetcites [9] KG Moons, 2009, Prognosis and prognostic research: application and impact of prognostic models in clinical practice., BMJ source
• Not checked yetcites [48] BM Reilly, 2006, Translating clinical research into clinical practice: impact of using prediction rules to make decisions., Ann Intern Med source

In the paper

“For time-to-event outcomes, the effective sample size is also highly related to the number of participants who experience the event [12].”

What the sources say

• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

In the paper

“However, the use of multivariable selection methods can also lead to overfitting and unstable models, especially when there are relatively few outcome events and many predictors analysed [10],[12],[13].”

What the sources say

• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
• Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source

In the paper

“Unfortunately, study registration is not mandated for prediction research, so it is generally impossible to assess whether some outcomes were analysed but not reported.”

In the paper

“Most guidelines, such as the STROBE guidelines for the reporting of observational studies or the REMARK guidelines for tumour marker prognostic studies, specifically advise investigators to report both unadjusted results (i.e., from univariable analysis, yielding the association of each candidate predictor with the outcome) and adjusted results (i.e., from a multivariable analysis) [4],[38],[63].”

What the sources say

• Not checked yetcites [4] LM McShane, 2005, Reporting recommendations for tumor marker prognostic studies (REMARK)., J Natl Cancer Inst source
• Not checked yetcites [38] E Von Elm, 2007, The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies., Lancet source
• Not checked yetcites [63] JP Vandenbroucke, 2007, Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration., PLoS Med source

In the paper

“Missing values in clinical study data rarely occur completely at random.”

What the sources say

• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
• Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
• Not checked yetcites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source
• Not checked yetcites [24] S Greenland, 1995, A critical look at methods for handling missing covariates in epidemiologic regression analyses., Am J Epidemiol source
• Not checked yetcites [57] A Burton, 2004, Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines., Br J Cancer source
• Not checked yetcites [58] MH Gorelick, 2006, Bias arising from missing data in predictive models., J Clin Epidemiol source

In the paper

“It is likely that the studies that did not or unclearly reported the method of handling missing values applied a complete case analysis as well.”

In the paper

“Even though this method is known to lead to biased results in almost all observational studies [13],[23],[60]–[62]”

What the sources say

• Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
• Not checked yetcites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source
• Not checked yetcites [60]–[62]

In the paper

“Overfitting is even more likely when the number of candidate predictors is large relative to the effective sample size [10],[12],[70].”

What the sources say

• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
• Not checked yetcites [70] JC Van Houwelingen, 1990, Predictive value of statistical models., Stat Med source

In the paper

“The extent of this optimism may be estimated with so-called internal validation techniques [10],[12],[43],[70]”

What the sources say

• Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
• Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
• Not checked yetcites [43] EW Steyerberg, 2001, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis., J Clin Epidemiol source

In the paper

“This approach may notably be problematic with low numbers of events and many candidate predictors.”

In the paper

“Accordingly, the generalisability of the performance of these reported models, especially in studies where prediction was the primary aim, is difficult to evaluate.”