CoChat
Sign Up
Cited by 600Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsMethodologies in Health Research and Practice

Reporting and Methods in Clinical Prediction Research: A Systematic Review

Open access
Bouwmeester, Walter, Zuithoff, Nicolaas P. A., Mallett, Susan, Geerlings, Mirjam I., Vergouwe, Yvonne, Steyerberg, Ewout W., Altman, Douglas G., Moons, Karel G. M.·2012·PLoS Medicine
DOI 10.1371/journal.pmed.1001221PMC PMC3358324
Open this paper in CoChat — free account

Abstract

BACKGROUND: We investigated the reporting and methods of prediction studies, focusing on aims, designs, participant selection, outcomes, predictors, statistical power, statistical methods, and predictive performance measures. METHODS AND FINDINGS: We used a full hand search to identify all prediction studies published in 2008 in six high impact general medical journals. We developed a comprehensive item list to systematically score conduct and reporting of the studies, based on recent recommendations for prediction research. Two reviewers independently scored the studies. We retrieved 71 papers for full text review: 51 were predictor finding studies, 14 were prediction model development studies, three addressed an external validation of a previously developed model, and three reported on a model's impact on participant outcome. Study design was unclear in 15% of studies, and a prospective cohort was used in most studies (60%). Descriptions of the participants and definitions of predictor and outcome were generally good. Despite many recommendations against doing so, continuous predictors were often dichotomized (32% of studies). The number of events per predictor as a measure of statistical power could not be determined in 67% of the studies; of the remainder, 53% had fewer than the commonly recommended value of ten events per predictor. Methods for a priori selection of candidate predictors were described in most studies (68%). A substantial number of studies relied on a p-value cut-off of p<0.05 to select predictors in the multivariable analyses (29%). Predictive model performance measures, i.e., calibration and discrimination, were reported in 12% and 27% of studies, respectively. CONCLUSIONS: The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review of 71 multivariable clinical prediction studies published in six high-impact general medical journals in 2008 found that most were predictor-finding studies and that key methodological and reporting recommendations were frequently not followed. Reporting was generally adequate for aims, participant selection, and predictor/outcome definitions, but poor for blinding, continuous predictors, power, missing data, multivariable result presentation, and model performance/validation.

Key findings

  • Of 1,204 hand-searched articles, 71 met inclusion criteria: 51 were predictor-finding studies, 14 were model development studies (11 without and 3 with external validation), 3 were external validation studies, and 3 were impact studies; these 71 papers studied 135 models or sets of predictors.
  • Statistical power was often inadequately reported: based on predictors in the final model, >10 events per variable occurred in 84 of 124 models, but candidate predictors were not described in the vast majority of studies; among studies clearly describing candidate predictors, effective sample size was <10 events per variable in 50% (n = 21) of presented models.
  • Missing data occurrence or handling was not described or unclear in 38% of all studies; complete case analysis was used in 65% (45/71), while multiple imputation was used in 7% (5/71) per Table 7 (the text states 8% of all studies).
  • Model performance and validation were limited: C-statistic/AUC-ROC was reported in 12% of predictor-finding and 80% of development/external-validation studies combined; calibration plots in 6% (4/66) and Hosmer-Lemeshow statistic in 9% (6/66); internal validation was performed in 33% (5/14) of model development studies, and external validation in only four studies.

Methods

  • Systematic review by hand search of the six highest-impact general medicine journals for 2008: *The New England Journal of Medicine*, *The Lancet*, *JAMA*, *Annals of Internal Medicine*, *BMJ*, and *PLoS Medicine*; original research with an abstract was eligible, and two reviewers independently assessed eligibility and extracted data, with median inter-reviewer agreement of 92% (IQR 75%–100%).
  • Included multivariable diagnostic and prognostic prediction studies predicting an outcome by two or more independent variables; excluded single predictor/test/marker studies, causality-only studies, and studies not contributing to patient care. An item list was developed from existing reporting statements/checklists (CONSORT, REMARK, STARD, STROBE, QUADAS) and prediction methodology recommendations.
  • Studies were classified into five types: predictor finding, model development without external validation, model development with external validation, external validation without or with model updating, and model impact; results were grouped by study type, medical specialty, and whether prediction was a primary or secondary aim.

Limitations

  • Stated: the review examined six high-impact journals, likely representing higher-quality studies; reporting may have improved since 2008, though the authors considered this unlikely because no major reporting guidelines for this research type had been produced recently; the work assessed reporting and statistical methods, not necessarily the appropriateness of design and conduct; conduct may be better than reported because journals limit paper length; poor reporting prevents readers from gauging relevance and reliability.
  • Stated: statistical power or effective sample size could not be assessed for many studies because of inadequate reporting of the number of candidate predictors; selective reporting of outcomes and predictors could not generally be assessed because study registration is not mandated for prediction research.
  • Evident from the text: the review is restricted to 2008 publications in six general medical journals, so findings may not generalize to other years, journals, or specialty journals; classification relied on full-text interpretation because authors generally did not explicitly state their aim, especially to distinguish predictor-finding from model-development studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pmed.1001221

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 74 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Study recruitment dates were reported in 88% of the studies.

    “Study recruitment dates were reported in 88% of the studies.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    11/68 studies had all cause mortality as the outcome, where such bias would not be a factor.

    “11/68 studies had all cause mortality as the outcome, where such bias would not be a factor.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    Effective sample size was sufficient in the majority of the external validation studies (9/13 models).

    “Given this, effective sample size was sufficient in the majority of the external validation studies (9/13 models).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Outcomes were well defined in 62/68 studies.

    “Outcomes were well defined in 62/68 studies (Table 3).”
    Quote in the paperNot a substantial claim
  • Findingown data

    A number of studies predicted a combined endpoint (14/71).

    “A number of studies predicted a combined endpoint (14/71).”
    Quote in the paperNot a substantial claim
  • May combine two claims. 37
  • Classified as probably not a substantial claim. 13

  • May combine two claims.

    Findingown data

    A randomized trial was used by two of the three impact studies; the third used an observational before-after (prospective) cohort design.

    “A randomized trial was used by two of the three impact studies; the third used an observational before-after (prospective) cohort design, comparing participant outcomes before and after the introduction of a prediction model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Of the prediction modelling studies (n = 20), the vast majority were model development studies, without (n = 11) or with (n = 3) external validation.

    “Of the prediction modelling studies (n = 20), the vast majority were model development studies, without (n = 11) or with (n = 3) external validation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Predictor interaction (non-additivity) was tested in 25 of the 51 predictor finding studies and in 11 of the 14 model development studies (total n = 36).

    “Predictor interaction (non-additivity) was tested in 25 of the 51 predictor finding studies and in 11 of the 14 model development studies (total n = 36).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Pure external validation (n = 3) and impact studies (n = 3) were rare.

    “Pure external validation (n = 3) and impact studies (n = 3) were rare.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Whether (all) consecutive participants were included or how many participants refused to participate was rarely reported and so could not be evaluated.

    “Whether (all) consecutive participants were included or how many participants refused to participate was rarely reported and so could not be evaluated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In outcome reporting, differences between studies with prediction as a primary versus a secondary aim were expected but not observed.

    “In outcome reporting, we expected differences between studies with prediction as a primary versus a secondary aim, but this was not observed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Hand searching identified 1,204 articles, of which 71 met the inclusion criteria.

    “We identified 1,204 articles by hand searching, of which 71 met the inclusion criteria (Figure 1 and Text S2).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Authors in general did not explicitly state their aim, so full text interpretations were used to classify studies as predictor finding or model development studies.

    “Authors in general did not explicitly state their aim, so we used full text interpretations to classify studies as predictor finding or model development studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Discrimination was assessed with the C-statistic or area under the ROC curve in 12% of predictor finding studies and 80% of model development and external validation studies.

    “Discrimination was assessed with the C-statistic or area under the receiver operating characteristic (ROC) curve in 12% of the predictor finding and, as expected, 80% of the model development and external validation studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority of predictor finding studies (65%) and model development studies (79%) reported predictor coefficients or effect estimates of the final model after predictor selection.

    “The majority, 65% of the predictor finding and 79% of the model development studies, reported the predictor coefficients or effect estimates of the model after predictor selection (the final model).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Multiple imputation was used in only 8% of all studies and was applied and reported in very few studies.

    “By comparison, multiple imputation, the most rigorous strategy for dealing with missing values, was used in only 8% of all studies.”
    • Also stated in: § Discussion · hedged
      “Recommended methods, such as multiple imputation, were applied and reported in very few studies, although this may be due to the fact that consensus in recommending these methods was arrived at only recently.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In descriptions of participant selection, it often remained unclear whether participants were included in an unbiased way, notably with respect to refusals and whether all consecutive eligible participants were included.

    “In descriptions of participant selection, it often remained unclear whether participants were included in an unbiased way, notably with respect to refusals and whether all consecutive eligible participants were included.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    External validation was performed in only four studies, and only a very few model development studies reported an external validation of the model in the same paper.

    “and external validation in only four studies.”
    • Also stated in: § Discussion · plain
      “Similarly, only a very few model development studies reported an external validation of the model in the same paper.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    External validation and model impact studies were rare (n = 6), and only a very small minority of papers involved external validations or impact assessments of a previously developed prediction model.

    “External validation and model impact studies were rare (n = 6).”
    • Also stated in: § Discussion · plain
      “Only a very small minority of the papers involved the most useful approaches in predicting participant clinical outcomes, namely, external validations or impact assessments of a previously developed prediction model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Study design, participant selection, definitions of outcomes and predictors, and predictor selection were generally well reported.

    “Study design, participant selection, definitions of outcomes and predictors, and predictor selection were generally well reported.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    18% of all studies investigated the added value of specific predictors, and 42% included predictors regardless of statistical significance.

    “18% of all studies investigated the added value of specific predictors, and 42% included predictors regardless of statistical significance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Complete case analysis was performed in the vast majority of studies and was by far the most commonly used approach to handle missing values.

    “Analysis of participants with complete data (i.e., complete case analysis) was performed in the vast majority of studies.”
    • Also stated in: § Discussion · plain
      “Complete case analysis was by far the most commonly used approach to handle missing values, despite many methodological recommendations to do otherwise [10],[12],[13],[23],[24],[57],[58].”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The most commonly reported criterion for predictor selection in multivariable models was a p-value of <0.05, used in 18% of all studies (and in 26% of the studies that indeed applied predictor selection in multivariable analyses).

    “The most commonly reported criterion for predictor selection in multivariable models was a p-value of <0.05, used in 18% of all studies (and in 26% of the studies that indeed applied predictor selection in multivariable analyses).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among the 71 prediction studies identified, the vast majority were predictor finding studies (n = 51), followed by model development studies (n = 14).

    “Our investigation found that among the 71 prediction studies identified, the vast majority were predictor finding studies (n = 51), followed by model development studies (n = 14).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Fourteen studies developed new prediction models, of which three included an external validation.

    “We found that 14 studies developed new prediction models (of which three included an external validation).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Three studies externally validated models, and three investigated the impact of an existing prediction model.

    “Three studies externally validated models, and three investigated the impact of an existing prediction model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Studies reported using backward selection in 17% of the primary-aim and 18% of the secondary-aim studies, whereas forward selection was reported in 6% and 0%, respectively.

    “Studies reported using backward selection in 17% of the primary-aim and 18% of the secondary-aim studies, whereas forward selection was reported in 6% and 0%, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In studies with prediction as the primary aim, the majority (71%) selected their candidate predictors based on existing literature, whereas this was less often the case (29%) in studies with prediction as a secondary aim.

    “In studies with prediction as the primary aim, the majority (71%) selected their candidate predictors based on existing literature, whereas this was less often the case (29%) in studies with prediction as a secondary aim.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Internal validation was performed in 33% (n = 5) of the 14 model development studies, and use of internal validation techniques was rare.

    “Internal validation was performed in 33% (n = 5) of the 14 model development studies”
    • Also stated in: § Discussion · plain
      “but use of these techniques was rare.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pre-selection of candidate predictors for inclusion in the multivariable analyses based on univariable predictor-outcome associations was used in 13% of the primary-aim and in 24% of the secondary-aim prediction studies.

    “Pre-selection of candidate predictors for inclusion in the multivariable analyses based on univariable predictor–outcome associations was used in 13% of the primary-aim and in 24% of the secondary-aim prediction studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Selection of candidate predictors was described for 36 (75%) of the studies where prediction was the primary aim and for eight (47%) of the studies where prediction was a secondary aim.

    “We found that selection of candidate predictors was described for 36 (75%) of the studies where prediction was the primary aim and for eight (47%) of the studies where prediction was a secondary aim (Table 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Sample size considerations, applied statistical methods, and procedures for dealing with missing data were poorly reported despite being highlighted in several reporting guidelines.

    “However, sample size considerations, applied statistical methods, and procedures for dealing with missing data were poorly reported despite being highlighted in several reporting guidelines.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.

  • May combine two claims.

    Findingnone given

    Most studies reported the occurrence of missing data but did not report sufficient detail.

    “Most studies reported the occurrence of missing data but did not report sufficient detail.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    C-statistics or area under the ROC curves were the most frequently reported discrimination statistics, whereas measures of calibration and overall measures of fit were rarely reported.

    “C-statistics or area under the ROC curves were the most frequently reported discrimination statistics, whereas measures of calibration (such as calibration slope) and overall measures of fit were rarely reported.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Encouraging findings included the frequent use of prospective studies, and adequate description of participant selection, predictor and outcome definitions, and the process for (pre)selection of candidate predictors.

    “However, encouraging findings included the frequent use of prospective studies, and adequate description of participant selection, predictor and outcome definitions, and the process for (pre)selection of candidate predictors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two authors independently extracted data, and in case of doubt items were discussed with a third and fourth reviewer.

    “Two authors (W. B., N. P. A. Z.) independently extracted the data. In case of doubt, items were discussed with a third and fourth reviewer (K. G. M. M., Y. V.).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Selection based on (significant) univariable predictor-outcome associations is not recommended, as this method increases the chance of biased results in terms of spurious predictors and overfitted and unstable models.

    “However, many methodological reports have shown that selection based on (significant) univariable predictor–outcome associations is not recommended, as this method increases the chance of biased results in terms of spurious predictors and overfitted and unstable models [12],[13],[43],[56].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

      Not checked in the free run.

    • cites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source

      Not checked in the free run.

    • cites [43] EW Steyerberg, 2001, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis., J Clin Epidemiol source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    To externally validate a prediction model, a minimum effective sample size of 100 participants with and 100 without the event has been recommended.

    “To externally validate a prediction model, a minimum effective sample size of 100 participants with and 100 without the event has been recommended [55].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [55] Y Vergouwe, 2005, Substantial effective sample sizes were required for external validation studies of predictive logistic regression models., J Clin Epidemiol source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A frequently mentioned rule of thumb is 10 events needed per candidate predictor.

    “A frequently mentioned rule of thumb is “10 events needed per candidate predictor” [12],[25],[26],[30],[54].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

      Not checked in the free run.

    • cites [25] P Peduzzi, 1996, A simulation study of the number of events per variable in logistic regression analysis., J Clin Epidemiol source

      Not checked in the free run.

    • cites [26] P Peduzzi, 1995, Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates., J Clin Epidemiol source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Analyzing continuous, categorical, and time-to-event data as binary outcomes is not recommended because less accurate predictions are likely to result.

    “a practice that is not recommended because less accurate predictions are likely to result, as with dichotomizing predictor variables [50].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [50] P Royston, 2006, Dichotomizing continuous predictors in multiple regression: a bad idea., Stat Med source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Adequate reporting of predictor selection methods used is important, because the number of candidate predictors and how they were selected can influence the specific predictors included in the final multivariable model and affect interpretation of results.

    “Adequate reporting of predictor selection methods used is important, because the number of candidate predictors and how they were selected at various stages of the study can both influence the specific predictors included in the final multivariable model, and thus affect the interpretation of the results 12,13,43,56.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

      Not checked in the free run.

    • cites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source

      Not checked in the free run.

    • cites [43] EW Steyerberg, 2001, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis., J Clin Epidemiol source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The two reviewers agreed on a median of 92% (interquartile range 75%-100%) of the items extracted.

    “The two reviewers agreed on a median of 92% (interquartile range 75%–100%) of the items extracted.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two authors independently assessed eligibility of studies, and a third independent reader was involved in case of doubt.

    “The full text of all thus selected studies was obtained, and two authors (W. B. and N. P. A. Z.) independently assessed eligibility; in case of doubt a third independent reader was involved (K. G. M. M. or Y. V.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Inter-reviewer agreement on data extraction was assessed by calculating the percentage of overall agreement between the two reviewers.

    “The inter-reviewer agreement on the data extraction was assessed by calculating the percentage of overall agreement between the two reviewers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Multiple imputation is often advocated to preserve power and obtain less biased results, assuming the reason for missing data is not entirely due to non-observed information (data are not missing not at random).

    “Imputation, notably multiple imputation, of missing values is often advocated to preserve power and obtain less biased results, on the assumption that the reason for the missing data is not entirely due to non-observed information (i.e., data are not “missing not at random”).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The reason for missing data is not entirely due to non-observed information (data are not missing not at random) when multiple imputation is used.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Various reports have indicated that in prediction modelling research there is an unfortunate practice of developing new models instead of externally validating or updating existing models.

    “Various reports have indicated that in prediction modelling research there is an unfortunate practice of developing new models instead of externally validating or updating existing models [8]–[10],[15],[29],[36],[48].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [8]–[10]

      Not checked in the free run.

    • cites [15] A Laupacis, 1997, Clinical prediction rules. a review and suggested modifications of methodological standards., JAMA source

      Not checked in the free run.

    • cites [29] JH Wasson, 1985, Clinical prediction rules. Applications and methodological standards., N Engl J Med source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Exclusion of participants with missing values often leads to loss of statistical power and biased results.

    “Exclusion of participants with missing values will therefore not only lead to loss of statistical power, but often also to biased results 10,12,13,23,24,57,58.”
    Quote in the paperSubstantial claim0 of 7 sources checked
    • cites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source

      Not checked in the free run.

    • cites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

      Not checked in the free run.

    • cites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source

      Not checked in the free run.

    • cites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source

      Not checked in the free run.

    • cites [24] S Greenland, 1995, A critical look at methods for handling missing covariates in epidemiologic regression analyses., Am J Epidemiol source

      Not checked in the free run.

    • cites [57] A Burton, 2004, Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines., Br J Cancer source

      Not checked in the free run.

    • cites [58] MH Gorelick, 2006, Bias arising from missing data in predictive models., J Clin Epidemiol source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    These basic items have been identified as important in several well-known guidelines for reporting of clinical research.

    “These items have been identified as important in several well-known guidelines for reporting of clinical research [4],[38]–[41].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [4] LM McShane, 2005, Reporting recommendations for tumor marker prognostic studies (REMARK)., J Natl Cancer Inst source

      Not checked in the free run.

    • cites [38]–[41]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Poor reporting of sample size rationale has also been observed by others.

    “Poor reporting of sample size rationale has also been observed by others [3],[21],[68].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3] DG Altman, 1998, Methodological challenges in the evaluation of prognostic factors in breast cancer., Breast Cancer Res Treat source

      Not checked in the free run.

    • cites [21] S Mallett, 2010, Reporting methods in studies developing prognostic models in cancer: a review., BMC Med source

      Not checked in the free run.

    • cites [68] LM Bachmann, 2006, Sample sizes of studies on diagnostic accuracy: literature survey., BMJ source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The review identified poor reporting and poor methods in many published prediction studies, which limits the reliability and applicability of the published findings.

    “To conclude, we identified poor reporting and poor methods in many published prediction studies, which limits the reliability and applicability of the published findings.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 71 prediction studies published in six high-impact general medical journals in 2008 are representative of current clinical prediction research.
    • Rests on an assumption: The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.

  • May combine two claims.

    Premisereasoning

    Apparent model performance is generally too optimistic, as the model has been tailored to the dataset at hand.

    “This apparent model performance, however, is generally too optimistic, as the model has been tailored to the dataset at hand.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source

      Not checked in the free run.

    • cites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis

      Not checked in the free run.

    • cites [70] JC Van Houwelingen, 1990, Predictive value of statistical models., Stat Med source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The REMARK guidelines for prognostic tumour marker studies recommend using a REMARK profile table instead of a flow diagram.

    “The REMARK guidelines for prognostic tumour marker studies recommend using a REMARK profile table instead of a flow diagram [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] S Mallett, 2010, Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines., Br J Cancer source

      Not checked in the free run.

47 of 47

  • The paper relies on this without support.

    Study registration is not mandated for prediction research.

    “study registration is not mandated for prediction research”

  • The authors state this as an assumption.

    The reason for missing data is not entirely due to non-observed information (data are not missing not at random) when multiple imputation is used.

    “on the assumption that the reason for the missing data is not entirely due to non-observed information (i.e., data are not “missing not at random”).”

  • The paper relies on this without support.

    The 71 prediction studies published in six high-impact general medical journals in 2008 are representative of current clinical prediction research.

    “We assessed the reporting and methods in all clinical prediction studies published in six high-impact general medical journals in 2008.”
    “We have described the state of current prediction research, and highlighted aspects that clearly need improvement.”

  • The paper relies on this without support.

    The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.

    “We note that our work assessed researchers' reporting and statistical methods in modelling, and not necessarily the appropriateness of the design and conduct of the studies.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Predictor-combination total does not equal subgroup counts

    Minor

    Table 6 lists eight primary-aim and one secondary-aim study combining similar predictors, but reports a total of seven rather than nine.

    Reported 7 · recomputed 9

  • Missing-data reporting percentage does not match its count

    Minor

    Table 7 reports 18 of 48 primary-aim studies as not reporting missing data clearly. That is 37.5%, not 35%.

    Reported 35 · recomputed 37.5

  • Time-to-event outcome percentage uses an inconsistent denominator

    Minor

    Table 3 covers 68 studies and lists 30 with time-to-event outcomes. That is 44%, not the reported 48%; the other outcome-type percentages use 68 as their denominator.

    Reported 48 · recomputed 44.117647

  • Table 6 note mislabels the study-aim groups

    Minor

    The Table 6 note calls the groups of 48 and 17 predictor-finding and model-development studies. Elsewhere, those study types are counted as 51 and 14; 48 and 17 are the primary- and secondary-aim groups.

  • Change-in-C-statistic total does not equal subgroup counts

    Minor

    Table 6 reports five primary-aim studies and zero secondary-aim studies using this criterion, but gives a total of six.

    Reported 6 · recomputed 5

5 of 5

Cites

Flagged first, then by citations

3 of the 88 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M.
Affiliations
  • University Medical Center Utrecht (NL)
  • University of Oxford (GB)
  • Erasmus MC (NL)
Corresponding author
Moons, Karel G. M.
ORCID
Venue
PLoS Medicine
Publisher
Public Library of Science (PLoS)
Published
2012-05-22
Volume
9
Issue
5
Pages
e1001221
ISSN
  • 1549-1277 (ISSN-L)
  • 1549-1676
DOI
10.1371/journal.pmed.1001221
PMC
PMC3358324
OpenAlex
W1998392635
Semantic Scholar
2d86e73e56aa4394fe4ac9a6ed742d850abc73ce
Cited by
600
References
88
Citations by year
2026: 24 · 2025: 39 · 2024: 36 · 2023: 47 · 2022: 65 · 2021: 50 · 2020: 65 · 2019: 36 · 2018: 40 · 2017: 43 · 2016: 40 · 2015: 46 · 2014: 38 · 2013: 24 · 2012: 7
FWCI
5.61
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Health Professions › Health Information Management › Methodologies in Health Research and Practice
Keywords
prediction model development, reporting quality, methodological quality, predictor selection, calibration, discrimination (statistics), external validation, prospective cohort, general medical journals
MeSH terms
Forecasting, Humans, Periodicals as Topic, Prospective Studies, Research Design, Biomedical Research, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review