The DOI matches its Crossref record; 83 of the 88 references it cites were found in a registry or an index; 2 of the 74 sources its claims cite were checked for support.
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Open accessAbstract
BACKGROUND: We investigated the reporting and methods of prediction studies, focusing on aims, designs, participant selection, outcomes, predictors, statistical power, statistical methods, and predictive performance measures. METHODS AND FINDINGS: We used a full hand search to identify all prediction studies published in 2008 in six high impact general medical journals. We developed a comprehensive item list to systematically score conduct and reporting of the studies, based on recent recommendations for prediction research. Two reviewers independently scored the studies. We retrieved 71 papers for full text review: 51 were predictor finding studies, 14 were prediction model development studies, three addressed an external validation of a previously developed model, and three reported on a model's impact on participant outcome. Study design was unclear in 15% of studies, and a prospective cohort was used in most studies (60%). Descriptions of the participants and definitions of predictor and outcome were generally good. Despite many recommendations against doing so, continuous predictors were often dichotomized (32% of studies). The number of events per predictor as a measure of statistical power could not be determined in 67% of the studies; of the remainder, 53% had fewer than the commonly recommended value of ten events per predictor. Methods for a priori selection of candidate predictors were described in most studies (68%). A substantial number of studies relied on a p-value cut-off of p<0.05 to select predictors in the multivariable analyses (29%). Predictive model performance measures, i.e., calibration and discrimination, were reported in 12% and 27% of studies, respectively. CONCLUSIONS: The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic review of 71 multivariable clinical prediction studies published in six high-impact general medical journals in 2008 found that most were predictor-finding studies and that key methodological and reporting recommendations were frequently not followed. Reporting was generally adequate for aims, participant selection, and predictor/outcome definitions, but poor for blinding, continuous predictors, power, missing data, multivariable result presentation, and model performance/validation.
Key findings
- Of 1,204 hand-searched articles, 71 met inclusion criteria: 51 were predictor-finding studies, 14 were model development studies (11 without and 3 with external validation), 3 were external validation studies, and 3 were impact studies; these 71 papers studied 135 models or sets of predictors.
- Statistical power was often inadequately reported: based on predictors in the final model, >10 events per variable occurred in 84 of 124 models, but candidate predictors were not described in the vast majority of studies; among studies clearly describing candidate predictors, effective sample size was <10 events per variable in 50% (n = 21) of presented models.
- Missing data occurrence or handling was not described or unclear in 38% of all studies; complete case analysis was used in 65% (45/71), while multiple imputation was used in 7% (5/71) per Table 7 (the text states 8% of all studies).
- Model performance and validation were limited: C-statistic/AUC-ROC was reported in 12% of predictor-finding and 80% of development/external-validation studies combined; calibration plots in 6% (4/66) and Hosmer-Lemeshow statistic in 9% (6/66); internal validation was performed in 33% (5/14) of model development studies, and external validation in only four studies.
Methods
- Systematic review by hand search of the six highest-impact general medicine journals for 2008: *The New England Journal of Medicine*, *The Lancet*, *JAMA*, *Annals of Internal Medicine*, *BMJ*, and *PLoS Medicine*; original research with an abstract was eligible, and two reviewers independently assessed eligibility and extracted data, with median inter-reviewer agreement of 92% (IQR 75%–100%).
- Included multivariable diagnostic and prognostic prediction studies predicting an outcome by two or more independent variables; excluded single predictor/test/marker studies, causality-only studies, and studies not contributing to patient care. An item list was developed from existing reporting statements/checklists (CONSORT, REMARK, STARD, STROBE, QUADAS) and prediction methodology recommendations.
- Studies were classified into five types: predictor finding, model development without external validation, model development with external validation, external validation without or with model updating, and model impact; results were grouped by study type, medical specialty, and whether prediction was a primary or secondary aim.
Limitations
- Stated: the review examined six high-impact journals, likely representing higher-quality studies; reporting may have improved since 2008, though the authors considered this unlikely because no major reporting guidelines for this research type had been produced recently; the work assessed reporting and statistical methods, not necessarily the appropriateness of design and conduct; conduct may be better than reported because journals limit paper length; poor reporting prevents readers from gauging relevance and reliability.
- Stated: statistical power or effective sample size could not be assessed for many studies because of inadequate reporting of the number of candidate predictors; selective reporting of outcomes and predictors could not generally be assessed because study registration is not mandated for prediction research.
- Evident from the text: the review is restricted to 2008 publications in six general medical journals, so findings may not generalize to other years, journals, or specialty journals; classification relied on full-text interpretation because authors generally did not explicitly state their aim, especially to distinguish predictor-finding from model-development studies.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Flowchart of included studies. aThe hand search included only studies with an abstract, published in 2008 in The New England Journal of Medicine, The Lancet, JAMA: the Journal of the American Medical Association, Annals of Internal Medicine, BMJ, and PLoS Medicine. The following publication types were excluded beforehand: editorials, bibliographies, biographies, comments, dictionaries, directories, festschrifts, interviews, letters, news, and periodical indexes. bStudies, generally conducted in a yet healthy population, aimed at quantifying a causal relationship between a particular determinant or risk factor and an outcome, adjusting for other risk factors (i.e., confounders). cFor example, see [72].](/paper-figures/4c846bb7a9c74f7e8834bd85f2216499/reporting-and-methods-in-clinical-prediction-research-a-systematic-review-figure-1.jpg)
Verification
Claims the paper makes
71 Own result · 26 Not checked yet · 12 No basis given · 5 Own interpretation · 4 Own method · 1 Partly supported · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Commonly missing values are related to observed participant or disease characteristics.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
What the sources say
- Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
- Supportedcites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source
Section 2, “Types of missing data,” says the probability of missingness commonly depends on observed patient characteristics. It gives disease status as an example of such an observed characteristic, directly supporting the claim. (LLM, conf=0.94)
- Not checked yetcites [24] S Greenland, 1995, A critical look at methods for handling missing covariates in epidemiologic regression analyses., Am J Epidemiol source
- Partly supportedcites [57] A Burton, 2004, Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines., Br J Cancer sourceExperimental — accuracy not yet measured
The Results section ('Exploration of the missing data mechanism'; page not provided) reports differences between patients with complete and missing data in seven articles, including comparisons of patient or clinical characteristics. But only 10 of the 100 reviewed articles examined such differences, so the paper does not establish that missingness is commonly related to observed participant or disease characteristics across studies. (LLM, conf=0.85)
- Not checked yetcites [58] MH Gorelick, 2006, Bias arising from missing data in predictive models., J Clin Epidemiol source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Study recruitment dates were reported in 88% of the studies.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Participant Recruitment, Follow-Up, and Setting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“11/68 studies had all cause mortality as the outcome, where such bias would not be a factor.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Given this, effective sample size was sufficient in the majority of the external validation studies (9/13 models).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Statistical Power
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Outcomes were well defined in 62/68 studies (Table 3).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A number of studies predicted a combined endpoint (14/71).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the studies that clearly described the number of candidate predictors, the effective sample size was <10 events per variable in 50% (n = 21) of the presented models.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Statistical Power
plainn = 21- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Prediction of more than one outcome was very common in predictor finding studies, apparently because of their exploratory aim (Table S1).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were few multivariable diagnostic studies (n = 5).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Study Aims
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Description of the cohort characteristics was clear in 68/69 of the relevant prediction studies (not applicable for two non-nested case-control studies).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Participant Recruitment, Follow-Up, and Setting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 13
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A randomized trial was used by two of the three impact studies; the third used an observational before-after (prospective) cohort design, comparing participant outcomes before and after the introduction of a prediction model.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Design of Participant Sampling
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the prediction modelling studies (n = 20), the vast majority were model development studies, without (n = 11) or with (n = 3) external validation.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Study Aims
plainn = 20- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Predictor interaction (non-additivity) was tested in 25 of the 51 predictor finding studies and in 11 of the 14 model development studies (total n = 36).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Candidate Predictors
plainn = 36- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pure external validation (n = 3) and impact studies (n = 3) were rare.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Study Aims
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Whether (all) consecutive participants were included or how many participants refused to participate was rarely reported and so could not be evaluated.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Participant Recruitment, Follow-Up, and Setting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In outcome reporting, we expected differences between studies with prediction as a primary versus a secondary aim, but this was not observed.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We identified 1,204 articles by hand searching, of which 71 met the inclusion criteria (Figure 1 and Text S2).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Authors in general did not explicitly state their aim, so we used full text interpretations to classify studies as predictor finding or model development studies.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Data Extraction and Reviewer Agreement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Discrimination was assessed with the C-statistic or area under the receiver operating characteristic (ROC) curve in 12% of the predictor finding and, as expected, 80% of the model development and external validation studies.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Model Performance and Internal and External Validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority, 65% of the predictor finding and 79% of the model development studies, reported the predictor coefficients or effect estimates of the model after predictor selection (the final model).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Presentation of Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By comparison, multiple imputation, the most rigorous strategy for dealing with missing values, was used in only 8% of all studies.”
- Also stated in: § Discussion · hedged
“Recommended methods, such as multiple imputation, were applied and reported in very few studies, although this may be due to the fact that consensus in recommending these methods was arrived at only recently.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Discussion · hedged
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In descriptions of participant selection, it often remained unclear whether participants were included in an unbiased way, notably with respect to refusals and whether all consecutive eligible participants were included.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and external validation in only four studies.”
- Also stated in: § Discussion · plain
“Similarly, only a very few model development studies reported an external validation of the model in the same paper.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Model Performance and Internal and External Validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“External validation and model impact studies were rare (n = 6).”
- Also stated in: § Discussion · plain
“Only a very small minority of the papers involved the most useful approaches in predicting participant clinical outcomes, namely, external validations or impact assessments of a previously developed prediction model.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plainn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Study design, participant selection, definitions of outcomes and predictors, and predictor selection were generally well reported.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“18% of all studies investigated the added value of specific predictors, and 42% included predictors regardless of statistical significance.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis of participants with complete data (i.e., complete case analysis) was performed in the vast majority of studies.”
- Also stated in: § Discussion · plain
“Complete case analysis was by far the most commonly used approach to handle missing values, despite many methodological recommendations to do otherwise [10],[12],[13],[23],[24],[57],[58].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most commonly reported criterion for predictor selection in multivariable models was a p-value of <0.05, used in 18% of all studies (and in 26% of the studies that indeed applied predictor selection in multivariable analyses).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our investigation found that among the 71 prediction studies identified, the vast majority were predictor finding studies (n = 51), followed by model development studies (n = 14).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plainn = 51- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that 14 studies developed new prediction models (of which three included an external validation).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plainn = 14- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three studies externally validated models, and three investigated the impact of an existing prediction model.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Studies reported using backward selection in 17% of the primary-aim and 18% of the secondary-aim studies, whereas forward selection was reported in 6% and 0%, respectively.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In studies with prediction as the primary aim, the majority (71%) selected their candidate predictors based on existing literature, whereas this was less often the case (29%) in studies with prediction as a secondary aim.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Internal validation was performed in 33% (n = 5) of the 14 model development studies”
- Also stated in: § Discussion · plain
“but use of these techniques was rare.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Model Performance and Internal and External Validity
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pre-selection of candidate predictors for inclusion in the multivariable analyses based on univariable predictor–outcome associations was used in 13% of the primary-aim and in 24% of the secondary-aim prediction studies.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that selection of candidate predictors was described for 36 (75%) of the studies where prediction was the primary aim and for eight (47%) of the studies where prediction was a secondary aim (Table 6).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, sample size considerations, applied statistical methods, and procedures for dealing with missing data were poorly reported despite being highlighted in several reporting guidelines.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Most studies reported the occurrence of missing data but did not report sufficient detail.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“C-statistics or area under the ROC curves were the most frequently reported discrimination statistics, whereas measures of calibration (such as calibration slope) and overall measures of fit were rarely reported.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, encouraging findings included the frequent use of prospective studies, and adequate description of participant selection, predictor and outcome definitions, and the process for (pre)selection of candidate predictors.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two authors (W. B., N. P. A. Z.) independently extracted the data. In case of doubt, items were discussed with a third and fourth reviewer (K. G. M. M., Y. V.).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Methods > Data Extraction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“However, many methodological reports have shown that selection based on (significant) univariable predictor–outcome associations is not recommended, as this method increases the chance of biased results in terms of spurious predictors and overfitted and unstable models [12],[13],[43],[56].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
What the sources say
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
- Not checked yetcites [43] EW Steyerberg, 2001, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis., J Clin Epidemiol source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To externally validate a prediction model, a minimum effective sample size of 100 participants with and 100 without the event has been recommended [55].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Statistical Power
What the sources say
- Not checked yetcites [55] Y Vergouwe, 2005, Substantial effective sample sizes were required for external validation studies of predictive logistic regression models., J Clin Epidemiol source
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“A frequently mentioned rule of thumb is “10 events needed per candidate predictor” [12],[25],[26],[30],[54].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Statistical Power
What the sources say
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [25] P Peduzzi, 1996, A simulation study of the number of events per variable in logistic regression analysis., J Clin Epidemiol source
- Not checked yetcites [26] P Peduzzi, 1995, Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates., J Clin Epidemiol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“a practice that is not recommended because less accurate predictions are likely to result, as with dichotomizing predictor variables [50].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
What the sources say
- Not checked yetcites [50] P Royston, 2006, Dichotomizing continuous predictors in multiple regression: a bad idea., Stat Med source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Adequate reporting of predictor selection methods used is important, because the number of candidate predictors and how they were selected at various stages of the study can both influence the specific predictors included in the final multivariable model, and thus affect the interpretation of the results 12,13,43,56.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
What the sources say
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
- Not checked yetcites [43] EW Steyerberg, 2001, Internal validation of predictive models: efficiency of some procedures for logistic regression analysis., J Clin Epidemiol source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The two reviewers agreed on a median of 92% (interquartile range 75%–100%) of the items extracted.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Data Extraction and Reviewer Agreement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The full text of all thus selected studies was obtained, and two authors (W. B. and N. P. A. Z.) independently assessed eligibility; in case of doubt a third independent reader was involved (K. G. M. M. or Y. V.).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Methods > Literature Search
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The inter-reviewer agreement on the data extraction was assessed by calculating the percentage of overall agreement between the two reviewers.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Methods > Data Extraction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Imputation, notably multiple imputation, of missing values is often advocated to preserve power and obtain less biased results, on the assumption that the reason for the missing data is not entirely due to non-observed information (i.e., data are not “missing not at random”).”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The reason for missing data is not entirely due to non-observed information (data are not missing not at random) when multiple imputation is used.
None of the sources cited for this claim have been checked yet.
In the paper
“Various reports have indicated that in prediction modelling research there is an unfortunate practice of developing new models instead of externally validating or updating existing models [8]–[10],[15],[29],[36],[48].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
What the sources say
- Not checked yetcites [8]–[10]
- Not checked yetcites [15] A Laupacis, 1997, Clinical prediction rules. a review and suggested modifications of methodological standards., JAMA source
- Not checked yetcites [29] JH Wasson, 1985, Clinical prediction rules. Applications and methodological standards., N Engl J Med source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Exclusion of participants with missing values will therefore not only lead to loss of statistical power, but often also to biased results 10,12,13,23,24,57,58.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
What the sources say
- Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [13] FE Harrell Jr, 1996, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors., Stat Med source
- Not checked yetcites [23] AR Donders, 2006, Review: a gentle introduction to imputation of missing values., J Clin Epidemiol source
- Not checked yetcites [24] S Greenland, 1995, A critical look at methods for handling missing covariates in epidemiologic regression analyses., Am J Epidemiol source
- Not checked yetcites [57] A Burton, 2004, Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines., Br J Cancer source
- Not checked yetcites [58] MH Gorelick, 2006, Bias arising from missing data in predictive models., J Clin Epidemiol source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“These items have been identified as important in several well-known guidelines for reporting of clinical research [4],[38]–[41].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
What the sources say
- Not checked yetcites [4] LM McShane, 2005, Reporting recommendations for tumor marker prognostic studies (REMARK)., J Natl Cancer Inst source
- Not checked yetcites [38]–[41]
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Poor reporting of sample size rationale has also been observed by others [3],[21],[68].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
What the sources say
- Not checked yetcites [3] DG Altman, 1998, Methodological challenges in the evaluation of prognostic factors in breast cancer., Breast Cancer Res Treat source
- Not checked yetcites [21] S Mallett, 2010, Reporting methods in studies developing prognostic models in cancer: a review., BMC Med source
- Not checked yetcites [68] LM Bachmann, 2006, Sample sizes of studies on diagnostic accuracy: literature survey., BMJ source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To conclude, we identified poor reporting and poor methods in many published prediction studies, which limits the reliability and applicability of the published findings.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The 71 prediction studies published in six high-impact general medical journals in 2008 are representative of current clinical prediction research.
- Rests on an assumption: The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.
None of the sources cited for this claim have been checked yet.
In the paper
“This apparent model performance, however, is generally too optimistic, as the model has been tailored to the dataset at hand.”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
What the sources say
- Not checked yetcites [10] EW Steyerberg, 2009, Clinical prediction models: a practical approach to development, validation, and updating source
- Not checked yetcites [12] FE Harrell, 2001, Regression modeling strategies with applications to linear models, logistic regression and survival analysis
- Not checked yetcites [70] JC Van Houwelingen, 1990, Predictive value of statistical models., Stat Med source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The REMARK guidelines for prognostic tumour marker studies recommend using a REMARK profile table instead of a flow diagram [69].”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Discussion
What the sources say
- Not checked yetcites [69] S Mallett, 2010, Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines., Br J Cancer source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Study registration is not mandated for prediction research.
“study registration is not mandated for prediction research”
The authors state this as an assumption.
The reason for missing data is not entirely due to non-observed information (data are not missing not at random) when multiple imputation is used.
“on the assumption that the reason for the missing data is not entirely due to non-observed information (i.e., data are not “missing not at random”).”
The paper relies on this without support.
The 71 prediction studies published in six high-impact general medical journals in 2008 are representative of current clinical prediction research.
“We assessed the reporting and methods in all clinical prediction studies published in six high-impact general medical journals in 2008.”
“We have described the state of current prediction research, and highlighted aspects that clearly need improvement.”
The paper relies on this without support.
The statistical methods reported in the published prediction studies reflect the methods actually used in those studies.
“We note that our work assessed researchers' reporting and statistical methods in modelling, and not necessarily the appropriateness of the design and conduct of the studies.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bouwmeester and colleagues [23] evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [23]
In the citing paper
“Many reviews have showed that the quality of reporting in published articles describing the development or validation of multivariable prediction models in medicine is poor [23–27,29].”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [23]
In the citing paper
“Unfortunately, extraction of calibration measures is often hampered by poor assessment and reporting.11,16”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [16]
In the citing paper
“In a meta-analysis without access to individual participant data (IPD), we are reliant on extracting these performance measures from study publications. However, they are often poorly reported.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [16]
In the citing paper
“For instance, validation studies rarely present information on the calibration intercept and slope, or different studies report estimates for different risk strata or time horizons.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [16]
In the citing paper
“However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor.”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [23]
In the citing paper
“Bouwmeester and colleagues23 evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [23]
In the citing paper
“Many reviews have showed that the quality of reporting in published articles describing the development or validation of multivariable prediction models in medicine is poor.23–27,29”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Similar predictors combinedd | 17 (8) | 6 (1) | 11 (7) |”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
Reported 7 · recomputed 9
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Not reported or unclear | 35 (18) | 53 (9) | 0 (0) | 0 (0) | 38 (27) |”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Missing Values
Reported 35 · recomputed 37.5
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Time to event | 48 (30) |”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Outcome
Reported 48 · recomputed 44.117647
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
“48 predictor finding studies, 17 model development studies, or 65 total”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Presentation of Results
“| Type of Result Presented | Predictor Finding Studies (n = 51) | Development Studies (n = 14) | Total (n = 65)a |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Change in C-statistic | 10; 14 (5) | 0; 0 (0) | 9; 13 (6) |”
§ Reporting and Methods in Clinical Prediction Research: A Systematic Review > Results > Selection of Candidate and Final Predictors
Reported 6 · recomputed 5
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
Self-citationA simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Self-citationVandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613 · cited 1×
“, from a multivariable analysis) [4,38,63].”
Self-citationRelaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
Vittinghoff, E., & McCulloch, C. E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwk052 · Cited by 4,359
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Vandenbroucke, J. P., Elm, E. V., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010-w1 · Cited by 3,543
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Self-citation
Cited by
Sorted by citations
QuPath: Open source software for digital pathology image analysis
Bankhead, P., Loughrey, M. B., Fernández, J. A., Dombrowski, Y., McArt, D. G., Dunne, P. D., McQuaid, S., Gray, R. T., Murray, L. J., Coleman, H. G., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-17204-5 · Cited by 8,908
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
“Bouwmeester and colleagues (23) evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
Self-citationCollins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationCollins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,319
“…have showed that the quality of reporting in published articles describing the development or validation of multivariable prediction models in medicine is poor (Laupacis et al, 1997; Ettema et al, 2010; Mallett et al, 2010; Collins et al, 2011; Bouwmeester et al, 2012; Collins et al, 2013, 2014).”
InfluentialSelf-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
“Bouwmeester and colleagues [23] evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
BackgroundSelf-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
“(1) Predictor finding studies (also known as risk factor or prognostic factor studies) aim to identify which predictors independently contribute to the prediction of a diagnostic or prognostic outcome.(1, 6) Prediction model studies typically aim to develop, validate or adjust (e.”
InfluentialSelf-citationCalibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
“Systematic reviews have found that calibration is assessed far less often than discrimination [2, 3, 5 – 7], which is problematic since poor calibration can make predictions misleading [8].”
BackgroundSelf-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
“Continuous or categorical predictors are frequently dichotomised for the analysis [2,42,56,60,78] despite strong evidence and recommendations to the contrary [92–94].”
BackgroundMethodInfluentialSelf-citationTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
“least reported poorly (4) (5) (6) (7).”
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
“These and other reviews provide a clear picture that, across different disease areas and different journals, there is a generally poor level of reporting of prediction model studies [6,2-27,29].”
InfluentialSelf-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationMa, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648
Reducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
“15 28 29-31 The full list of extracted items is available on request.”
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Surgery · doi:10.1002/bjs.9736 · Cited by 889
Self-citationCollins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852
Self-citationSupervised Machine Learning: A Brief Primer
Jiang, T., Gradus, J. L., & Rosellini, A. J. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.05.002 · Cited by 953
“Finally, one underreported metric of model performance is calibration, or goodness of fit (Bouwmeester et al., 2012).”
BackgroundSample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K. G. M., de Groot, J. A. H., Collins, P. G. S., Altman, D. G., Eijkemans, M. J. C., & Reitsma, J. B. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 820
“Binary logistic regression modeling is among the most frequently used approaches for developing multivariable clinical prediction models for binary outcomes.(1,2) Two major categories are: diagnostic prediction models that estimate the probability of a target disease being currently present versus not present; and pro…”
MethodSelf-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793
“Bouwmeester and colleagues (23) evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
Self-citationA guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J. A., Collins, P. G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 771
Self-citationReporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701
Self-citationCollins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729
“Both components are extremely important and should be reported for all studies evaluating the performance of a prediction model, yet calibration, which assesses how close the prediction for an individual is to their true risk, is inexplicably rarely reported, as observed in this and other reviews [1,23,47].”
BackgroundMethodResultInfluentialSelf-citationA guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657
“12 27 Therefore, we propose to extract information on the total number of observed (O) and expected (E) events, which are statistics most likely to be reported or derivable (appendix 7).”
BackgroundSelf-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585
“21 23 A major problem of external validation studies is that they are often based on small and local datasets.”
BackgroundSelf-citationDebray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597
“Differences in study-level characteristics (such as inclusion and exclusion criteria, details on subject recruitment, or study design choices) may provide additional insights into achieved model performance and could therefore further improve the interpretation of prediction model validation results (step 3) [47].”
Self-citationLogistic regression was as good as machine learning for predicting major chronic diseases
Nusinovici, S., Tham, Y. C., Yan, M. Y. C., Ting, D. S. W., Li, J., Sabanayagam, C., Wong, T. Y., & Cheng, C. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.002 · Cited by 600
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 474
“Recently published systematic reviews of studies in prognostic research field have raised concerns about the proper validation and reporting strategies of new risk models [2,75,76].”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439
Self-citationPavlou, M., Ambler, G., Seaman, S. R., Iorio, M. D., & Omar, R. (2015) · Statistics in Medicine · doi:10.1002/sim.6782 · Cited by 441
“[11], and is useful for quantifying the amount of information in the data relative to model complexity.”
BackgroundAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415
“…publications have raised concern about the methodological quality of prediction models developed with conventional statistical techniques, 6 17 18 a formal methodological and risk of bias assessment of supervised machine learning based prediction model studies across all medical disciplines…”
BackgroundSelf-citationGerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P., & Collins, P. G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 415
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371
Shaikhina, T., Lowe, D., Daga, S., Briggs, D., Higgins, R. M., & Khovanova, N. A. (2017) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2017.01.012 · Cited by 370
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.010 · Cited by 336
“Bouwmeester and colleagues [23] evaluated 71 reports, published in 2008 in 6 high-impact general medical journals, and likewise observed an overwhelmingly poor level of reporting.”
Self-citationGuo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00314-2 · Cited by 345
“Systematic reviews of prediction models are a new and evolving area and are increasingly undertaken to systematically identify, appraise, and summarize evidence on the performance of prediction models 71 – 73 .”
BackgroundCalibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278
“However, they are often poorly reported.(11,16) We now discuss how to retrieve the necessary but not explicitly reported predictive performance estimates from the primary prediction model (validation) studies using other reported data.”
BackgroundMethodSelf-citationCowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 271
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Self-citationDonzé, J. D., Williams, M. V., Robinson, E. J., Zimlichman, E., Aujesky, D., Vasilevskis, E. E., Kripalani, S. B., Metlay, J. P., Wallington, T., Fletcher, G. S., et al. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.8462 · Cited by 250
Fundamentals of Clinical Data Science
Kubben, P., Dumontier, M., & Dekker, A. L. A. J. (2018) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 258
Guide to presenting clinical prediction models for use in clinical settings
Bonnett, L. J., Snell, K. I. E., Collins, P. G. S., & Riley, R. D. (2019) · BMJ · doi:10.1136/bmj.l737 · Cited by 235
Vistisen, D., Andersen, G. S., Hansen, C. S., Hulman, A., Henriksen, J. E., Bech-Nielsen, H., & Jørgensen, M. E. (2016) · Circulation · doi:10.1161/circulationaha.115.018844 · Cited by 229
“In addition, we used clinical judgment, as is recommended, to choose between highly correlated predictors of equal predictive power.(29) Before analysis, predictors with a highly skewed distribution (triglycerides, eGFR and TSH) were log-transformed to improve model calibration.”
BackgroundMethodInfluentialDiagnostic and prognostic prediction models
Hendriksen, J. M. T., Geersing, G. J., Moons, K. G. M., & de Groot, J. A. H. (2013) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12262 · Cited by 229
Self-citationRisk Prediction Models for Patients With Chronic Kidney Disease
Tangri, N., Kitsios, G. D., Inker, L. A., Griffith, J. L., Naimark, D. M. J., Walker, S., Rigatto, C., Uhlig, K., Kent, D. M., & Levey, A. S. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-8-201304160-00004 · Cited by 226
“Incorporating a prediction model into clinical practice requires an evaluation of its internal and external validity and clinical usefulness (10, 11).”
MethodEvaluating the impact of prediction models: lessons learned, challenges, and recommendations
Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0033-6 · Cited by 215
“Currently, the number of published prediction models is overwhelming [3, 10, 55].”
BackgroundSelf-citationPrognostic models for breast cancer: a systematic review
Phung, M. T., Tin, S. T., & Elwood, M. (2019) · BMC Cancer · doi:10.1186/s12885-019-5442-6 · Cited by 214
A risk prediction score for acute kidney injury in the intensive care unit
Malhotra, R., KASHANI, K. B., Macedo, E. M., Kim, J., Bouchard, J., Wynn, S., Li, G., Lucila Ohno-machado, A., & Mehta, R. L. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfx026 · Cited by 202
“Using the development cohort at UCSD, the regression coefficient-based models were constructed using a 5-fold crossvalidation procedure [60, 61].”
MethodIndividual Participant Data Meta-Analysis for a Binary Outcome: One-Stage or Two-Stage?
Debray, T. P. A., Moons, K. G. M., Abo-Zaid, G., Koffijberg, H., & Riley, R. D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0060650 · Cited by 201
Self-citationKorolev, I. O., Symonds, L. L., Bozoki, A., & Initiative, A. D. N. (2016) · PLoS ONE · doi:10.1371/journal.pone.0138866 · Cited by 203
“As the next step, it will be important to externally validate our model on an independent dataset [18,59].”
BackgroundPrognostic models in obstetrics: available, but far from applicable
Kleinrouweler, C. E., Cheong-See, F., Collins, P. G. S., Kwee, A., Thangaratinam, S., Khan, K. S., Mol, B. W., Pajkrt, E., Moons, K. G. M., & Schuit, E. (2015) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2015.06.013 · Cited by 200
Self-citationWilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
Collins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6986 · Cited by 185
Self-citationA conceptual framework for prognostic research
Kent, P. M., Cancelliere, C., Boyle, E., Cassidy, J. D., & Kongsted, A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01050-7 · Cited by 204
Do prostate cancer risk models improve the predictive accuracy of PSA screening? A meta-analysis
Louie, K. S., Seigneurin, A., Cathcart, P. J., & Sasieni, P. D. (2014) · Annals of Oncology · doi:10.1093/annonc/mdu525 · Cited by 184
“[19] for methodological assessment of prediction research.”
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4295 · Cited by 184
“[16] Bouwmeester W, Zuithoff NP, Mallett S, et al.”
BackgroundPoor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 173
Self-citationManuel, D. G., Perez, R., Sanmartin, C., Taljaard, M., Hennessy, D. A., Wilson, K. R., Tanuseputro, P., Manson, H. E., Bennett, C. L., Tuna, M., et al. (2016) · PLoS Medicine · doi:10.1371/journal.pmed.1002082 · Cited by 167
“Continuous measures were assessed and preferentially included for two reasons: for improved predictive performance (see below) [13,18,19], and to allow counterfactual examination of small changes in population exposure [12,13].”
Backgroundvan Loon, F. H. J., van Hooff, L. W. E., de Boer, H. D., Koopman, S. S. H. A., Buise, M. P., Korsten, H. H. M., Dierick-van Daele, A. T. M., & Bouwman, A. R. A. (2019) · Journal of Clinical Medicine · doi:10.3390/jcm8020144 · Cited by 157
“05 from the univariate analysis for examining their independent association with the primary outcome [14,18,20].”
MethodInfluentialA tutorial on methodological studies: the what, when, how and why
Mbuagbaw, L. C. E., Lawson, D. O., Puljak, L., Allison, D. B., & Thabane, L. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01107-7 · Cited by 181
Peat, G., Riley, R. D., Croft, P. R., Morley, K. I., Kyzas, P., Moons, K. G. M., Perel, P., Steyerberg, E. W., Schroter, S., Altman, D. G., et al. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001671 · Cited by 152
“Most do not explicitly build on previous findings, are often too small to answer the research question, and typically do not provide sample size calculations [24,25].”
BackgroundSelf-citationHeus, P., Damen, J. A., Pajouheshnia, R., Scholten, R. J. P. M., Reitsma, J. B., Collins, P. G. S., Altman, D. G., Moons, K. G. M., & Hooft, L. (2018) · BMC Medicine · doi:10.1186/s12916-018-1099-2 · Cited by 147
Self-citationBattle, C. E., Hutchings, H. A., Lovett, S., Bouamra, O., Jones, S., Sen, A., Gagg, J., Robinson, D., Hartford-Beynon, J., Williams, J. R., et al. (2014) · Critical Care · doi:10.1186/cc13873 · Cited by 147
“Discrimination was assessed with the c-statistic (equivalent to the area under the receiver operator curve) [17].”
BackgroundMethodInfluentialAidi, H. E., Adams, A. L. L., Moons, K. G. M., den Ruijter, H. M., Mali, W. P. T. M., Doevendans, P. A., Nagel, E. C., Schalla, S. M., Bots, M. L., & Leiner, T. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.048 · Cited by 142
“5 None C B B B C B B 18 (7–23) d CM, MI, CHF: 16 (14)”
MethodSelf-citationvan de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Screening for Chronic Obstructive Pulmonary Disease
Guirguis‐Blake, J., Senger, C. A., Webber, E. M., Mularski, R. A., & Whitlock, E. P. (2016) · JAMA · doi:10.1001/jama.2016.2654 · Cited by 146
“Shortness of breath, presence of phlegm or mucus, functional limitations due to breathing problems, smoking history, age Tsukuya et al,28 2015 Japan Smokers and nonsmokers ≥4 67 (60-74)a 73 (71-75)a 15 (12-17)a 97 (96-98)a 0.”
BackgroundMahar, A. L., Compton, C. C., Halabi, S., Hess, K. R., Weiser, M. R., & Groome, P. A. (2017) · Journal of Surgical Oncology · doi:10.1002/jso.24774 · Cited by 139
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
“However, compared to the number of prediction models that are developed, very few are used in clinical practice and many models contribute to research waste [14–17].”
BackgroundCollins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13244 · Cited by 145
Self-citationInfluence of Household Rat Infestation on Leptospira Transmission in the Urban Slum Environment
Costa, F., Ribeiro, G. S., Felzemburgh, R. D. M., Santos, N. D. J., Reis, R. B., dos Santos, A. C., Fraga, D. B. M., de Araújo, W. N., Santana, C., Childs, J. E., et al. (2014) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0003338 · Cited by 136
“The use of mutually adjusted weights per predictor is the standard methodology to develop a prognostic risk score [30].”
MethodOnland, W., Debray, T. P. A., Laughon, M. M., Miedema, M., Cools, F., Askie, L. M., Asselin, J. M., Calvert, S., Courtney, S. E., Dani, C., et al. (2013) · BMC Pediatrics · doi:10.1186/1471-2431-13-207 · Cited by 135
“In contrast with the former mentioned quality assessment instruments, this instrument should address not only reporting issues like the STROBE, but assess all the different aspects of the development, validation and implementation of a prediction model, as described in a series of recently published articles [8-11,60].”
BackgroundMethodSelf-citationBlagus, R., & Lusa, L. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0784-9 · Cited by 138
“Prediction models represent a valuable tool in the decision making process of clinicians and health policy makers and are extensively used in medicine [1], however the majority of prediction studies in high impact journals do not follow the current methodological recommendations, limiting their reliability and applica…”
BackgroundHenderson, J. T., Thompson, J. H., Burda, B. U., & Cantor, A. G. (2017) · JAMA · doi:10.1001/jama.2016.18315 · Cited by 137
Navarro, C. L. A., Damen, J. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, P. G. S., Bajpai, R. C., Riley, R. D., Moons, K. G. M., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01469-6 · Cited by 142
“Previous systematic reviews have shown poor reporting of regression-based prediction model studies [7, 8, 10, 11].”
BackgroundSelf-citationMortality prediction models in the general trauma population: A systematic review
de Munter, L., Polinder, S., Lansink, K. W. W., Cnossen, M. C., Steyerberg, E. W., & de Jongh, M. A. C. (2016) · Injury · doi:10.1016/j.injury.2016.12.009 · Cited by 129
Self-citationAndaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121
Self-citationRahman, M. S., & Sultana, M. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0313-9 · Cited by 120
“Although the choice of this cut-off has some criticisms [6] for not being based on scientific reasoning except empirical evidence, it is found useful for quantifying the amount of information in the data relative to model complexity [7, 8].”
BackgroundSnell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125
“Calibration and clinical utility, in particular, are often underappreciated [24-26].”
BackgroundCnossen, M. C., Winkler, E. A., Yue, J. K., Okonkwo, D. O., Valadka, A. B., Steyerberg, E. W., Lingsma, H. F., Manley, G. T., & the TRACK-TBI Investigators (2017) · Journal of Neurotrauma · doi:10.1089/neu.2016.4819 · Cited by 120
Self-citationEstimation of Newborn Risk for Child or Adolescent Obesity: Lessons from Longitudinal Birth Cohorts
Morandi, A., Meyre, D., Lobbens, S., KLEINMAN, K. P., Kaakinen, M., Rifas‐Shiman, S. L., Vatin, V., Gaget, S., Pouta, A. M., Hartikainen, A., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049919 · Cited by 117
“The main strengths include: the novelty and the potential strong public health impact of multivariate obesity predicting tools valid for newborns; the optimization of results reliability and robustness by the adoption of several recommended methods shown recently to be lacking in several recent high impact prediction…”
Machine Learning for Predictive Modelling based on Small Data in Biomedical Engineering
Shaikhina, T., Lowe, D., Daga, S., Briggs, D., Higgins, R. M., & Khovanova, N. A. (2015) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2015.10.185 · Cited by 115
Serghiou, S., Kyriakopoulou, A., & Ioannidis, J. P. A. (2016) · Molecular Cancer · doi:10.1186/s12943-016-0535-1 · Cited by 112
“dealing with prognosis [30, 38] and recommendations regarding reporting of biomarker studies [39, 40].”
BackgroundPrediction versus aetiology: common pitfalls and how to avoid them
van Diepen, M., Ramspek, C. L., Jager, K. J., Zoccali, C., & Dekker, F. W. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfw459 · Cited by 116
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3018 · Cited by 116
“13-16 This has led to high profile controversies over apparent discrepancies in results and reduced confidence in evidence generated from healthcare databases.”
Dhiman, P., Ma, J., Navarro, C. L. A., Speich, B., Bullock, G. S., Damen, J. A., Kirtley, S., Hooft, L., Riley, R. D., CALSTER, B. V., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.024 · Cited by 113
“Results from several systematic reviews informally assessing reporting quality and TRIPOD adherence, in on-cology and otherwise, are comparable with our findings [ 6 , 8 , 22 , 24 , 25 , 36–38 ].”
BackgroundResultSelf-citationScientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPreprint
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2016) · doi:10.31219/osf.io/k4bgq · Cited by 114
Reuland, D. S., Brenner, A. T., Hoffman, R. M., McWilliams, A. D., Rhyne, R. L., Getrich, C. M., Tapp, H., Weaver, M. A., Callan, D., Cubillos, L., et al. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.1294 · Cited by 110
Developing and validating risk prediction models in an individual participant data meta-analysis
Ahmed, I., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-3 · Cited by 106
“In particular, our work allows us to provide some key recommendations for improving the design, conduct and reporting of future research this field (Table 3), which echo those for IPD meta-analysis of prognostic factor [37] and modelling [16] studies, and concur with previous [38,39] and ongoing [40] work toward impro…”
MethodSelf-citationGlucose homeostasis, obesity and diabetes
Bano, G. (2013) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2013.02.007 · Cited by 110
Perel, P., Prieto‐Merino, D., Shakur‐Still, H., Clayton, T. C., Lecky, F. E., Bouamra, O., Russell, R. J., Faulkner, M., Steyerberg, E. W., & Roberts, I. (2012) · BMJ · doi:10.1136/bmj.e5166 · Cited by 103
Self-citationHodgson, L. E., Sarnowski, A., Roderick, P. J., Dimitrov, B. D., Venn, R. M., & Forni, L. G. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016591 · Cited by 101
Graafsma, J., Murphy, R. M., van de Garde, E. M. W., Karapinar‐Çarkit, F., Derijks, H. J., Hoge, R. H. L., Klopotowska, J. E., & van den Bemt, P. M. L. A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae076 · Cited by 115
“53 Once the models are developed and validated, adequate attention must be given to their implementation into hospital practice.”
BackgroundFahey, M., Crayton, E., Wolfe, C. D. A., & Douiri, A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0185402 · Cited by 102
Ma, G. K., & Ladabaum, U. (2014) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2014.01.042 · Cited by 97
“Although universally accepted criteria for study quality in the development of risk prediction models are not available, the investigators independently applied predefined criteria to assess themodel quality as good, fair, or poor on the basis of previously proposed principles.(36) Quality in studies reporting risk sc…”
MethodFredriksson, S. V., Alley, S., Rebar, A. L., Hayman, M., Vandelanotte, C., & Schöeppe, S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207003 · Cited by 97
Metadata
- Authors
- Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M.
- Affiliations
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Erasmus MC (NL)
- Corresponding author
- Moons, Karel G. M.
- ORCID
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2012-05-22
- Volume
- 9
- Issue
- 5
- Pages
- e1001221
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC3358324
- OpenAlex
- W1998392635
- Semantic Scholar
- 2d86e73e56aa4394fe4ac9a6ed742d850abc73ce
- Cited by
- 600
- References
- 88
- Citations by year
- 2026: 24 · 2025: 39 · 2024: 36 · 2023: 47 · 2022: 65 · 2021: 50 · 2020: 65 · 2019: 36 · 2018: 40 · 2017: 43 · 2016: 40 · 2015: 46 · 2014: 38 · 2013: 24 · 2012: 7
- FWCI
- 5.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Research and Theory › Methodologies in Health Research and Practice
- Keywords
- prediction model development, reporting quality, methodological quality, predictor selection, calibration, discrimination (statistics), external validation, prospective cohort, general medical journals
- MeSH terms
- Forecasting, Humans, Periodicals as Topic, Prospective Studies, Research Design, Biomedical Research, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
