The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Use of reclassification for assessment of improved prediction: an empirical evaluation
Abstract
BACKGROUND: An increasing number of studies evaluate the ability of predictors to change risk stratification and alter medical decisions, i.e. reclassification performance. We examined the reported design and analysis of recent studies of reclassification and the robustness of their claims for improved reclassification. METHODS: Two independent investigators searched PubMed and citations to the article that introduced the currently most popular reclassification metric (net reclassification index, NRI) to identify studies performing reclassification analysis (January 2006-January 2010). We focused on articles that included any analyses comparing the performance of a baseline predictive model vs the baseline model plus some additional predictor for a prospectively assessed outcome. We recorded information on the baseline model used, outcomes assessed, choice of risk thresholds and features of reclassification analyses. RESULTS: Of 58 baseline models used in 51 eligible papers, only 14 (24%) were previously described, used as described and had same outcomes as originally intended. Calibration was examined in 53% of the studies. Sixteen studies (31%) provided a reference for the choice of risk thresholds and only six used the previously proposed categories or justified the use of alternative thresholds. Only 14 studies (27%) stated that the chosen risk thresholds had different therapeutic intervention implications. NRI was calculated in 38 studies and was smaller in studies with adequately referenced or justified risk thresholds vs others (P < 0.0001). CONCLUSIONS: Reclassification studies would benefit from more rigorous methodological standards; otherwise claims for improved reclassification may remain spurious.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While making revisions, conscious efforts were made to harmonize our recommendations with other reporting guidelines, and where possible we chose the same or similar wording for items (37,39,41,45,46).”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [41]
In the citing paper
“the focus of the REMARK checklist is primarily on prognostic factors and not prediction models, whereas the GRIPS statement is aimed at risk prediction using genetic risk factors and the specific methodological issues around handling large numbers of genetic variants.”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [41]
In the citing paper
“The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42], also fits entirely within the remit of TRIPOD (see the accompanying explanation and elaboration document [43]).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [42]
In the citing paper
“An ever-increasing number of studies are evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42].”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [42]
Citations
Cites
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
Chobanian, A. V. (2003) · JAMA · doi:10.1001/jama.289.19.2560 · Cited by 25,386
Detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III)
Grundy, S. M., Becker, D. M., Clark, L. T., Cooper, R., Denke, M. A., Howard, J. L., Hunninghake, D. B., Illingworth, D. R., Luepker, R. V., McBride, P. E., et al. (2002) · Civil War Book Review · Cited by 12,198
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project
Conroy, R. (2003) · European Heart Journal · doi:10.1016/s0195-668x(03)00114-3 · Cited by 5,290
A Model to Predict Survival in Patients With End–Stage Liver Disease
Kamath, P. S., Wiesner, R. H., Malinchoc, M., Kremers, W., Therneau, T. M., Kosberg, C. L., D'Amico, G., Dickson, R. E., & Kim, R. W. (2001) · Hepatology · doi:10.1053/jhep.2001.22172 · Cited by 4,975
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., Bastos, P. G., Sirio, C. A., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · Chest · doi:10.1378/chest.100.6.1619 · Cited by 3,439
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
“The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models (18, 42), also fits entirely within the remit of TRIPOD (see the accompanying explanation and elaboration document [43], available at www .”
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
“An ever-increasing number of studies are evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models (Tzoulaki et al, 2011; Steyerberg et al, 2012).”
InfluentialCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
“The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42]; also fits entirely within the remit of TRIPOD (see the accompanying explanation and elaboration document [43]; available at www.”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
“The reporting of studies evaluating the incremental value of specific predictors, beyond established predictors or even beyond existing prediction models [18,42], also fits entirely within the remit of TRIPOD (see the accompanying”
Metadata
- Authors
- Tzoulaki, I., Liberopoulos, G., & Ioannidis, J. P. A.
- Affiliations
- Imperial College London (GB)
- Tufts University (US)
- Corresponding author
- Ioannidis, John P A
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-02-16
- Volume
- 40
- Issue
- 4
- Pages
- 1094-1105
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2106597074
- Cited by
- 70
- References
- 93
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2023: 3 · 2022: 7 · 2020: 1 · 2018: 4 · 2017: 2 · 2016: 2 · 2015: 16 · 2014: 5 · 2013: 9 · 2012: 11
- FWCI
- 7.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- reclassification, net reclassification improvement, risk stratification, risk thresholds, calibration, risk prediction model, methodological quality, prediction model evaluation
- MeSH terms
- Chronic Disease, Humans, Risk Factors, Models, Statistical, Treatment Outcome, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Use of reclassification for assessment of improved prediction: an empirical evaluation. https://app.cochat.ai/papers/10.1093/ije/dyr013
- The paper
- Tzoulaki, I., Liberopoulos, G., & Ioannidis, J. P. A. (2011). Use of reclassification for assessment of improved prediction: an empirical evaluation. International Journal of Epidemiology. https://doi.org/10.1093/ije/dyr013
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.