Performance of reclassification statistics in comparing risk prediction models
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Concerns have been raised about the use of traditional measures of model fit in evaluating risk prediction models for clinical use, and reclassification tables have been suggested as an alternative means of assessing the clinical utility of a model. Several measures based on the table have been proposed, including the reclassification calibration (RC) statistic, the net reclassification improvement (NRI), and the integrated discrimination improvement (IDI), but the performance of these in practical settings has not been fully examined. We used simulations to estimate the type I error and power for these statistics in a number of scenarios, as well as the impact of the number and type of categories, when adding a new marker to an established or reference model. The type I error was found to be reasonable in most settings, and power was highest for the IDI, which was similar to the test of association. The relative power of the RC statistic, a test of calibration, and the NRI, a test of discrimination, varied depending on the model assumptions. These tools provide unique but complementary information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 4×
“…with both types of size restrictions, was examined in simulations. r 2011 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim www.biometrical-journal.com 2.4 Discrimination improvement Pencina et al. (2008) suggested an alternative approach that divides the reclassification table into cases and controls.”
BackgroundInfluentialDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 2×
“Since costs may change over time or by personal preference, flexible decision rules, such as net benefit (Vickers and Elkin, 2006) or relative utility curves (Baker et al.”
BackgroundAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Self-citationA Randomized Trial of Low-Dose Aspirin in the Primary Prevention of Cardiovascular Disease in Women
Ridker, P. M., Cook, N. R., Lee, I.-M., Gordon, D., Gaziano, J. M., Manson, J. E., Hennekens, C. H., & Buring, J. E. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa050613 · Cited by 1,945
Self-citationPepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276
The effect of including C-reactive protein in cardiovascular risk prediction models for women
Cook, N. R., Buring, J. E., & Ridker, P. M. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-1-200607040-00128 · Cited by 504 · cited 3×
“These data have been described in more detail elsewhere (Cook et al., 2006; Cook and Ridker, 2009).”
BackgroundMethodGoodness-of-fit processes for logistic regression: simulation results
Hosmer, D. W., & Hjort, N. L. (2002) · Statistics in Medicine · doi:10.1002/sim.1200 · Cited by 164
Using Relative Utility Curves to Evaluate Risk Prediction
Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00592.x · Cited by 142
Self-citation
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Den Ruijter, H. M., Peters, S. A. E., Anderson, T. J., Britton, A. R., Dekker, J. M., Eijkemans, M. J., Engström, G., Evans, G. W., de Graaf, J., Grobbee, D. E., et al. (2012) · JAMA · doi:10.1001/jama.2012.9630 · Cited by 760
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594
FRAX underestimates fracture risk in patients with diabetes
Giangregorio, L. M., Leslie, W. D., Lix, L. M., Johansson, H., Oden, A., McCloskey, E., & Kanis, J. A. (2012) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.556 · Cited by 387
Tests of calibration and goodness‐of‐fit in the survival setting
Demler, O. V., Paynter, N. P., & Cook, N. R. (2015) · Statistics in Medicine · doi:10.1002/sim.6428 · Cited by 287
Self-citation
Metadata
- Authors
- Cook, N. R., & Paynter, N. P.
- Venue
- Biometrical Journal
- Publisher
- Wiley
- Published
- 2011-02-03
- Volume
- 53
- Issue
- 2
- Pages
- 237-258
- ISSN
- 0323-3847 (ISSN-L)
- 1521-4036
- OpenAlex
- W2084033659
- Cited by
- 122
- References
- 8
- FWCI
- 11.68
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of reclassification statistics in comparing risk prediction models. https://app.cochat.ai/papers/10.1002/bimj.201000078
- The paper
- Cook, N. R., & Paynter, N. P. (2011). Performance of reclassification statistics in comparing risk prediction models. Biometrical Journal. https://doi.org/10.1002/bimj.201000078
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.