The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets
Open accessAbstract
The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates. 48”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
In the citing paper
“Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
In the citing paper
“Dedicated measures, such as the widely used but improper net reclassification improvement, are available for evaluating competing models.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 2×
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · cited 3×
“’ See Gneiting and Raftery [3] for an in-depth discussion of proper scoring rules.”
InfluentialClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Thompson, I. M. (2005) · JAMA · doi:10.1001/jama.294.1.66 · Cited by 701 · cited 1×
“An example of a biomarker with this sort of predictive power is prostate specific antigen [19], one of the few cancer biomarkers currently used in cancer screening.”
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 595 · cited 1×
“Our results underscore the need to check if risk functions fit the test data set as a crucial part of the exercise of evaluating risk prediction models [18].”
Net Reclassification Indices for Evaluating Risk Prediction Instruments
Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & PEPE, M. S. (2013) · Epidemiology · doi:10.1097/ede.0000000000000018 · Cited by 454 · cited 1×
“have recently provided a critical review of the NRI statistic that focuses on conceptual aspects [8].”
Pencina, M. J., D’Agostino, R. B., & Demler, O. (2011) · Statistics in Medicine · doi:10.1002/sim.4348 · Cited by 348 · cited 1×
“The continuous NRI statistic [12,13] is calculated as NRI=2{P[ ˆ risk(X,Y )> ˆ risk(X)|D = 1]−P[ ˆ risk(X,Y )> ˆ risk(X)|D = 0]} (1) the proportion of cases in the test dataset for whom ˆ risk(X,Y ) > ˆ risk(X) minus the corresponding proportion of controls, multiplied by 2.”
Assessment of claims of improved prediction beyond the Framingham risk score.
Tzoulaki, I., Liberopoulos, G., & Ioannidis, J. (2009) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2009.1757 · Cited by 281 · cited 1×
“75 [20] which is in the middle of this range.”
Combining Several Screening Tests: Optimality of the Risk Score
McIntosh, M. W., & PEPE, M. S. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00657.x · Cited by 254 · cited 1×
“In addition, the AUC and ROC( f ) measures are proper since the ROC curve for (X,Y ) is maximized at all points by the risk function [10].”
Testing for improvement in prediction model performance
PEPE, M. S., Kerr, K. F., Longton, G., & Wang, Z. (2013) · Statistics in Medicine · doi:10.1002/sim.5727 · Cited by 239
Evaluating the Incremental Value of New Biomarkers With Integrated Discrimination Improvement
Kerr, K. F., McClelland, R. L., Brown, E. R., & Lumley, T. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr086 · Cited by 210
Hilden, J., & Gerds, T. A. (2013) · Statistics in Medicine · doi:10.1002/sim.5804 · Cited by 200
One statistical test is sufficient for assessing new predictive markers
Vickers, A. J., Cronin, A. M., & Begg, C. B. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-13 · Cited by 198 · cited 1×
“We and others have argued previously that testing the null hypothesis about the regression coefficient associated with Y in the risk model risk(X,Y ) is equivalent and more powerful to tests based on estimates of prediction improvement performance measures [15,24].”
Using Relative Utility Curves to Evaluate Risk Prediction
Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00592.x · Cited by 142 · cited 2×
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Vickers, A. J., & Cronin, A. M. (2010) · Seminars in Oncology · doi:10.1053/j.seminoncol.2009.12.004 · Cited by 108 · cited 1×
“A variety of statistics have been proposed and there has been some debate in the literature about which measures are most appropriate [14,21].”
Does the Net Reclassification Improvement Help Us Evaluate Models and Markers?
Vickers, A. J., & PEPE, M. S. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2841 · Cited by 47
Multicategory reclassification statistics for assessing improvements in diagnostic accuracy
Li, J., Jiang, B., & Fine, J. P. (2012) · Biostatistics · doi:10.1093/biostatistics/kxs047 · Cited by 47
Two Criteria for Evaluating Risk Prediction Models
Pfeiffer, R. M., & Gail, M. H. (2010) · Biometrics · doi:10.1111/j.1541-0420.2010.01523.x · Cited by 46
Methods for Evaluating Prediction Performance of Biomarkers and Tests
PEPE, M. S., & Janes, H. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8_7 · Cited by 44
Risk Assessment and Evaluation of Predictions
Lee, M. T., Gail, M. H., Pfeiffer, R. M., Satten, G. A., Cai, T., & Gandy, A. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8 · Cited by 40
Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update
Baker, S. G., Calster, B. V., & Steyerberg, E. W. (2012) · The International Journal of Biostatistics · doi:10.1515/1557-4679.1395 · Cited by 36
Hilden, J. (2014) · Epidemiology · doi:10.1097/ede.0000000000000063 · Cited by 33 · cited 9×
“M. S. Pepe (B) · J. Fan Biostatistics and Biomathematics Program, Fred Hutchinson Cancer Research Center, 1100 Fairview Ave N, M2B500, Seattle, WA 98109, USA e-mail: mspepe@u.washington.edu Z. Feng The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Houston, TX 77030, USA T. Gerds · J. Hilden Depar…”
InfluentialNet Reclassification Index: a Misleading Measure of Prediction Improvement
PEPE, M. S., Janes, H., Kerr, K. F., & Psaty, B. M. (2013) · Collection of Biostatistics Research Archive · Cited by 2
Cited by
Sorted by citations
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697
Developing prediction models for clinical use using logistic regression: an overview
Shipe, M. E., Deppen, S. A., Farjah, F., & Grogan, E. L. (2019) · Journal of Thoracic Disease · doi:10.21037/jtd.2019.01.25 · Cited by 515
“However, this can be insensitive to detecting small changes in model performance, especially if the AUC of the model is already large, and the information may not be clinically relevant (24,25).”
BackgroundYadlowsky, S., Hayward, R. A., Sussman, J. B., McClelland, R. L., Min, Y., & Basu, S. (2018) · Annals of Internal Medicine · doi:10.7326/m17-3011 · Cited by 271
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 271
Trimethylamine N-oxide and prognosis in acute heart failure
Suzuki, T., Heaney, L. M., Bhandari, S. S., Jones, D. J. L., & Ng, L. L. (2016) · Heart · doi:10.1136/heartjnl-2015-308826 · Cited by 253
Metadata
- Authors
- Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J.
- Affiliations
- Fred Hutch Cancer Center (US)
- The University of Texas MD Anderson Cancer Center (US)
- University of Copenhagen (DK)
- Corresponding author
- Pepe, Margaret S.
- Venue
- Statistics in Biosciences
- Published
- 2014-08-23
- Volume
- 7
- Issue
- 2
- Pages
- 282-295
- ISSN
- 1867-1764 (ISSN-L)
- 1867-1772
- PMC
- PMC4615606
- OpenAlex
- W2014187955
- Cited by
- 191
- References
- 27
- Citations by year
- 2026: 14 · 2025: 18 · 2024: 18 · 2023: 21 · 2022: 20 · 2021: 22 · 2020: 18 · 2019: 15 · 2018: 16 · 2017: 12 · 2016: 12 · 2015: 3 · 2014: 2
- FWCI
- 10.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Statistics in Biosciences (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Funders
- National Institutes of Health
- National Cancer Institute (R01CA152089, P30CA015704, U24CA086368)
- National Institute of General Medical Sciences (R01GM054438)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- net reclassification improvement, risk prediction model, statistical properties, overfitting, risk function, ROC curve, net benefit, Brier score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
