The Net Reclassification Index (NRI): a misleading measure of prediction improvement even with independent test data sets
Open accessAbstract
The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates.
“AUROC, AUPRC, and pAUROC are semi-proper, because these rank-based measures are invariant to monotonic transformations of probability estimates. 48”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.
“Net benefit and standardised net benefit are semi-proper; the probability estimates below the threshold can be anything as long as they are below the threshold, and the same goes for probability estimates above the threshold.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
The net reclassification improvement is improper.
“Dedicated measures, such as the widely used but improper net reclassification improvement, are available for evaluating competing models.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 3× · see citing works
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 2× · see citing works
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · cited 3× · see citing works
“’ See Gneiting and Raftery [3] for an in-depth discussion of proper scoring rules.”
InfluentialClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,478 · cited 3× · see citing works
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 1,526 · cited 1× · see citing works
“Our results underscore the need to check if risk functions fit the test data set as a crucial part of the exercise of evaluating risk prediction models [18].”
Thompson, I. M. (2005) · JAMA · doi:10.1001/jama.294.1.66 · Cited by 701 · cited 1× · see citing works
“An example of a biomarker with this sort of predictive power is prostate specific antigen [19], one of the few cancer biomarkers currently used in cancer screening.”
Net Reclassification Indices for Evaluating Risk Prediction Instruments
Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & PEPE, M. S. (2013) · Epidemiology · doi:10.1097/ede.0000000000000018 · Cited by 454 · cited 1× · see citing works
“have recently provided a critical review of the NRI statistic that focuses on conceptual aspects [8].”
Pencina, M. J., D’Agostino, R. B., & Demler, O. (2011) · Statistics in Medicine · doi:10.1002/sim.4348 · Cited by 348 · cited 1× · see citing works
“The continuous NRI statistic [12,13] is calculated as NRI=2{P[ ˆ risk(X,Y )> ˆ risk(X)|D = 1]−P[ ˆ risk(X,Y )> ˆ risk(X)|D = 0]} (1) the proportion of cases in the test dataset for whom ˆ risk(X,Y ) > ˆ risk(X) minus the corresponding proportion of controls, multiplied by 2.”
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · cited 1× · see citing works
“75 [20] which is in the middle of this range.”
Combining Several Screening Tests: Optimality of the Risk Score
McIntosh, M. W., & PEPE, M. S. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00657.x · Cited by 254 · cited 1× · see citing works
“In addition, the AUC and ROC( f ) measures are proper since the ROC curve for (X,Y ) is maximized at all points by the risk function [10].”
Testing for improvement in prediction model performance
PEPE, M. S., Kerr, K. F., Longton, G., & Wang, Z. (2013) · Statistics in Medicine · doi:10.1002/sim.5727 · Cited by 239 · see citing works
Evaluating the Incremental Value of New Biomarkers With Integrated Discrimination Improvement
Kerr, K. F., McClelland, R. L., Brown, E. R., & Lumley, T. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr086 · Cited by 210 · see citing works
Hilden, J., & Gerds, T. A. (2013) · Statistics in Medicine · doi:10.1002/sim.5804 · Cited by 200 · see citing works
One statistical test is sufficient for assessing new predictive markers
Vickers, A. J., Cronin, A. M., & Begg, C. B. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-13 · Cited by 198 · cited 1× · see citing works
“We and others have argued previously that testing the null hypothesis about the regression coefficient associated with Y in the risk model risk(X,Y ) is equivalent and more powerful to tests based on estimates of prediction improvement performance measures [15,24].”
Using Relative Utility Curves to Evaluate Risk Prediction
Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00592.x · Cited by 142 · cited 2× · see citing works
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Vickers, A. J., & Cronin, A. M. (2010) · Seminars in Oncology · doi:10.1053/j.seminoncol.2009.12.004 · Cited by 108 · cited 1× · see citing works
“A variety of statistics have been proposed and there has been some debate in the literature about which measures are most appropriate [14,21].”
Does the Net Reclassification Improvement Help Us Evaluate Models and Markers?
Vickers, A. J., & PEPE, M. S. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2841 · Cited by 47 · see citing works
Multicategory reclassification statistics for assessing improvements in diagnostic accuracy
Li, J., Jiang, B., & Fine, J. P. (2012) · Biostatistics · doi:10.1093/biostatistics/kxs047 · Cited by 47 · see citing works
Two Criteria for Evaluating Risk Prediction Models
Pfeiffer, R. M., & Gail, M. H. (2010) · Biometrics · doi:10.1111/j.1541-0420.2010.01523.x · Cited by 46 · see citing works
Methods for Evaluating Prediction Performance of Biomarkers and Tests
PEPE, M. S., & Janes, H. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8_7 · Cited by 44 · see citing works
Risk Assessment and Evaluation of Predictions
Lee, M. T., Gail, M. H., Pfeiffer, R. M., Satten, G. A., Cai, T., & Gandy, A. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8 · Cited by 40 · see citing works
Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update
Baker, S. G., Calster, B. V., & Steyerberg, E. W. (2012) · The International Journal of Biostatistics · doi:10.1515/1557-4679.1395 · Cited by 36 · see citing works
Hilden, J. (2014) · Epidemiology · doi:10.1097/ede.0000000000000063 · Cited by 33 · cited 9× · see citing works
“M. S. Pepe (B) · J. Fan Biostatistics and Biomathematics Program, Fred Hutchinson Cancer Research Center, 1100 Fairview Ave N, M2B500, Seattle, WA 98109, USA e-mail: mspepe@u.washington.edu Z. Feng The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Houston, TX 77030, USA T. Gerds · J. Hilden Depar…”
InfluentialNet Reclassification Index: a Misleading Measure of Prediction Improvement
PEPE, M. S., Janes, H., Kerr, K. F., & Psaty, B. M. (2013) · Collection of Biostatistics Research Archive · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697 · see citing works
Developing prediction models for clinical use using logistic regression: an overview
Shipe, M. E., Deppen, S. A., Farjah, F., & Grogan, E. L. (2019) · Journal of Thoracic Disease · doi:10.21037/jtd.2019.01.25 · Cited by 515 · see citing works
“However, this can be insensitive to detecting small changes in model performance, especially if the AUC of the model is already large, and the information may not be clinically relevant (24,25).”
BackgroundYadlowsky, S., Hayward, R. A., Sussman, J. B., McClelland, R. L., Min, Y., & Basu, S. (2018) · Annals of Internal Medicine · doi:10.7326/m17-3011 · Cited by 271 · see citing works
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 265 · see citing works
Trimethylamine <i>N</i>-oxide and prognosis in acute heart failure
Suzuki, T., Heaney, L. M., Bhandari, S. S., Jones, D. J. L., & Ng, L. L. (2016) · Heart · doi:10.1136/heartjnl-2015-308826 · Cited by 253 · see citing works
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Customized growth charts: rationale, validation and clinical benefits
Gardosi, J. O., Francis, A. I., Turner, S., & Williams, M. (2018) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2017.12.011 · Cited by 238 · see citing works
Merlo, J., Wagner, P., Ghith, N. M. M., & Leckie, G. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153778 · Cited by 230 · see citing works
Three myths about risk thresholds for prediction models
on behalf of the Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Wynants, L., van Smeden, M., McLernon, D. J., Timmerman, D., Steyerberg, E. W., & Van Calster, B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1425-3 · Cited by 213 · see citing works
Cardiovascular disease: The rise of the genetic risk score
Knowles, J. W., & Ashley, E. A. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002546 · Cited by 182 · see citing works
The c-index is not proper for the evaluation of $t$-year predicted risks
Blanche, P., Kattan, M. W., & Gerds, T. A. (2018) · Biostatistics · doi:10.1093/biostatistics/kxy006 · Cited by 158 · see citing works
Polygenic Scores to Assess Atherosclerotic Cardiovascular Disease Risk
Aragam, K. G., & Natarajan, P. (2020) · Circulation Research · doi:10.1161/circresaha.120.315928 · Cited by 146 · see citing works
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models
Assel, M., Sjoberg, D., & Vickers, A. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0020-3 · Cited by 133 · see citing works
Johnson, A. E., Zhu, J., Garrard, W., Thoma, F. W., Mulukutla, S. R., Kershaw, K. N., & Magnani, J. W. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.020466 · Cited by 124 · see citing works
Clinical lipidomics: realizing the potential of lipid profiling
Meikle, T. G., Huynh, K., Giles, C., & Meikle, P. J. (2021) · Journal of Lipid Research · doi:10.1016/j.jlr.2021.100127 · Cited by 121 · see citing works
“While debate continues over the best approach (28), it will be important for any new test to provide some comparison to existing measures and clearly demonstrate an improvement in the clinical utility of lipidomic tests over existing measures.”
BackgroundWahl, S., Boulesteix, A., Zierer, A., Thorand, B., & van de Wiel, M. A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0239-7 · Cited by 90 · see citing works
“The optimistic bias of NRI has lead to critical discussion [47].”
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 95 · see citing works
Breast Arterial Calcification: a Novel Cardiovascular Risk Enhancer Among Postmenopausal Women
Iribarren, C., Chandra, M., Lee, C., Sanchez, G., Sam, D. L., Azamian, F. F., Cho, H.-M., Ding, H., Wong, N. D., & Molloi, S. Y. (2022) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.121.013526 · Cited by 86 · see citing works
Application of the International IgA Nephropathy Prediction Tool one or two years post-biopsy
Barbour, S. J., Coppo, R., Zhang, H., Liu, Z., Suzuki, Y., Matsuzaki, K., Er, L., Reich, H. N., Barratt, J., Cattran, D., et al. (2022) · Kidney International · doi:10.1016/j.kint.2022.02.042 · Cited by 86 · see citing works
Huang, C.-X., Li, S.-X., Caraballo, C., Masoudi, F., Rumsfeld, J., Spertus, J., Normand, S.-L. T., Mortazavi, B., & Krumholz, H. (2021) · Circulation. Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007526 · Cited by 73 · see citing works
“11,48 Moreover, these summary statistics cannot assess the difference of the models in predicting risks for an individual patient.”
BackgroundThe emerging field of polygenic risk scores and perspective for use in clinical care
Yanes, T., McInerney-Leo, A. M., Law, M. H., & Cummings, S. A. (2020) · Human Molecular Genetics · doi:10.1093/hmg/ddaa136 · Cited by 75 · see citing works
Więckowska, B., Kubiak, K. B., Jóźwiak, P., Moryson, W., & Stawińska-Witoszyńska, B. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191610213 · Cited by 82 · see citing works
“The reason for this is evident in the doubts raised regarding the correctness of some calculations and interpretations that have appeared in statistics journals [26] and in reputable medical journals [24,27–32].”
ResultLi, A., La, J., May, S., Guffey, D., da Costa, W. L., Amos, C. I., Bandyo, R., Milner, E. M., Kurian, K. M., Chen, D. C. R., et al. (2023) · Journal of Clinical Oncology · doi:10.1200/jco.22.01542 · Cited by 76 · see citing works
Polygenic risk scores for the prediction of cardiometabolic disease
O’Sullivan, J. W., Ashley, E. A., & Elliot, P. M. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac648 · Cited by 73 · see citing works
Refining the CHA2DS2VASc risk stratification scheme: shall we drop the sex category criterion?
Yoshimura, H., Providencia, R. A., Finan, C., Schmidt, A. F., & Lip, G. Y. H. (2024) · EP Europace · doi:10.1093/europace/euae280 · Cited by 54 · see citing works
“49,50 In this analysis, repeated assessment of risk prediction schemes over time showed the same trends and results, thereby increasing the robustness of our findings.”
Lipidomics and Biomarker Discovery in Kidney Disease
Afshinnia, F., Rajendiran, T. M., Wernisch, S., Soni, T., Jadoon, A., Karnovsky, A., Michailidis, G. C., & Pennathur, S. (2018) · Seminars in Nephrology · doi:10.1016/j.semnephrol.2018.01.004 · Cited by 52 · see citing works
Parodi, S., Verda, D., Bagnasco, F., & Marco Muselli, B. (2022) · Epidemiology and Health · doi:10.4178/epih.e2022088 · Cited by 47 · see citing works
Solares, J. R. A., Canoy, D., Raimondi, F. E. D., Zhu, Y., Hassaïne, A., Salimi‐Khorshidi, G., Tran, J., Copland, E., Zottoli, M., Pinho‐Gomes, A., et al. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.012129 · Cited by 45 · see citing works
Omics and Cardiometabolic Disease Risk Prediction
Tahir, U. A., & Gerszten, R. E. (2020) · Annual Review of Medicine · doi:10.1146/annurev-med-042418-010924 · Cited by 45 · see citing works
Net Reclassification Improvement
Jewell, E. S., Maile, M. D., Engoren, M., & Elliott, M. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001141 · Cited by 40 · see citing works
Which N Descriptor Is More Predictive of Prognosis in Resected Non-small Cell Lung Cancer
Xu, L., Su, H., She, Y., Dai, C., Zhao, M., Gao, J., Xie, H., Ren, Y., Xie, D., & Chen, C. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.12.012 · Cited by 40 · see citing works
Elhaï, M., Sritharan, N., Boubaya, M., Balbir‐Gurman, A., Siegert, E., HACHULLA, E., de Vries‐Bouwstra, J., Riemekasten, G., Distler, J. H. W., Rosato, E., et al. (2022) · The Lancet Rheumatology · doi:10.1016/s2665-9913(22)00217-x · Cited by 39 · see citing works
Pamoukdjian, F., Aparicio, T., Zebachi, S., Zelek, L. H., Paillaud, É., & Canouï‐Poitrine, F. (2019) · The Journals of Gerontology Series A · doi:10.1093/gerona/glz024 · Cited by 37 · see citing works
Statistical Methods for Cohort Studies of CKD: Prediction Modeling
Roy, J. A., Shou, H., Xie, D., Hsu, J. Y., Yang, W. P., Anderson, A. H., Landis, J. R., Jepson, C., He, J., Liu, K. D., et al. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.06210616 · Cited by 36 · see citing works
Predictive accuracy of combined genetic and environmental risk scores
Dudbridge, F., Pashayan, N., & Yang, J. (2017) · Genetic Epidemiology · doi:10.1002/gepi.22092 · Cited by 36 · see citing works
“The NRI has been criticized on theoretical and practical grounds (Kerr et al., 2014; Pepe et al., 2015), yet it has an intuitive appeal that should not be overlooked.”
Yamamoto, T., Kimura, H., Hayashi, K., Imamura, Y., & Mori, M. (2018) · Neuroradiology · doi:10.1007/s00234-018-2046-9 · Cited by 36 · see citing works
Coronary Microcirculatory Dysfunction and Acute Cellular Rejection After Heart Transplantation
Lee, J. M., Choi, K. H., Choi, J.-O., Shin, D., Park, Y., Kim, J., Lee, S. H., Kim, D., Yang, J. H., Cho, Y. H., et al. (2021) · Circulation · doi:10.1161/circulationaha.121.056158 · Cited by 36 · see citing works
Prognosis and Reclassification by YKL‐40 in Stable Coronary Artery Disease
Schrøder, J., Jakobsen, J. C., Winkel, P., Hilden, J., Jensen, G. B., Sajadieh, A., Larsson, A. O., Ärnlöv, J., Harutyunyan, M., Johansen, J. S., et al. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.014634 · Cited by 34 · see citing works
“NRI may however be more unreliable and misleading than initially thought with the risk of yielding false-positive NRI statistics.(49,50) The majority of candidate CVD biomarkers lack sufficient reclassification assessment, and even some of the most thoroughly investigated candidate biomarkers have failed to exhibit cl…”
Helmersson‐Karlqvist, J., Lipcsey, M., Ärnlöv, J., Bell, M., Ravn, B., Dardashti, A., & Larsson, A. O. (2021) · Scientific Reports · doi:10.1038/s41598-021-85370-8 · Cited by 36 · see citing works
Liu, H., Zhang, L., Xu, F., Li, S., Wang, Z., Han, D., Zhang, F., Lyu, J., & Yin, H. (2021) · Journal of International Medical Research · doi:10.1177/03000605211044892 · Cited by 35 · see citing works
Kidney Disease Biomarkers Improve Heart Failure Risk Prediction in the General Population
Nowak, C., & Ärnlöv, J. (2020) · Circulation Heart Failure · doi:10.1161/circheartfailure.120.006904 · Cited by 34 · see citing works
Polygenic risk scores in coronary artery disease
Rao, A. S., & Knowles, J. W. (2019) · Current Opinion in Cardiology · doi:10.1097/hco.0000000000000629 · Cited by 33 · see citing works
“The widely used NRI statistic, which evaluates the incremental predictive ability of a new marker added to a set of baseline predictors, has been shown to provide inflated estimates of a predictor’s performance [19].”
BackgroundMethodTuan, T. A., Zeevi, T., Haider, S. P., Karam, G. A., Berson, E. R., Tharmaseelan, H., Qureshi, A. I., Sanelli, P. C., Werring, D. J., Malhotra, A., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01007-w · Cited by 32 · see citing works
Reconsidering lactate as a sepsis risk biomarker
Moran, J. L., & Santamaria, J. D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185320 · Cited by 31 · see citing works
Gao, F., Shan, L., Wang, C., Meng, X., Chen, J., Han, L., Zhang, Y., & Li, Z. (2021) · International Journal of General Medicine · doi:10.2147/ijgm.s338819 · Cited by 31 · see citing works
“Consistency of the two evaluation systems in predicting in-hospital and medium-term mortality was tested by the net reclassification index (NRI).(20) Patients were divided into two groups by different standards.”
MethodInflammation biomarkers and mortality prediction in patients with type 2 diabetes (ZODIAC-27)
Landman, G. W. D., Kleefstra, N., Groenier, K. H., Bakker, S. J. L., Groeneveld, G. H., Bilo, H. J. G., & van Hateren, K. J. J. (2016) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2016.04.015 · Cited by 30 · see citing works
Calibration of medical diagnostic classifier scores to the probability of disease
Chen, W., Sahiner, B., Samuelson, F., Pezeshk, A., & Petrick, N. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280216661371 · Cited by 29 · see citing works
Han, Y., Yan, L., Li, P., Zhang, L., Ouyang, P.-H., & Hu, Z.-D. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027287 · Cited by 28 · see citing works
Mori, M., Durant, T. J. S., Huang, C., Mortazavi, B. J., Coppi, A. C., Jean, R. A., Geirsson, A., Schulz, W. L., & Krumholz, H. M. (2021) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007363 · Cited by 28 · see citing works
McKearnan, S. B., Wolfson, J. A., Vock, D. M., Vazquez‐Benitez, G., & O’Connor, P. J. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx374 · Cited by 26 · see citing works
Hayashi, K., & Eguchi, S. (2019) · Statistics in Medicine · doi:10.1002/sim.8135 · Cited by 27 · see citing works
Basili, S., Loffredo, L., Pastori, D., Proietti, M., Farcomeni, A., Vestri, A. R., Pignatelli, P., Davı̀, G., Hiatt, W. R., Lip, G. Y. H., et al. (2017) · International Journal of Cardiology · doi:10.1016/j.ijcard.2017.01.001 · Cited by 27 · see citing works
Otaki, Y., Watanabe, T., Konta, T., Watanabe, M., Asahi, K., Yamagata, K., Fujimoto, S., Tsuruya, K., Narita, I., Kasahara, M., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-71301-6 · Cited by 26 · see citing works
Twelve practical recommendations for developing and applying clinical predictive models
Feng, G., Xu, H., Wan, S., Wang, H., Chen, X., Magari, R. T., Han, Y. K., Wei, Y., & Gu, H. (2024) · The Innovation Medicine · doi:10.59717/j.xinn-med.2024.100105 · Cited by 27 · see citing works
Meo, M., Pambrun, T., Derval, N., Dumas-Pomier, C., Puyo, S., Duchâteau, J., Jaı̈s, P., Hocini, M., Haı̈ssaguerre, M., & Dubois, R. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00929 · Cited by 25 · see citing works
“Finally, the NRI metric itself may not be suitable for the comparison between two classification models, especially if they do not fit the training datasets accurately (Pepe et al., 2015).”
BackgroundYajima, T., Yajima, K., & Takahashi, H. (2020) · Nutrients · doi:10.3390/nu12113333 · Cited by 24 · see citing works
“However, recently, some statisticians raised a concern regarding the overestimation of the improvement of predictability among predicting models using the NRI [22,23].”
Bungo, B., Chaudhury, P., Arustamyan, M., Rikhi, R. R., Hussain, M., Collier, P. H., Kanj, M. H., Khorana, A. A., Mentias, A. G., & Moudgil, R. (2022) · IJC Heart & Vasculature · doi:10.1016/j.ijcha.2022.101072 · Cited by 24 · see citing works
Recalibration Methods for Improved Clinical Utility of Risk Scores
Mishra, A., McClelland, R. L., Inoue, L. Y. T., & Kerr, K. F. (2021) · Medical Decision Making · doi:10.1177/0272989x211044697 · Cited by 23 · see citing works
Prediction of Cardiovascular Events by Pulse Waveform Parameters: Analysis of CARTaGENE
Desbiens, L., Fortier, C., Nadeau‐Fredette, A., Madore, F., Hametner, B., Wassertheurer, S., Agharazii, M., & Goupil, R. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.122.026603 · Cited by 23 · see citing works
“49,52,53 Similar analyses were conducted for traditional clinical risk factors (BMI, eGFR) not included in the ASCVD score, for comparison purposes.”
Net Reclassification Index Statistics Do Not Help Assess New Risk Models
Kerr, K. F. (2022) · Radiology · doi:10.1148/radiol.222343 · Cited by 24 · see citing works
Xu, K.-Z., Xu, P., Li, J., Zuo, A.-F., Wang, S., & Han, F. (2023) · BMC Anesthesiology · doi:10.1186/s12871-023-02189-8 · Cited by 24 · see citing works
“And a nomogram for predicting 28-day mortality of SIMI was constructed in training cohort according to Occam′s Law, in which fewer variables should be included to achieve the aim [14] Concordance index (C-index) and the area under the receiver operating characteristic curve (AUC) were performed to assess the effective…”
MethodA Systematic Framework for Assessing the Clinical Impact of Polygenic Risk ScoresPreprint
Kulm, S., Marderstein, A. R., Mezey, J. G., & Elemento, O. (2020) · medRxiv · doi:10.1101/2020.04.06.20055574 · Cited by 24 · see citing works
Mark, D. G., Huang, J., Kene, M. V., Sax, D. R., Cotton, D. M., Lin, J. S., Bouvet, S. C., Chettipally, U. K., Anderson, M. L., McLachlan, I. D., et al. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.020082 · Cited by 22 · see citing works
Ding, T.-Y., Liu, C.-T., Huang, B.-L., Chu, L., Wei, L., Lin, Y., Luo, Y., Zhang, B., Hong, C., Xu, Y., et al. (2021) · Cancer Management and Research · doi:10.2147/cmar.s329687 · Cited by 22 · see citing works
Are polygenic risk scores ready for the cancer clinic?—a perspective
Klein, R. J., & Gumus, Z. H. (2022) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-698 · Cited by 21 · see citing works
“For example, when a popular measure, the Net Reclassification Index (NRI) (18) is calculated on a large test dataset, it is likely to be positive even when the addition of the PRS to an existing model has no predictive information.”
Hendriks, S. H., van Dijk, P. R., van Hateren, K. J. J., van Pelt, L. J., Groenier, K. H., Bilo, H. J. G., Bakker, S. J. L., Landman, G. W. D., & Kleefstra, N. (2015) · American Heart Journal · doi:10.1016/j.ahj.2015.12.015 · Cited by 22 · see citing works
Chapman, B. P., Lin, F. V., Roy, S., Benedict, R. H. B., & Lyness, J. M. (2019) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000300 · Cited by 22 · see citing works
McKeigue, P. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218776989 · Cited by 21 · see citing works
“However, these indices are not proper scoring rules(12); this means that adding a biomarker to the predictive model can apparently ‘improve’ such an index even when that biomarker contains no predictive information.(13,14) The authors of a widely quoted set of guidelines on reporting of multivariate models for diagnos…”
BackgroundComparison of Questionnaire-Based Breast Cancer Prediction Models in the Nurses' Health Study
Glynn, R. J., Colditz, G. A., Tamimi, R. M., Chen, W. Y., Hankinson, S. E., Willett, W. C., & ROSNER, B. A. (2019) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-18-1039 · Cited by 19 · see citing works
“Even when reclassification is separately considered among cases and noncases, interpretation of these indices is problematic whenmodels exhibit some level of miscalibration (37).”
BackgroundAimo, A., Fabiani, I., Vergaro, G., Arzilli, C., Chubuchny, V., Pasanisi, E., Petersen, C., Poggianti, E., Taddei, C., Pugliese, N. R., et al. (2021) · ESC Heart Failure · doi:10.1002/ehf2.13396 · Cited by 19 · see citing works
Hwang, A. B., Schuepfer, G., Pietrini, M., & Boes, S. (2021) · PLOS ONE · doi:10.1371/journal.pone.0258338 · Cited by 19 · see citing works
Statistical methods for validation of predictive models
Diniz, M. A. (2022) · Journal of Nuclear Cardiology · doi:10.1007/s12350-022-02994-7 · Cited by 21 · see citing works
Development and validation of novel multimorbidity indices for older adults
Steinman, M. A., Jing, B., Shah, S. J., Rizzo, A., Lee, S. J., Covinsky, K. E., Ritchie, C. S., & Boscardin, W. J. (2022) · Journal of the American Geriatrics Society · doi:10.1111/jgs.18052 · Cited by 18 · see citing works
Fujimoto, Y., Matsue, Y., Maeda, D., Kagiyama, N., Sunayama, T., Dotare, T., Jujo, K., Saito, K., Kamiya, K., Saito, H., et al. (2023) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2023.11.020 · Cited by 17 · see citing works
Mohebi, R., van Kimmenade, R. R. J., McCarthy, C. P., Gaggin, H. K., Mehran, R., Dangas, G., & Januzzi, J. L. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.122.025729 · Cited by 16 · see citing works
“28 Finally, while CASABLANCA was originally designed and powered to evaluate for risk of CA-AKI, we did not correct for multiple comparisons in our statistical methods.”
Sheng, W., Wang, X., Xu, W., Hao, Z., Ma, H., & Zhang, S. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1198526 · Cited by 18 · see citing works
First things first: risk model performance metrics should reflect the clinical application
Kerr, K. F., & Janes, H. (2017) · Statistics in Medicine · doi:10.1002/sim.7341 · Cited by 16 · see citing works
Hobabagabo, A. F., OSEI-TUTU, N. H., Hormenu, T., SHOUP, E. M., DuBose, C. W., Mabundo, L. S., Ha, J., Sherman, A., Chung, S. T., Sacks, D. B., et al. (2020) · Diabetes Care · doi:10.2337/dc20-1119 · Cited by 15 · see citing works
Renal Disease and Atrial Fibrillation
Stefil, M., Nabrdalik, K., & Lip, G. Y. H. (2021) · Cardiac Electrophysiology Clinics · doi:10.1016/j.ccep.2020.11.001 · Cited by 13 · see citing works
Jennings, G., Audehm, R. G., Bishop, W., Chow, C. K., Liaw, S., Liew, D., & Linton, S. M. (2021) · The Medical Journal of Australia · doi:10.5694/mja2.51039 · Cited by 13 · see citing works
Al‐Wahsh, H., Lam, N. N., Quinn, R. R., Ronksley, P. E., Sood, M. M., Hemmelgarn, B. R., Tangri, N., Ferguson, T. W., Tonelli, M. A., Ravani, P., et al. (2022) · Kidney International · doi:10.1016/j.kint.2022.02.034 · Cited by 13 · see citing works
Mühlenbruch, K., Kuxhaus, O., di Giuseppe, R., Boeing, H., Weikert, C., & Schulze, M. B. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.12.019 · Cited by 13 · see citing works
Engelmann, C., Verma, N., Qi, T., Kerbert, A. J. C., Pohl, J., Morgan, C., Kowalski, M., Andreola, F., Silva, M. O., Hernández, J. L. P., et al. (2026) · Journal of Hepatology · doi:10.1016/j.jhep.2026.02.017 · Cited by 12 · see citing works
Choi, J., Lee, H., Kuang, A., Huerta‐Chagoya, A., Scholtens, D. M., Choi, D.-H., Han, M., Lowe, W. L., Manning, A. K., Jang, H. C., et al. (2024) · Diabetes Care · doi:10.2337/dc24-0022 · Cited by 13 · see citing works
Venous thromboembolism and transfusion after major abdominopelvic surgery
McAlpine, K., Breau, R. H., Knee, C., Carrier, M., Violette, P. D., van Walraven, C., McIsaac, D. I., Mallick, R., Cagiannos, I., MORASH, C. G., et al. (2019) · Surgery · doi:10.1016/j.surg.2019.05.050 · Cited by 12 · see citing works
Criteria for evaluating risk prediction of multiple outcomes
Dudbridge, F. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220929039 · Cited by 10 · see citing works
“Among several proposed measures the net reclassification index has attained a default status among practitioners yet has received strong criticism.(51,52) Such issues are likely to be magnified when predicting multiple traits.”
BackgroundDynamic Amyloid PET: Relationships to 18F-Flortaucipir Tau PET Measures
Raman, F., Fang, Y.-H. D., Grandhi, S., Murchison, C., Kennedy, R., Morris, J. C., Massoumzadeh, P., Benzinger, T. L. S., Roberson, E. D., & McConathy, J. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.120.254490 · Cited by 12 · see citing works
van Hateren, K. J. J., Landman, G. W. D., Arnold, J. F. H., Joosten, H., Groenier, K. H., Navis, G., Sparwaßer, A., Bakker, S. J. L., Bilo, H. J. G., & Kleefstra, N. (2015) · PLoS ONE · doi:10.1371/journal.pone.0133065 · Cited by 12 · see citing works
Wang, W., Zhang, Y., Lee, E. T., Howard, B. V., Devereux, R. B., Cole, S. A., Best, L. G., Welty, T. K., Rhoades, E. R., Yeh, J. L., et al. (2017) · World Journal of Cardiovascular Diseases · doi:10.4236/wjcd.2017.75014 · Cited by 12 · see citing works
Israr, M. Z., Heaney, L. M., Ng, L. L., & Suzuki, T. (2018) · American Heart Journal · doi:10.1016/j.ahj.2018.02.016 · Cited by 12 · see citing works
Commonly Used Patient-Reported Outcomes Do Not Improve Prediction of COPD Exacerbations
Strassmann, A., Frei, A. L., Haile, S. R., ter Riet, G., & Puhan, M. A. (2017) · CHEST Journal · doi:10.1016/j.chest.2017.09.003 · Cited by 11 · see citing works
Wang, Y., Zhou, C., Ying, L., Lee, E., Chan, H., Chughtai, A. R., Hadjiiski, L. M., & Kazerooni, E. A. (2024) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.230196 · Cited by 11 · see citing works
Mortality Risk Prediction Models for People With Kidney Failure
Jarrar, F., Pasternak, M., Harrison, T. G., James, M. T., Quinn, R. R., Lam, N. N., Donald, M., Elliott, M. J., Lorenzetti, D., Strippoli, G. F., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.53190 · Cited by 11 · see citing works
Winkel, P., Jakobsen, J. C., Hilden, J., Lange, T., Jensen, G. B., Kjøller, E., Sajadieh, A., Kastrup, J., Kolmos, H. J., Larsson, A., et al. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0009-y · Cited by 11 · see citing works
“biomarker, even in independent data [50].”
Li, H., Huang, W., Wang, S., Balasubramanian, P. S., Wu, G., Fang, M., Xie, X., Zhang, J., Dong, D., Tian, J., et al. (2023) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-023-00149-0 · Cited by 11 · see citing works
Rentroia‐Pacheco, B., Bellomo, D., Lakeman, I. M. M., Wakkee, M., Hollestein, L. M., & Klaveren, D. V. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02213-6 · Cited by 10 · see citing works
Evidence-Based Diagnosis: An Introduction to Clinical Epidemiology
Newman, T. B., & Kohn, M. A. (2020) · Cited by 10 · see citing works
European Heart Journal quality standards
Alfonso, F., Torp-Pedersen, C. T., Carter, R. E., & Crea, F. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab324 · Cited by 10 · see citing works
Lanting, V. R., Vágó, E., Horváth–Puhó, E., Mulder, F. I., Nisio, M. D., Kamphuisen, P. W., Pedersen, L. H., van Es, N., & Sørensen, H. T. (2024) · Journal of Thrombosis and Haemostasis · doi:10.1016/j.jtha.2024.10.021 · Cited by 9 · see citing works
- Authors
- Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J.
- Affiliations
- Fred Hutch Cancer Center (US)
- The University of Texas MD Anderson Cancer Center (US)
- University of Copenhagen (DK)
- Corresponding author
- Pepe, Margaret S.
- Venue
- Statistics in Biosciences
- Published
- 2014-08-23
- Volume
- 7
- Issue
- 2
- Pages
- 282-295
- ISSN
- 1867-1764 (ISSN-L)
- 1867-1772
- PMC
- PMC4615606
- OpenAlex
- W2014187955
- Cited by
- 191
- References
- 27
- Citations by year
- 2026: 14 · 2025: 18 · 2024: 18 · 2023: 21 · 2022: 20 · 2021: 22 · 2020: 18 · 2019: 15 · 2018: 16 · 2017: 12 · 2016: 12 · 2015: 3 · 2014: 2
- FWCI
- 10.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Statistics in Biosciences (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Funders
- National Institutes of Health
- National Cancer Institute (R01CA152089, P30CA015704, U24CA086368)
- National Institute of General Medical Sciences (R01GM054438)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- net reclassification improvement, risk prediction model, statistical properties, overfitting, risk function, ROC curve, net benefit, Brier score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
