Cited by 191Statistical Methods in EpidemiologyStatistical Methods and InferenceAdvanced Causal Inference Techniques

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Open access

Pepe, Margaret S., Fan, Jing, Feng, Ziding, Gerds, Thomas, Hilden, Jorgen·2015·Statistics in Biosciences

DOI 10.1007/s12561-014-9118-0PMC PMC4615606

Open this paper in CoChat — free account

Abstract

The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1007/s12561-014-9118-0

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 2×
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · cited 3×
“’ See Gneiting and Raftery [3] for an in-depth discussion of proper scoring rules.”
Influential
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Operating Characteristics of Prostate-Specific Antigen in Men With an Initial PSA Level of 3.0 ng/mL or Lower
Thompson, I. M. (2005) · JAMA · doi:10.1001/jama.294.1.66 · Cited by 701 · cited 1×
“An example of a biomarker with this sort of predictive power is prostate specific antigen [19], one of the few cancer biomarkers currently used in cancer screening.”
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 595 · cited 1×
“Our results underscore the need to check if risk functions fit the test data set as a crucial part of the exercise of evaluating risk prediction models [18].”
Net Reclassification Indices for Evaluating Risk Prediction Instruments
Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & PEPE, M. S. (2013) · Epidemiology · doi:10.1097/ede.0000000000000018 · Cited by 454 · cited 1×
“have recently provided a critical review of the NRI statistic that focuses on conceptual aspects [8].”

Show all 27 references

Sorted by citations

Items 101–191 of 191

Predicting cognitive decline from neuropsychiatric symptoms and Alzheimer's disease biomarkers: A machine learning approach to a population-based data
Shah, J., Krell‐Roesch, J., Forzani, E., Knopman, D. S., Jack, C. R., Petersen, R. C., Che, Y., Wu, T., & Geda, Y. E. (2025) · Journal of Alzheimer s Disease · doi:10.1177/13872877241306654 · Cited by 10
“49 Pepe et al. (2014) 47 suggested that the AUC difference ( Δ AUC) between the full and reduced models and the Brier Score difference ( Δ Brier) demonstrate more consistent performance in re ﬂ ecting true prediction improvement.”
Weighted Brier Score—An Overall Summary Measure for Risk Prediction Models with Clinical Utility Consideration
Zhu, K., Zheng, Y., & Chan, K. C. G. (2025) · Statistics in Biosciences · doi:10.1007/s12561-025-09505-5 · Cited by 9
“The predicted risk for Model 3, r 3 , is generated based on r 2 as: Construction of Model 3 with over-fitted risk is similar to the examples from Hilden and Gerds (2014) and Pepe et al. (2015) demonstrating the non-properness of IDI and NRI.”
Incremental prognostic value of renal function for stroke prediction in atrial fibrillation
O'Brien, E. C., Holmes, D. N., Thomas, L. E., Singer, D. E., Fonarow, G. C., Mahaffey, K. W., Kowey, P. R., Hylek, E. M., Pokorney, S. D., Ansell, J., et al. (2018) · International Journal of Cardiology · doi:10.1016/j.ijcard.2018.07.113 · Cited by 9
Tissue-tracking in the assessment of late gadolinium enhancement in myocarditis and myocardial infarction
Doimo, S., Ricci, F., Aung, N., Cooper, J. A., Boubertakh, R., Sanghvi, M. M., Sinagra, G., & Petersen, S. E. (2020) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2020.08.008 · Cited by 8
Plasma Extracellular Vesicle Serpin G1 and CD14 Levels are Associated with Major Adverse Cardiovascular Events and Major Adverse Limb Events in Patients Undergoing Femoral Endarterectomy
Verwer, M. C., Mekke, J. M., Timmerman, N., van de Pol, Q. Y., Frissen, C., Pasterkamp, G., de Borst, G. J., Hazenberg, C. E. V. B., & de Kleijn, D. P. V. (2022) · European Journal of Vascular and Endovascular Surgery · doi:10.1016/j.ejvs.2022.10.045 · Cited by 8
“34 Comparing the AUC of models has its limitations too, as it is somewhat optimistic when case controls are imbalanced, it does not take goodness of ﬁ t of the model into account, and it summarises the test performance across the whole reporter operating characteristic space, even when these are…”
Assess the Performance and Cost-Effectiveness of LACE and HOSPITAL Re-Admission Prediction Models as a Risk Management Tool for Home Care Patients: An Evaluation Study of a Medical Center Affiliated Home Care Unit in Taiwan
Su, M.-C., Wang, Y.-J., Chen, T., Chiu, S.-H., Chang, H., Huang, M.-S., Hu, L.-H., Li, C.-C., Yang, S.-J., Wu, J., et al. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17030927 · Cited by 8
“However, the NRI is not without problems and may be misleading in certain conditions [32].”
Background
Serum insulin-like growth factor binding protein 3 as a promising diagnostic and prognostic biomarker in esophagogastric junction adenocarcinoma
Ding, T.-Y., Peng, Y., Hong, C., Huang, B.-L., Liu, C.-T., Luo, Y., Chu, L., Zhang, B., Li, X.-H., Qu, Q.-Q., et al. (2022) · Discover Oncology · doi:10.1007/s12672-022-00591-1 · Cited by 7
Development of a plasma-free amino acid-based risk score for the incidence of cardiovascular diseases in a general population: The Nagahama study
Takeshita, M., Tabara, Y., Setoh, K., Nagao, K., Imaizumi, A., Kageyama, Y., Matsuda, F., Tabara, Y., Kawaguchi, T., Setoh, K., et al. (2023) · Clinical Nutrition · doi:10.1016/j.clnu.2023.10.024 · Cited by 7
A Case–Control Study of Socio-Economic and Nutritional Characteristics as Determinants of Dental Caries in Different Age Groups, Considered as Public Health Problem: Data from NHANES 2013–2014
Zanella-Calzada, L. A., Galvan-Tejada, C. E., Chávez-Lamas, N. M., Gracia-Cortés, M. D. C., Moreno-Báez, A., Arceo-Olague, J. G., Celaya-Padilla, J. M., Galván-Tejada, J. I., & Gamboa-Rosales, H. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15050957 · Cited by 7
Biomarkers for Prediction of Stroke and Bleeds in Atrial Fibrillation
Hylek, E. M. (2019) · Circulation · doi:10.1161/circulationaha.118.038635 · Cited by 7
Urinary Neutrophil Gelatinase–Associated Lipocalin/Hepcidin-25 Ratio for Early Identification of Patients at Risk for Renal Replacement Therapy After Cardiac Surgery: A Substudy of the BICARBONATE Trial
Elitok, S., Kuppe, H., Devarajan, P., Bellomo, R., Isermann, B., Westphal, S., Kube, J., Albert, C., Ernst, M., Kropf, S., et al. (2021) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000005741 · Cited by 6
Incorporating polygenic risk into the Leicester Risk Assessment score for 10-year risk prediction of type 2 diabetes
Liu, X., Littlejohns, T. J., Bešević, J., Bragg, F., Clifton, L., Collister, J. A., Trichia, E., Gray, L. J., Khunti, K. K., & Hunter, D. J. (2024) · Diabetes & Metabolic Syndrome Clinical Research & Reviews · doi:10.1016/j.dsx.2024.102996 · Cited by 6
“Previous literature has identified limitations with using NRI as a measure to assess prediction improvement between new and old models, however, bias is reduced when using larger datasets with a small number of predictors [31,32].”
A Novel Nomogram for Predicting Morbidity Risk in Patients with Secondary Malignant Neoplasm of Bone and Bone Marrow: An Analysis Based on the Large MIMIC-III Clinical Database
Miao, G., Li, Z., Chen, L., Li, W., Lan, G., Chen, Q., Luo, Z., Liu, R., & Zhao, X. (2022) · International Journal of General Medicine · doi:10.2147/ijgm.s352761 · Cited by 6
“Model accuracy was compared by determining net reclassification improvement (NRI), with the effectiveness of improvements determined by calculating integrated discrimination improvement (IDI).(11) The clinical value of the predicted models was determined by decision curve analysis (DCA).”
Method
Development of an ensemble machine learning prognostic model to predict 60-day risk of major adverse cardiac events in adults with chest painPreprint
Kennedy, C. J., Mark, D. G., Huang, J., van der Laan, M. J., Hubbard, A., & Reed, M. E. (2021) · medRxiv · doi:10.1101/2021.03.08.21252615 · Cited by 6
Calibration of models is not sufficient to justify NRI
Gerds, T. A., & Hilden, J. (2014) · Statistics in Medicine · doi:10.1002/sim.6212 · Cited by 5
Blood transcriptomics as non-invasive marker for kidney transplant rejection
Loon, E. V., & Naesens, M. (2021) · Néphrologie & Thérapeutique · doi:10.1016/j.nephro.2020.02.012 · Cited by 5
Additional Effect of Coronary High-Intensity Plaque on T1-Weighted Magnetic Resonance Imaging With Circulating Malondialdehyde-Modified Low-Density Lipoprotein on Cardiac Events
Hiraya, D., Sato, A., Hoshi, T., Sakai, S., Watabe, H., & Ieda, M. (2021) · Circulation Journal · doi:10.1253/circj.cj-21-0220 · Cited by 5
Improved 30 day Heart Failure Rehospitalization Prediction through the Addition of Device-Measured Parameters
Brown, J. R., Alonso, Á., Mazimba, S., Warman, E. N., & Bilchick, K. C. (2020) · ESC Heart Failure · doi:10.1002/ehf2.12956 · Cited by 5
Risk Prediction in Cardio-Oncology: Conceptual and Methodological Considerations
Sen, J., Amir, E., Austin, P. C., Marwick, T. H., McIntosh, C., Thavendiranathan, P., & Abdel-Qadir, H. M. (2026) · JACC CardioOncology · doi:10.1016/j.jaccao.2026.01.006 · Cited by 5
Development and validation of a point‐of‐care nursing mobile tool to guide the diagnosis of malnutrition in hospitalized adult patients: a multicenter, prospective cohort study
Lin, N., Zhou, X., Chen, W., He, C., Wang, X., Wei, Y., Long, Z., Shen, T., Zhong, L., Yang, C., et al. (2024) · MedComm · doi:10.1002/mco2.526 · Cited by 4
“To further validate our model’s superiority, we conducted DeLong test, 22 Net Reclassification Improvement (NRI) test 23 and Integrated Discrimination Improvement (IDI) test.”
Method
Development and Validation of Prediction Models for Incident Reversible Cognitive Frailty Based on Social-Ecological Predictors Using Generalized Linear Mixed Model and Machine Learning Algorithms: A Prospective Cohort Study
Liu, Q., Si, H., Li, Y., Zhou, W., Yu, J., Bian, Y., & Wang, C. (2024) · Journal of Applied Gerontology · doi:10.1177/07334648241270052 · Cited by 4
“Nevertheless, we need to be cautious about these ﬁ ndings because when models are over ﬁ t to training data, NRI tends to suffer from optimistic bias even in independent test datasets (Pepe et al., 2015).”
Predicting stroke with machine learning techniques in a sub-Saharan African population
Aribisala, B. S., Edward, D., Ogbole, G. I., Akpa, O. M., Ayilara, S., Sarfo, F. S., Olabanjo, O. A., Fakunle, A. G., Macaulay, B. O., Yaria, J. O., et al. (2025) · Neuroscience Informatics · doi:10.1016/j.neuri.2025.100216 · Cited by 4
Establishment and Validation of a Model for Disease-Free Survival Rate Prediction Using the Combination of microRNA-381 and Clinical Indicators in Patients with Breast Cancer
Shen, J., Wang, M., Fan, L., Yan, H., Wang, R., & Zhou, J. Y. (2022) · Breast Cancer Targets and Therapy · doi:10.2147/bctt.s383121 · Cited by 4
Clinical utility gains from incorporating comorbidity and geographic location information into risk estimation equations for atherosclerotic cardiovascular disease
Xu, Y.-Z., Foryciarz, A., Steinberg, E., & Shah, N. (2022) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocad017 · Cited by 3
A modified net reclassification improvement statistic
Heller, G. (2023) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2023.03.001 · Cited by 3
“To determine its null reference distribution, Pepe et al. (2013) demonstrate that for correctly specified nested models (C1), ρ(θ0;θ ; π0) = 0 iff γ0 = 0.”
Method
Common pitfalls in critical care research
Besen, B. A. M. P., Nassar, A. P., Ferreira, J. C., & Ranzani, O. (2025) · Critical Care Science · doi:10.62675/2965-2774.20250339 · Cited by 3
“(11) While Bayesian analysis, (12) machine learning, and artificial intelligence (13,14) are in the spotlight, they should not drive the research question.”
Machine learning prediction of overall survival in patients with cT1b renal cell carcinoma after surgical resection using the SEER database
Zhang, Z., Chen, L., Chen, Z., Guan, S., He, D., Song, H., Jiang, F., Sun, W., Tian, F., Lv, L., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-22342-2 · Cited by 3
“…NRI is more widely used to compare the accuracy of two predictive models, and IDI reflects the change in the gap between the predictive probabilities of the two models, with an improvement in NRI or IDI > 0 indicating that the new model has improved predictive ability over the old model 27,28 .”
A methodology review on the incremental prognostic value of computed tomography biomarkers in addition to Framingham risk score in predicting cardiovascular disease: the use of association, discrimination and reclassification
Pang, C. L., Pilkington, N., Wei, Y., Peters, J., Roobottom, C. A., & Hyde, C. (2018) · BMC Cardiovascular Disorders · doi:10.1186/s12872-018-0777-5 · Cited by 3
“Transparent reporting of NRI should be compulsory, for example use of a reclassification table (38), and readers should be aware of the controversies surrounding NRI [80].”
Method
The Evolving Cardiovascular Disease Risk Scores for Persons with Diabetes Mellitus
Zhao, Y., & Wong, N. D. (2018) · Current Cardiology Reports · doi:10.1007/s11886-018-1069-5 · Cited by 3
The value of blood-based measures of liver function and urate in lung cancer risk prediction: A cohort study and health economic analysis
Horsfall, L., Clarke, C. S., Nazareth, I., & Ambler, G. (2023) · Cancer Epidemiology · doi:10.1016/j.canep.2023.102354 · Cited by 3
“The Net Reclassification Index is particularly popular for evaluating the improvement in the prediction performance but has been criticised along with measures of sensitivity and specificity [34,38,39].”
Methodological concerns about “concordance-statistic for benefit” as a measure of discrimination in predicting treatment benefit
Xia, Y., Gustafson, P., & Sadatsafavi, M. (2023) · Diagnostic and Prognostic Research · doi:10.1186/s41512-023-00147-z · Cited by 3
“(2015) expanded on the concept of properness for metrics that evaluate the improvement in prediction performance gained by adding extra covariates [16].”
Background
Visualising and quantifying the usefulness of new predictors stratified by outcome class: The U-smile method
Kubiak, K. B., Więckowska, B., Jodłowska-Siewert, E., & Guzik, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303276 · Cited by 3
“Propriety is a desired feature of a performance metric since adding a superfluous predictor to the reference model does not increase the values of the proper measures but can increase the I [30].”
Background
Enhancing Cardiovascular Risk Prediction with a Simplified Carotid IMT Protocol: Evidence from the IMPROVE Study
Veglia, F., Malagoni, A. M., Amato, M., Strawbridge, R. J., Savonen, K. P., Giral, P., Gallo, A., Pirro, M., Gigante, B., Eriksson, P. O., et al. (2025) · Biomedicines · doi:10.3390/biomedicines13030584 · Cited by 3
“Reclassification analysis, with computation of the NRI (Net Reclas-sification Improvement) and the IDI (Integrated Discrimination Improvement), was not employed because it has been repeatedly criticized, mainly for the high number of false positives produced [22–24].”
A fundamental principle of coronary pathophysiology for risk stratifying coronary artery disease
Gould, K. L., & Johnson, N. P. (2020) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jeaa333 · Cited by 3
Beyond Global Metrics: The U-Smile Method for Explainable, Interpretable, and Transparent Variable Selection in Risk Prediction Models
Kubiak, K. B., Konieczna, A., Tyrańska-Fobke, A., & Więckowska, B. (2025) · Applied Sciences · doi:10.3390/app15158303 · Cited by 2
Cardiac magnetic resonance versus echocardiography: An analysis examining risk reclassification by left ventricular ejection fraction and cost burden analysis
Tsay, A. J., Sangha, K., Lee, L., Wang, S., Liu, S., Zahid, A. M., Poonawalla, M., Lang, R. M., Kramer, C. M., & Patel, A. R. (2025) · Journal of Cardiovascular Magnetic Resonance · doi:10.1016/j.jocmr.2025.101959 · Cited by 2
“NRI statistics have been criticized as being flawed in that they focus on the “direction” of the “risk reclassification” and not the “magnitude” of change, and therefore should be used to “compare risk models” and not describe a “model’s performance” [45,46].”
Developing risk prediction models for type 2 diabetes and assessing the role of circulating metabolic biomarkers in five independent Finnish cohorts with over 22,000 individuals
Kiviniemi, E., Karhunen, V., Perola, M., Kettunen, J., & Sillanpää, M. J. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111978 · Cited by 3
Validation of a Disease-Free Survival Prediction Model Using UBE2C and Clinical Indicators in Breast Cancer Patients
Shen, J., Yan, H., Yang, C., Lin, H., Li, F., & Zhou, J. Y. (2023) · Breast Cancer Targets and Therapy · doi:10.2147/bctt.s402109 · Cited by 2
Establishment and Validation of a Predictive Model for Post-Treatment Anxiety Based on Patient Attributes and Pre-Treatment Anxiety Scores
Sun, W.-W., Shen, J., Sun, R., Zhou, D., & Li, H.-H. (2023) · Psychology Research and Behavior Management · doi:10.2147/prbm.s425055 · Cited by 2
“22 In our subsequent model comparison, we compared the differences between each model individually.”
Method
External validation of a novel HR-pQCT fracture risk assessment tool (μFRAC) in a female cohort: Sahlgrenska University Hospital Prospective Evaluation of Risk of Bone Fractures (SUPERB) study
Bugbird, A. R., Jaiswal, R., Whittier, D. E., Boyd, S. K., & Lorentzon, M. (2025) · JBMR Plus · doi:10.1093/jbmrpl/ziaf117 · Cited by 2
“Despite the known limitations of the NRI measure, 42,43 we found it to be a useful complement to the findings based on the ROC/AUC analysis.”
Severity of Illness Scores and Biomarkers for Prognosis of Patients with Coronavirus Disease 2019
Cavallazzi, R., Bradley, J. A., Chandler, T. R., Furmanek, S. P., & Ramírez, J. A. (2023) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-0042-1759567 · Cited by 2
Bringing Order to Disorder
Wu, K. C. (2018) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2018.07.002 · Cited by 2
“The incremental value of including entropy in a prediction model may also be overemphasized by the use of the net reclassification index, which can be misleading (7).”
BackgroundMethod
Cardiovascular risk estimation: can a risk prediction model derived in one country be used in another?
Graham, I. M. (2024) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwae286 · Cited by 2
Development and Validation of a Clinical Model for Predicting Delay in Postoperative Transfer Out of the Post-Anesthesia Care Unit: A Retrospective Cohort Study
Xie, G.-H., Shen, J., Li, F., Yan, H., & Qian, Y. X. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s458784 · Cited by 1
“We further examined the differences between model 6 and the other models using NRI and IDI and did not find any significant differences (Figures 4 and 5 and Table 3).”
MethodInfluential
Prediction of cardiovascular events by algorithm- and formula-based pulse wave velocity
Desbiens, L., Veillette, S., Fortier, C., Nadeau‐Fredette, A., Hametner, B., Wassertheurer, S., Madore, F., Agharazii, M., & Goupil, R. (2025) · Journal of Hypertension · doi:10.1097/hjh.0000000000004176 · Cited by 1
A novel cumulative social advantage metric for enhancing the cardiovascular health construct and predicting mortality risk
Wood, E. P., Martinez, S., Santos, M. G., Mullachery, P. H., Isasi, C. R., Duncan, D. T., & Makarem, N. (2025) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwaf752 · Cited by 1
Characteristics of Cardiovascular Disease Prediction Models Considering Mental Disorders: A Systematic Review
Siddiqi, S., Haggerty, T., Akther, A., Rusu, S., Blair, D.-L., Eccles, H., Yu, J., Shorr, R., Gupta, A., Bouchard, K., et al. (2026) · Journal of the American Heart Association · doi:10.1161/jaha.125.046964 · Cited by 1
“94,95 Considering model performance with inclusion of mental disorders as covariates, most studies demonstrated moderate performance metrics (c=~0.70).”
Influential
The Asymptomatic Patient With a Pulmonary Nodule
Nair, V. S., & Gould, M. K. (2018) · Elsevier eBooks · doi:10.1016/b978-0-323-48565-4.00001-1 · Cited by 1
Elevated Risk of Adverse Prognosis in Patients with T2-3 Stage Breast Cancer Exhibiting Non-Pathological Complete Response Following Neoadjuvant Chemotherapy: Significance of Regenerating Islet-Derived Family Member 4
Fan, L., Chen, C.-G., Wei, J.-F., Lin, J., Dou, Z.-A., Shen, J., & Li, S. (2024) · Breast Cancer Targets and Therapy · doi:10.2147/bctt.s473920 · Cited by 1
Development and Validation of a Predictive Model for Anxiety Trajectories in Patients with Breast Cancer: A Retrospective Study
Li, X., Wei, B.-K., Li, F., Yan, H., & Shen, J. (2025) · Psychology Research and Behavior Management · doi:10.2147/prbm.s501127 · Cited by 1
“28 The results indicated that the Lm model demonstrated superior predictive capability compared to Model 1 and Model 2 in both the training and testing sets.”
“Trending” statistical methods
Howard, G., & Long, D. L. (2018) · Research and Practice in Thrombosis and Haemostasis · doi:10.1002/rth2.12075 · Cited by 1
Assessing predictive accuracy of survival regressions subject to nonindependent censoring
Wang, M., Long, Q., Chen, C., & Zhang, L. (2019) · Statistics in Medicine · doi:10.1002/sim.8420 · Cited by 1
Discussion of “A Risk-Based Measure of Time-Varying Prognostic Discrimination for Survival Models,” by C. Jason Liang and Patrick J. Heagerty
Gerds, T. A., & Schumacher, M. C. (2016) · Biometrics · doi:10.1111/biom.12629 · Cited by 1
Accurate prediction of coronary and cerebrovascular events: Adding small pieces to an emerging puzzle
Brener, S. J. (2017) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2017.06.015 · Cited by 1
Comments on “Net reclassification index at event rate: Properties and relationships”
Vickers, A. J. (2019) · Statistics in Medicine · doi:10.1002/sim.7631 · Cited by 1
Reply to: “Not yet time to A-Tango in redefining acute-on-chronic liver failure”
Engelmann, C., Verma, N., & Jalan, R. (2026) · Journal of Hepatology · doi:10.1016/j.jhep.2026.04.007 · Cited by 1
Efficient design and modeling strategies for follow-up studies with time-varying covariates
Reinikainen, J. (2015) · Jyväskylä University Digital Archive (University of Jyväskylä)
Investigating Predictive Disease Model Transportability through Cohort Simulation and Causal Analysis
Singleton, K. (2016) · eScholarship (California Digital Library)
Well‐Discriminated Forecasts
Gross, S. T. (2017) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat07884
Absolute Risk. Ruth M. Pfeiffer and Mitchell H. Gail. Boca Raton: CRC Press.
Gerds, T. A. (2019) · Biometrics · doi:10.1111/biom.13127
Does Pediatric Index of Mortality “Score” in Colombia?
Rahiman, S. N. (2020) · Indian Journal of Critical Care Medicine · doi:10.5005/jp-journals-10071-23647
Perfusion imaging combined with coronary calcium scoring. A step further towards individualized medicine?
Frost, L., Gerds, T. A., & Vinter, N. (2021) · IJC Heart & Vasculature · doi:10.1016/j.ijcha.2021.100845
“The authors examined the predictive performance using the Net Reclassiﬁcation Index (NRI).”
Background
Risk Predictions
Newman, T. B., & Kohn, M. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108500111.007
Response
Pepe, M. S. (2014) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dju356
Response to Re: Modified Sequential Organ Failure Assessment score for predicting mortality in emergency department patients with sepsis
Vosseteig, A., & Jones, P. (2023) · Emergency Medicine Australasia · doi:10.1111/1742-6723.14197
Zur Bedeutung von mechanischer Beatmung und Lungenversagen als Risikofaktoren im akut-auf-chronischen Leberversagen
Mengers, J. (2022) · doi:10.21248/gups.73410
Peripheral artery disease
Verwer, M. C. (2023) · doi:10.33540/1839
Traditional Versus New Models of Risk Prediction
Olesen, T. B., & Olsen, M. H. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-15512-3.00019-2
Development and validation of a neoadjuvant chemotherapy pathological complete remission model based on Reg IV expression in breast cancer tissues: a clinical retrospective study
Wei, J.-F., Li, F., Lin, J., Dou, Z.-A., Li, S., & Shen, J. (2024) · Breast Cancer · doi:10.1007/s12282-024-01609-y
“NRI and IDI were used to assess differences between various models [12, 13].”
Method
Prognostic value of left ventricular mass measured on coronary computed tomography angiography
Abiragi, M., Chen, M. T., Lin, B., Gransar, H., Dey, D., Slomka, P. J., Hayes, S. W., Thomson, L. E., Friedman, J. D., Berman, D. S., et al. (2024) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2024.10.010
Precautions to Consider in the Analysis of Prognostic and Predictive Indices
Chartier, L., Belot, A., Chaillol, I., Elsensohn, M., Portugues, C., Fournier, M., Joubert, C., Gat, E., Pizot, C., Fogarty, P., et al. (2024) · Journal of Nuclear Medicine · doi:10.2967/jnumed.123.267021
“…cation Index, the potential for misinterpretation of the Net Reclassi ﬁ cation Index (in terms of discrimination or calibration, whereas it assesses reclassi ﬁ cation improvement only), and the use of the category-free Net Reclassi ﬁ cation Index, which may lose the clinical intuition ( 16 – 18 ).”
Assessing Incremental Value of Added Predictors in Cardiovascular Risk Prediction Models
Gajjar, P., Nayor, M. G., & Duncan, M. S. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-15734-9.00030-3
Development and validation of a predictive model for disease-free progression in triple-negative breast cancer: A retrospective study using color Doppler ultrasound and magnetic resonance imaging
Fan, L., Yan, H., Wei, B.-K., & Shen, J. (2025) · The Breast · doi:10.1016/j.breast.2025.104560
“TNBC is a highly malignant form of breast tumor, accounting for approximately 10 – 15 % of all breast cancer cases [22]., TNBC is distinguished by its earlier age of onset, high metastatic potential, the highest recurrence rate, and the poorest survival prognosis, which has attracted significant…”
Improving the diagnostic performance for prior COVID-19 with T-SPOT, an interferon-gamma release assay
Yamamoto, M., White, M. T., Kawaguchi, T., Setoh, K., Tsuda, Y., Shinohara, K., Tsuchido, Y., Matsumura, Y., Matsuda, F., & Nagao, M. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1675605
Additional file 5 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647447
Additional file 4 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647444
Additional file 3 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647441
Additional file 2 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647438
Additional file 6 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647450
Additional file 1 of Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., li, J.-J., 左阿芳, Wang, S.-B., & Han, F. (2023) · Figshare · doi:10.6084/m9.figshare.23647435
Predicting Recurrence and Outcomes After Stressor‐Associated Atrial Fibrillation Using ECG‐Based Deep Learning
Haimovich, J. S., Friedman, S. F., Reeder, C., Dsouza, V., Sommers, T. F., Usuda, K., Kany, S., Benjamin, E. J., Lubitz, S. A., Maddah, M., et al. (2026) · Journal of the American Heart Association · doi:10.1161/jaha.125.047146
Pronostic dynamique de l'évolution de l'état de santé de patients atteints d'une maladie chronique
Fournier, M. (2016) · doi:10.70675/14b87590z5783z4f17za37eze3cec25ff57e
Incremental value of placental miR-7641 and ferroptosis-related phenotypes in risk stratification of severe preeclampsia
Hu, H., Wang, R., Wang, X. R., & Hu, M. (2026) · Frontiers in Physiology · doi:10.3389/fphys.2026.1834708
APOE genetic variation ε4 allele associate with an increased susceptibility to intracerebral hemorrhage in a Chinese Han populationPreprint
Zhang, L., Wu, C., Huang, Y., He, L., Wang, Q., & Yang, L. Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9933317/v1
Head-to-Head Comparison of the Prognostic Value of Actual Versus Percent Predicted 6-Minute Walk Distance in Patients With Heart Failure
Scrutinio, D., Guida, P., & Passantino, A. (2026) · Journal of Cardiopulmonary Rehabilitation and Prevention · doi:10.1097/hcr.0000000000001064
Can Patient-Reported Anxiety Improve Prediction of Heart Failure Functional Status Beyond Electronic Medical Record Data? A Pilot Observational Study (Preprint)Preprint
Moudden, I. E., Adeleke, S., Ruiz-Avila, D., Chen, C., Lynch, T., Robles, M., Mungee, S., Wurster, J., & Dodani, S. (2026) · doi:10.2196/preprints.110512
Lipidomic profiles and their relationship to incident diabetic retinopathy in type 2 diabetes: The Fremantle Diabetes Study Phase II
Davis, T. M. E., Drinkwater, J. J., James, A. G.-S., Holmes, E. C., Davis, W. A., & Gray, N. (2026) · Journal of Diabetes and its Complications · doi:10.1016/j.jdiacomp.2026.109412
Patient-Reported Anxiety and Cross-Sectional Classification of Heart Failure Functional Status Beyond Electronic Medical Record Data: A Feasibility Pilot Study (Preprint)Preprint
Moudden, I. E., Adeleke, S., Ruiz-Avila, D., Chen, C., Lynch, T., Robles, M., Mungee, S., Wurster, J., & Dodani, S. (2026) · doi:10.2196/preprints.111721
Clinical Prediction Models in Cardiovascular Disease: Foundations, Clinical Applications and Future Directions
Wang, B., Franciosi, A. N., O’Neill, L. M., Holden, S., McDonald, K., Gallagher, J., & Doran, P. (2026) · Journal of the American Heart Association · doi:10.1161/jaha.126.053665
A multimodal model for clinically defined MCI integrating plasma biomarkers and brain microstructural features: a dual-cohort external validation study
Ke, J., Xuekelati, S., Maimaitiwusiman, Z., Halan, B., Xu, Q., Guo, S., Xu, L., & Wang, H. (2026) · Neurological Sciences · doi:10.1007/s10072-026-09378-6
“…and IDI showed sup - portive probability reclassification in several comparisons, but these analyses were considered exploratory because no clini - cally meaningful risk categories were prespecified and con - tinuous NRI may be numerically large despite modest changes in rank discrimination [ 18 ].”
Beyond TRISS: A Contemporary Expected Mortality Model for Trauma quality review—Trauma Quality Improvement Program, 2017–2024
Dallal, R. M., Abdelhamid, S. S., Cuellar, E., Emery, R. J., Kaplan, M., Bloom, A., Moran, B. J., & Leung, P. S. (2026) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000005201

Authors: Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J.
Affiliations: Fred Hutch Cancer Center (US)
The University of Texas MD Anderson Cancer Center (US)
University of Copenhagen (DK)
Corresponding author: Pepe, Margaret S.
ORCID: Fan, Jing
Feng, Ziding
Gerds, Thomas
Venue: Statistics in Biosciences
Publisher: Springer Science and Business Media LLC
Published: 2014-08-23
Volume: 7
Issue: 2
Pages: 282-295
ISSN: 1867-1764 (ISSN-L)
1867-1772
DOI: 10.1007/s12561-014-9118-0
PMC: PMC4615606
OpenAlex: W2014187955
Cited by: 191
References: 27
Citations by year: 2026: 14 · 2025: 18 · 2024: 18 · 2023: 21 · 2022: 20 · 2021: 22 · 2020: 18 · 2019: 15 · 2018: 16 · 2017: 12 · 2016: 12 · 2015: 3 · 2014: 2
FWCI: 10.21
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: www.springer.com/tdm (tdm)
Locations: Statistics in Biosciences (publishedVersion)
PubMed (publishedVersion)
Europe PMC (PubMed Central) (acceptedVersion)
PubMed Central (acceptedVersion)
Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
Funders: National Institutes of Health
National Cancer Institute (R01CA152089, P30CA015704, U24CA086368)
National Institute of General Medical Sciences (R01GM054438)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: net reclassification improvement, risk prediction model, statistical properties, overfitting, risk function, ROC curve, net benefit, Brier score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks