Cited by 191Statistical Methods in EpidemiologyStatistical Methods and InferenceAdvanced Causal Inference Techniques

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Open access

Pepe, Margaret S., Fan, Jing, Feng, Ziding, Gerds, Thomas, Hilden, Jorgen·2015·Statistics in Biosciences

DOI 10.1007/s12561-014-9118-0PMC PMC4615606

Open this paper in CoChat — free account

Abstract

The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1007/s12561-014-9118-0

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 2×
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793 · cited 3×
“’ See Gneiting and Raftery [3] for an in-depth discussion of proper scoring rules.”
Influential
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483 · cited 3×
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Operating Characteristics of Prostate-Specific Antigen in Men With an Initial PSA Level of 3.0 ng/mL or Lower
Thompson, I. M. (2005) · JAMA · doi:10.1001/jama.294.1.66 · Cited by 701 · cited 1×
“An example of a biomarker with this sort of predictive power is prostate specific antigen [19], one of the few cancer biomarkers currently used in cancer screening.”
Clinical Prediction Models—a Practical Approach to Development, Validation and Updating
Bedogni, G. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_16.x · Cited by 595 · cited 1×
“Our results underscore the need to check if risk functions fit the test data set as a crucial part of the exercise of evaluating risk prediction models [18].”
Net Reclassification Indices for Evaluating Risk Prediction Instruments
Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & PEPE, M. S. (2013) · Epidemiology · doi:10.1097/ede.0000000000000018 · Cited by 454 · cited 1×
“have recently provided a critical review of the NRI statistic that focuses on conceptual aspects [8].”

Show all 27 references

Sorted by citations

Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697
Developing prediction models for clinical use using logistic regression: an overview
Shipe, M. E., Deppen, S. A., Farjah, F., & Grogan, E. L. (2019) · Journal of Thoracic Disease · doi:10.21037/jtd.2019.01.25 · Cited by 515
“However, this can be insensitive to detecting small changes in model performance, especially if the AUC of the model is already large, and the information may not be clinically relevant (24,25).”
Background
Clinical Implications of Revised Pooled Cohort Equations for Estimating Atherosclerotic Cardiovascular Disease Risk
Yadlowsky, S., Hayward, R. A., Sussman, J. B., McClelland, R. L., Min, Y., & Basu, S. (2018) · Annals of Internal Medicine · doi:10.7326/m17-3011 · Cited by 271
Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 271
Trimethylamine N-oxide and prognosis in acute heart failure
Suzuki, T., Heaney, L. M., Bhandari, S. S., Jones, D. J. L., & Ng, L. L. (2016) · Heart · doi:10.1136/heartjnl-2015-308826 · Cited by 253
Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Customized growth charts: rationale, validation and clinical benefits
Gardosi, J. O., Francis, A. I., Turner, S., & Williams, M. (2018) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2017.12.011 · Cited by 238
An Original Stepwise Multilevel Logistic Regression Analysis of Discriminatory Accuracy: The Case of Neighbourhoods and Health
Merlo, J., Wagner, P., Ghith, N. M. M., & Leckie, G. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153778 · Cited by 230
Three myths about risk thresholds for prediction models
on behalf of the Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Wynants, L., van Smeden, M., McLernon, D. J., Timmerman, D., Steyerberg, E. W., & Van Calster, B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1425-3 · Cited by 213
Cardiovascular disease: The rise of the genetic risk score
Knowles, J. W., & Ashley, E. A. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002546 · Cited by 182
The c-index is not proper for the evaluation of $t$-year predicted risks
Blanche, P., Kattan, M. W., & Gerds, T. A. (2018) · Biostatistics · doi:10.1093/biostatistics/kxy006 · Cited by 159
Polygenic Scores to Assess Atherosclerotic Cardiovascular Disease Risk
Aragam, K. G., & Natarajan, P. (2020) · Circulation Research · doi:10.1161/circresaha.120.315928 · Cited by 148
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models
Assel, M., Sjoberg, D., & Vickers, A. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0020-3 · Cited by 133
Area Deprivation Index and Cardiac Readmissions: Evaluating Risk‐Prediction in an Electronic Health Record
Johnson, A. E., Zhu, J., Garrard, W., Thoma, F. W., Mulukutla, S. R., Kershaw, K. N., & Magnani, J. W. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.020466 · Cited by 127
Clinical lipidomics: realizing the potential of lipid profiling
Meikle, T. G., Huynh, K., Giles, C., & Meikle, P. J. (2021) · Journal of Lipid Research · doi:10.1016/j.jlr.2021.100127 · Cited by 121
“While debate continues over the best approach (28), it will be important for any new test to provide some comparison to existing measures and clearly demonstrate an improvement in the clinical utility of lipidomic tests over existing measures.”
Background
Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation
Wahl, S., Boulesteix, A., Zierer, A., Thorand, B., & van de Wiel, M. A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0239-7 · Cited by 115
“The optimistic bias of NRI has lead to critical discussion [47].”
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 99
Breast Arterial Calcification: a Novel Cardiovascular Risk Enhancer Among Postmenopausal Women
Iribarren, C., Chandra, M., Lee, C., Sanchez, G., Sam, D. L., Azamian, F. F., Cho, H.-M., Ding, H., Wong, N. D., & Molloi, S. Y. (2022) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.121.013526 · Cited by 86
Application of the International IgA Nephropathy Prediction Tool one or two years post-biopsy
Barbour, S. J., Coppo, R., Zhang, H., Liu, Z., Suzuki, Y., Matsuzaki, K., Er, L., Reich, H. N., Barratt, J., Cattran, D., et al. (2022) · Kidney International · doi:10.1016/j.kint.2022.02.042 · Cited by 86
Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning
Huang, C., Li, S., Caraballo, C., Masoudi, F. A., Rumsfeld, J. S., Spertus, J. A., Normand, S. T., Mortazavi, B. J., & Krumholz, H. M. (2021) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007526 · Cited by 90
“11,48 Moreover, these summary statistics cannot assess the difference of the models in predicting risks for an individual patient.”
Background
The emerging field of polygenic risk scores and perspective for use in clinical care
Yanes, T., McInerney-Leo, A. M., Law, M. H., & Cummings, S. A. (2020) · Human Molecular Genetics · doi:10.1093/hmg/ddaa136 · Cited by 75
Cohen’s Kappa Coefficient as a Measure to Assess Classification Improvement following the Addition of a New Marker to a Regression Model
Więckowska, B., Kubiak, K. B., Jóźwiak, P., Moryson, W., & Stawińska-Witoszyńska, B. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191610213 · Cited by 82
“The reason for this is evident in the doubts raised regarding the correctness of some calculations and interpretations that have appeared in statistics journals [26] and in reputable medical journals [24,27–32].”
Result
Derivation and Validation of a Clinical Risk Assessment Model for Cancer-Associated Thrombosis in Two Unique US Health Care Systems
Li, A., La, J., May, S., Guffey, D., da Costa, W. L., Amos, C. I., Bandyo, R., Milner, E. M., Kurian, K. M., Chen, D. C. R., et al. (2023) · Journal of Clinical Oncology · doi:10.1200/jco.22.01542 · Cited by 76
Polygenic risk scores for the prediction of cardiometabolic disease
O’Sullivan, J. W., Ashley, E. A., & Elliot, P. M. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac648 · Cited by 74
Refining the CHA2DS2VASc risk stratification scheme: shall we drop the sex category criterion?
Yoshimura, H., Providencia, R. A., Finan, C., Schmidt, A. F., & Lip, G. Y. H. (2024) · EP Europace · doi:10.1093/europace/euae280 · Cited by 54
“49,50 In this analysis, repeated assessment of risk prediction schemes over time showed the same trends and results, thereby increasing the robustness of our findings.”
Lipidomics and Biomarker Discovery in Kidney Disease
Afshinnia, F., Rajendiran, T. M., Wernisch, S., Soni, T., Jadoon, A., Karnovsky, A., Michailidis, G. C., & Pennathur, S. (2018) · Seminars in Nephrology · doi:10.1016/j.semnephrol.2018.01.004 · Cited by 52
The clinical meaning of the area under a receiver operating characteristic curve for the evaluation of the performance of disease markers
Parodi, S., Verda, D., Bagnasco, F., & Marco Muselli, B. (2022) · Epidemiology and Health · doi:10.4178/epih.e2022088 · Cited by 47
Long‐Term Exposure to Elevated Systolic Blood Pressure in Predicting Incident Cardiovascular Disease: Evidence From Large‐Scale Routine Electronic Health Records
Solares, J. R. A., Canoy, D., Raimondi, F. E. D., Zhu, Y., Hassaïne, A., Salimi‐Khorshidi, G., Tran, J., Copland, E., Zottoli, M., Pinho‐Gomes, A., et al. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.012129 · Cited by 44
Omics and Cardiometabolic Disease Risk Prediction
Tahir, U. A., & Gerszten, R. E. (2020) · Annual Review of Medicine · doi:10.1146/annurev-med-042418-010924 · Cited by 45
Net Reclassification Improvement
Jewell, E. S., Maile, M. D., Engoren, M., & Elliott, M. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001141 · Cited by 40
Which N Descriptor Is More Predictive of Prognosis in Resected Non-small Cell Lung Cancer
Xu, L., Su, H., She, Y., Dai, C., Zhao, M., Gao, J., Xie, H., Ren, Y., Xie, D., & Chen, C. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.12.012 · Cited by 40
Stratification in systemic sclerosis according to autoantibody status versus skin involvement: a study of the prospective EUSTAR cohort
Elhaï, M., Sritharan, N., Boubaya, M., Balbir‐Gurman, A., Siegert, E., HACHULLA, E., de Vries‐Bouwstra, J., Riemekasten, G., Distler, J. H. W., Rosato, E., et al. (2022) · The Lancet Rheumatology · doi:10.1016/s2665-9913(22)00217-x · Cited by 39
Comparison of Mobility Indices for Predicting Early Death in Older Patients With Cancer: The Physical Frailty in Elderly Cancer Cohort Study
Pamoukdjian, F., Aparicio, T., Zebachi, S., Zelek, L. H., Paillaud, É., & Canouï‐Poitrine, F. (2019) · The Journals of Gerontology Series A · doi:10.1093/gerona/glz024 · Cited by 37
Statistical Methods for Cohort Studies of CKD: Prediction Modeling
Roy, J. A., Shou, H., Xie, D., Hsu, J. Y., Yang, W. P., Anderson, A. H., Landis, J. R., Jepson, C., He, J., Liu, K. D., et al. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.06210616 · Cited by 37
Predictive accuracy of combined genetic and environmental risk scores
Dudbridge, F., Pashayan, N., & Yang, J. (2017) · Genetic Epidemiology · doi:10.1002/gepi.22092 · Cited by 38
“The NRI has been criticized on theoretical and practical grounds (Kerr et al., 2014; Pepe et al., 2015), yet it has an intuitive appeal that should not be overlooked.”
Pseudo-continuous arterial spin labeling MR images in Warthin tumors and pleomorphic adenomas of the parotid gland: qualitative and quantitative analyses and their correlation with histopathologic and DWI and dynamic contrast enhanced MRI findings
Yamamoto, T., Kimura, H., Hayashi, K., Imamura, Y., & Mori, M. (2018) · Neuroradiology · doi:10.1007/s00234-018-2046-9 · Cited by 36
Coronary Microcirculatory Dysfunction and Acute Cellular Rejection After Heart Transplantation
Lee, J. M., Choi, K. H., Choi, J.-O., Shin, D., Park, Y., Kim, J., Lee, S. H., Kim, D., Yang, J. H., Cho, Y. H., et al. (2021) · Circulation · doi:10.1161/circulationaha.121.056158 · Cited by 36
Prognosis and Reclassification by YKL‐40 in Stable Coronary Artery Disease
Schrøder, J., Jakobsen, J. C., Winkel, P., Hilden, J., Jensen, G. B., Sajadieh, A., Larsson, A. O., Ärnlöv, J., Harutyunyan, M., Johansen, J. S., et al. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.014634 · Cited by 37
“NRI may however be more unreliable and misleading than initially thought with the risk of yielding false-positive NRI statistics.(49,50) The majority of candidate CVD biomarkers lack sufficient reclassification assessment, and even some of the most thoroughly investigated candidate biomarkers have failed to exhibit cl…”
Cystatin C predicts long term mortality better than creatinine in a nationwide study of intensive care patients
Helmersson‐Karlqvist, J., Lipcsey, M., Ärnlöv, J., Bell, M., Ravn, B., Dardashti, A., & Larsson, A. O. (2021) · Scientific Reports · doi:10.1038/s41598-021-85370-8 · Cited by 40
Establishment of a prognostic model for patients with sepsis based on SOFA: a retrospective cohort study
Liu, H., Zhang, L., Xu, F., Li, S., Wang, Z., Han, D., Zhang, F., Lyu, J., & Yin, H. (2021) · Journal of International Medical Research · doi:10.1177/03000605211044892 · Cited by 37
Kidney Disease Biomarkers Improve Heart Failure Risk Prediction in the General Population
Nowak, C., & Ärnlöv, J. (2020) · Circulation Heart Failure · doi:10.1161/circheartfailure.120.006904 · Cited by 34
Polygenic risk scores in coronary artery disease
Rao, A. S., & Knowles, J. W. (2019) · Current Opinion in Cardiology · doi:10.1097/hco.0000000000000629 · Cited by 33
“The widely used NRI statistic, which evaluates the incremental predictive ability of a new marker added to a set of baseline predictors, has been shown to provide inflated estimates of a predictor’s performance [19].”
BackgroundMethod
Uncertainty-aware deep-learning model for prediction of supratentorial hematoma expansion from admission non-contrast head computed tomography scan
Tuan, T. A., Zeevi, T., Haider, S. P., Karam, G. A., Berson, E. R., Tharmaseelan, H., Qureshi, A. I., Sanelli, P. C., Werring, D. J., Malhotra, A., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01007-w · Cited by 32
Reconsidering lactate as a sepsis risk biomarker
Moran, J. L., & Santamaria, J. D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185320 · Cited by 32
Predictive Ability of European Heart Surgery Risk Assessment System II (EuroSCORE II) and the Society of Thoracic Surgeons (STS) Score for in-Hospital and Medium-Term Mortality of Patients Undergoing Coronary Artery Bypass Grafting
Gao, F., Shan, L., Wang, C., Meng, X., Chen, J., Han, L., Zhang, Y., & Li, Z. (2021) · International Journal of General Medicine · doi:10.2147/ijgm.s338819 · Cited by 31
“Consistency of the two evaluation systems in predicting in-hospital and medium-term mortality was tested by the net reclassification index (NRI).(20) Patients were divided into two groups by different standards.”
Method
Inflammation biomarkers and mortality prediction in patients with type 2 diabetes (ZODIAC-27)
Landman, G. W. D., Kleefstra, N., Groenier, K. H., Bakker, S. J. L., Groeneveld, G. H., Bilo, H. J. G., & van Hateren, K. J. J. (2016) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2016.04.015 · Cited by 30
Calibration of medical diagnostic classifier scores to the probability of disease
Chen, W., Sahiner, B., Samuelson, F., Pezeshk, A., & Petrick, N. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280216661371 · Cited by 29
A Study Investigating Markers in PLeural Effusion (SIMPLE): a prospective and double-blind diagnostic study
Han, Y., Yan, L., Li, P., Zhang, L., Ouyang, P.-H., & Hu, Z.-D. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027287 · Cited by 29
Toward Dynamic Risk Prediction of Outcomes After Coronary Artery Bypass Graft: Improving Risk Prediction With Intraoperative Events Using Gradient Boosting
Mori, M., Durant, T. J. S., Huang, C., Mortazavi, B. J., Coppi, A. C., Jean, R. A., Geirsson, A., Schulz, W. L., & Krumholz, H. M. (2021) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007363 · Cited by 28
Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative
McKearnan, S. B., Wolfson, J. A., Vock, D. M., Vazquez‐Benitez, G., & O’Connor, P. J. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx374 · Cited by 26
The power‐integrated discriminant improvement: An accurate measure of the incremental predictive value of additional biomarkers
Hayashi, K., & Eguchi, S. (2019) · Statistics in Medicine · doi:10.1002/sim.8135 · Cited by 27
Carotid plaque detection improves the predictive value of CHA2DS2-VASc score in patients with non-valvular atrial fibrillation: The ARAPACIS Study
Basili, S., Loffredo, L., Pastori, D., Proietti, M., Farcomeni, A., Vestri, A. R., Pignatelli, P., Davı̀, G., Hiatt, W. R., Lip, G. Y. H., et al. (2017) · International Journal of Cardiology · doi:10.1016/j.ijcard.2017.01.001 · Cited by 27
Impact of hyperuricemia on mortality related to aortic diseases: a 3.8-year nationwide community-based cohort study
Otaki, Y., Watanabe, T., Konta, T., Watanabe, M., Asahi, K., Yamagata, K., Fujimoto, S., Tsuruya, K., Narita, I., Kasahara, M., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-71301-6 · Cited by 26
Twelve practical recommendations for developing and applying clinical predictive models
Feng, G., Xu, H., Wan, S., Wang, H., Chen, X., Magari, R. T., Han, Y. K., Wei, Y., & Gu, H. (2024) · The Innovation Medicine · doi:10.59717/j.xinn-med.2024.100105 · Cited by 27
Noninvasive Assessment of Atrial Fibrillation Complexity in Relation to Ablation Characteristics and Outcome
Meo, M., Pambrun, T., Derval, N., Dumas-Pomier, C., Puyo, S., Duchâteau, J., Jaı̈s, P., Hocini, M., Haı̈ssaguerre, M., & Dubois, R. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00929 · Cited by 25
“Finally, the NRI metric itself may not be suitable for the comparison between two classification models, especially if they do not fit the training datasets accurately (Pepe et al., 2015).”
Background
Impact of Annual Change in Geriatric Nutritional Risk Index on Mortality in Patients Undergoing Hemodialysis
Yajima, T., Yajima, K., & Takahashi, H. (2020) · Nutrients · doi:10.3390/nu12113333 · Cited by 24
“However, recently, some statisticians raised a concern regarding the overestimation of the improvement of predictability among predicting models using the NRI [22,23].”
Better prediction of stroke in atrial fibrillation with incorporation of cancer in CHA2DS2VASC score: CCHA2DS2VASC score
Bungo, B., Chaudhury, P., Arustamyan, M., Rikhi, R. R., Hussain, M., Collier, P. H., Kanj, M. H., Khorana, A. A., Mentias, A. G., & Moudgil, R. (2022) · IJC Heart & Vasculature · doi:10.1016/j.ijcha.2022.101072 · Cited by 24
Recalibration Methods for Improved Clinical Utility of Risk Scores
Mishra, A., McClelland, R. L., Inoue, L. Y. T., & Kerr, K. F. (2021) · Medical Decision Making · doi:10.1177/0272989x211044697 · Cited by 23
Prediction of Cardiovascular Events by Pulse Waveform Parameters: Analysis of CARTaGENE
Desbiens, L., Fortier, C., Nadeau‐Fredette, A., Madore, F., Hametner, B., Wassertheurer, S., Agharazii, M., & Goupil, R. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.122.026603 · Cited by 23
“49,52,53 Similar analyses were conducted for traditional clinical risk factors (BMI, eGFR) not included in the ASCVD score, for comparison purposes.”
Net Reclassification Index Statistics Do Not Help Assess New Risk Models
Kerr, K. F. (2022) · Radiology · doi:10.1148/radiol.222343 · Cited by 24
Predictors and nomogram of in-hospital mortality in sepsis-induced myocardial injury: a retrospective cohort study
Xu, K.-Z., Xu, P., Li, J., Zuo, A.-F., Wang, S., & Han, F. (2023) · BMC Anesthesiology · doi:10.1186/s12871-023-02189-8 · Cited by 24
“And a nomogram for predicting 28-day mortality of SIMI was constructed in training cohort according to Occam′s Law, in which fewer variables should be included to achieve the aim [14] Concordance index (C-index) and the area under the receiver operating characteristic curve (AUC) were performed to assess the effective…”
Method
A Systematic Framework for Assessing the Clinical Impact of Polygenic Risk ScoresPreprint
Kulm, S., Marderstein, A. R., Mezey, J. G., & Elemento, O. (2020) · SSRN Electronic Journal · doi:10.1101/2020.04.06.20055574 · Cited by 25
Prospective Validation and Comparative Analysis of Coronary Risk Stratification Strategies Among Emergency Department Patients With Chest Pain
Mark, D. G., Huang, J., Kene, M. V., Sax, D. R., Cotton, D. M., Lin, J. S., Bouvet, S. C., Chettipally, U. K., Anderson, M. L., McLachlan, I. D., et al. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.020082 · Cited by 22
A Survival Prediction Nomogram for Esophageal Squamous Cell Carcinoma Treated with Neoadjuvant Chemoradiotherapy Followed by Surgery
Ding, T.-Y., Liu, C.-T., Huang, B.-L., Chu, L., Wei, L., Lin, Y., Luo, Y., Zhang, B., Hong, C., Xu, Y., et al. (2021) · Cancer Management and Research · doi:10.2147/cmar.s329687 · Cited by 21
Are polygenic risk scores ready for the cancer clinic?—a perspective
Klein, R. J., & Gumus, Z. H. (2022) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-698 · Cited by 21
“For example, when a popular measure, the Net Reclassification Index (NRI) (18) is calculated on a large test dataset, it is likely to be positive even when the addition of the PRS to an existing model has no predictive information.”
High-sensitive troponin T is associated with all-cause and cardiovascular mortality in stable outpatients with type 2 diabetes (ZODIAC-37)
Hendriks, S. H., van Dijk, P. R., van Hateren, K. J. J., van Pelt, L. J., Groenier, K. H., Bilo, H. J. G., Bakker, S. J. L., Landman, G. W. D., & Kleefstra, N. (2015) · American Heart Journal · doi:10.1016/j.ahj.2015.12.015 · Cited by 21
Health risk prediction models incorporating personality data: Motivation, challenges, and illustration.
Chapman, B. P., Lin, F. V., Roy, S., Benedict, R. H. B., & Lyness, J. M. (2019) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000300 · Cited by 25
Quantifying performance of a diagnostic test as the expected information for discrimination: Relation to the C -statistic
McKeigue, P. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218776989 · Cited by 21
“However, these indices are not proper scoring rules(12); this means that adding a biomarker to the predictive model can apparently ‘improve’ such an index even when that biomarker contains no predictive information.(13,14) The authors of a widely quoted set of guidelines on reporting of multivariate models for diagnos…”
Background
Comparison of Questionnaire-Based Breast Cancer Prediction Models in the Nurses' Health Study
Glynn, R. J., Colditz, G. A., Tamimi, R. M., Chen, W. Y., Hankinson, S. E., Willett, W. C., & ROSNER, B. A. (2019) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-18-1039 · Cited by 19
“Even when reclassification is separately considered among cases and noncases, interpretation of these indices is problematic whenmodels exhibit some level of miscalibration (37).”
Background
Prognostic Value of Reverse Remodelling Criteria in Heart Failure with Reduced or Mid-Range Ejection Fraction
Aimo, A., Fabiani, I., Vergaro, G., Arzilli, C., Chubuchny, V., Pasanisi, E., Petersen, C., Poggianti, E., Taddei, C., Pugliese, N. R., et al. (2021) · ESC Heart Failure · doi:10.1002/ehf2.13396 · Cited by 19
External validation of EPIC’s Risk of Unplanned Readmission model, the LACE+ index and SQLape as predictors of unplanned hospital readmissions: A monocentric, retrospective, diagnostic cohort study in Switzerland
Hwang, A. B., Schuepfer, G., Pietrini, M., & Boes, S. (2021) · PLOS ONE · doi:10.1371/journal.pone.0258338 · Cited by 19
Statistical methods for validation of predictive models
Diniz, M. A. (2022) · Journal of Nuclear Cardiology · doi:10.1007/s12350-022-02994-7 · Cited by 22
Development and validation of novel multimorbidity indices for older adults
Steinman, M. A., Jing, B., Shah, S. J., Rizzo, A., Lee, S. J., Covinsky, K. E., Ritchie, C. S., & Boscardin, W. J. (2022) · Journal of the American Geriatrics Society · doi:10.1111/jgs.18052 · Cited by 21
Association and Prognostic Value of Multidomain Frailty Defined by Cumulative Deficit and Phenotype Models in Patients With Heart Failure
Fujimoto, Y., Matsue, Y., Maeda, D., Kagiyama, N., Sunayama, T., Dotare, T., Jujo, K., Saito, K., Kamiya, K., Saito, H., et al. (2023) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2023.11.020 · Cited by 17
A Biomarker‐Enhanced Model for Prediction of Acute Kidney Injury and Cardiovascular Risk Following Angiographic Procedures: CASABLANCA AKI Prediction Substudy
Mohebi, R., van Kimmenade, R. R. J., McCarthy, C. P., Gaggin, H. K., Mehran, R., Dangas, G., & Januzzi, J. L. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.122.025729 · Cited by 16
“28 Finally, while CASABLANCA was originally designed and powered to evaluate for risk of CA-AKI, we did not correct for multiple comparisons in our statistical methods.”
Development and validation of machine learning models for venous thromboembolism risk assessment at admission: a retrospective study
Sheng, W., Wang, X., Xu, W., Hao, Z., Ma, H., & Zhang, S. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1198526 · Cited by 18
First things first: risk model performance metrics should reflect the clinical application
Kerr, K. F., & Janes, H. (2017) · Statistics in Medicine · doi:10.1002/sim.7341 · Cited by 16
Improved Detection of Abnormal Glucose Tolerance in Africans: The Value of Combining Hemoglobin A1c With Glycated Albumin
Hobabagabo, A. F., OSEI-TUTU, N. H., Hormenu, T., SHOUP, E. M., DuBose, C. W., Mabundo, L. S., Ha, J., Sherman, A., Chung, S. T., Sacks, D. B., et al. (2020) · Diabetes Care · doi:10.2337/dc20-1119 · Cited by 15
Renal Disease and Atrial Fibrillation
Stefil, M., Nabrdalik, K., & Lip, G. Y. H. (2021) · Cardiac Electrophysiology Clinics · doi:10.1016/j.ccep.2020.11.001 · Cited by 13
National Heart Foundation of Australia: position statement on coronary artery calcium scoring for the primary prevention of cardiovascular disease in Australia
Jennings, G., Audehm, R. G., Bishop, W., Chow, C. K., Liaw, S., Liew, D., & Linton, S. M. (2021) · The Medical Journal of Australia · doi:10.5694/mja2.51039 · Cited by 13
Calculated versus measured albumin-creatinine ratio to predict kidney failure and death in people with chronic kidney disease
Al‐Wahsh, H., Lam, N. N., Quinn, R. R., Ronksley, P. E., Sood, M. M., Hemmelgarn, B. R., Tangri, N., Ferguson, T. W., Tonelli, M. A., Ravani, P., et al. (2022) · Kidney International · doi:10.1016/j.kint.2022.02.034 · Cited by 13
Multiple imputation was a valid approach to estimate absolute risk from a prediction model based on case–cohort data
Mühlenbruch, K., Kuxhaus, O., di Giuseppe, R., Boeing, H., Weikert, C., & Schulze, M. B. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.12.019 · Cited by 13
Development and validation of the A-TANGO organ failure score for acute-on-chronic liver failure in global cohorts
Engelmann, C., Verma, N., Qi, T., Kerbert, A. J. C., Pohl, J., Morgan, C., Kowalski, M., Andreola, F., Silva, M. O., Hernández, J. L. P., et al. (2026) · Journal of Hepatology · doi:10.1016/j.jhep.2026.02.017 · Cited by 12
Genome-Wide Polygenic Risk Score Predicts Incident Type 2 Diabetes in Women With History of Gestational Diabetes
Choi, J., Lee, H., Kuang, A., Huerta‐Chagoya, A., Scholtens, D. M., Choi, D.-H., Han, M., Lowe, W. L., Manning, A. K., Jang, H. C., et al. (2024) · Diabetes Care · doi:10.2337/dc24-0022 · Cited by 13
Venous thromboembolism and transfusion after major abdominopelvic surgery
McAlpine, K., Breau, R. H., Knee, C., Carrier, M., Violette, P. D., van Walraven, C., McIsaac, D. I., Mallick, R., Cagiannos, I., MORASH, C. G., et al. (2019) · Surgery · doi:10.1016/j.surg.2019.05.050 · Cited by 12
Criteria for evaluating risk prediction of multiple outcomes
Dudbridge, F. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220929039 · Cited by 10
“Among several proposed measures the net reclassification index has attained a default status among practitioners yet has received strong criticism.(51,52) Such issues are likely to be magnified when predicting multiple traits.”
Background
Dynamic Amyloid PET: Relationships to 18F-Flortaucipir Tau PET Measures
Raman, F., Fang, Y.-H. D., Grandhi, S., Murchison, C., Kennedy, R., Morris, J. C., Massoumzadeh, P., Benzinger, T. L. S., Roberson, E. D., & McConathy, J. (2021) · Journal of Nuclear Medicine · doi:10.2967/jnumed.120.254490 · Cited by 12
Serum Proenkephalin A Levels and Mortality After Long-Term Follow-Up in Patients with Type 2 Diabetes Mellitus (ZODIAC-32)
van Hateren, K. J. J., Landman, G. W. D., Arnold, J. F. H., Joosten, H., Groenier, K. H., Navis, G., Sparwaßer, A., Bakker, S. J. L., Bilo, H. J. G., & Kleefstra, N. (2015) · PLoS ONE · doi:10.1371/journal.pone.0133065 · Cited by 12
Risk Factors and Prediction of Stroke in a Population with High Prevalence of Diabetes: The Strong Heart Study
Wang, W., Zhang, Y., Lee, E. T., Howard, B. V., Devereux, R. B., Cole, S. A., Best, L. G., Welty, T. K., Rhoades, E. R., Yeh, J. L., et al. (2017) · World Journal of Cardiovascular Diseases · doi:10.4236/wjcd.2017.75014 · Cited by 12
B-type natriuretic peptide molecular forms for risk stratification and prediction of outcome after acute myocardial infarction
Israr, M. Z., Heaney, L. M., Ng, L. L., & Suzuki, T. (2018) · American Heart Journal · doi:10.1016/j.ahj.2018.02.016 · Cited by 12
Commonly Used Patient-Reported Outcomes Do Not Improve Prediction of COPD Exacerbations
Strassmann, A., Frei, A. L., Haile, S. R., ter Riet, G., & Puhan, M. A. (2017) · CHEST Journal · doi:10.1016/j.chest.2017.09.003 · Cited by 11
Leveraging Serial Low-Dose CT Scans in Radiomics-based Reinforcement Learning to Improve Early Diagnosis of Lung Cancer at Baseline Screening
Wang, Y., Zhou, C., Ying, L., Lee, E., Chan, H., Chughtai, A. R., Hadjiiski, L. M., & Kazerooni, E. A. (2024) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.230196 · Cited by 12
Mortality Risk Prediction Models for People With Kidney Failure
Jarrar, F., Pasternak, M., Harrison, T. G., James, M. T., Quinn, R. R., Lam, N. N., Donald, M., Elliott, M. J., Lorenzetti, D., Strippoli, G. F., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.53190 · Cited by 14
Predictors for major cardiovascular outcomes in stable ischaemic heart disease (PREMAC): statistical analysis plan for data originating from the CLARICOR (clarithromycin for patients with stable coronary heart disease) trial
Winkel, P., Jakobsen, J. C., Hilden, J., Lange, T., Jensen, G. B., Kjøller, E., Sajadieh, A., Kastrup, J., Kolmos, H. J., Larsson, A., et al. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0009-y · Cited by 11
“biomarker, even in independent data [50].”
Comprehensive integrated analysis of MR and DCE-MR radiomics models for prognostic prediction in nasopharyngeal carcinoma
Li, H., Huang, W., Wang, S., Balasubramanian, P. S., Wu, G., Fang, M., Xie, X., Zhang, J., Dong, D., Tian, J., et al. (2023) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-023-00149-0 · Cited by 11
Weighted metrics are required when evaluating the performance of prediction models in nested case–control studies
Rentroia‐Pacheco, B., Bellomo, D., Lakeman, I. M. M., Wakkee, M., Hollestein, L. M., & Klaveren, D. V. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02213-6 · Cited by 10
Evidence-Based Diagnosis: An Introduction to Clinical Epidemiology
Newman, T. B., & Kohn, M. A. (2020) · Cited by 10
European Heart Journal quality standards
Alfonso, F., Torp-Pedersen, C. T., Carter, R. E., & Crea, F. (2021) · European Heart Journal · doi:10.1093/eurheartj/ehab324 · Cited by 10
Validation of clinical risk assessment scores for venous thromboembolism in patients with cancer: a population-based cohort study
Lanting, V. R., Vágó, E., Horváth–Puhó, E., Mulder, F. I., Nisio, M. D., Kamphuisen, P. W., Pedersen, L. H., van Es, N., & Sørensen, H. T. (2024) · Journal of Thrombosis and Haemostasis · doi:10.1016/j.jtha.2024.10.021 · Cited by 9

Authors: Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J.
Affiliations: Fred Hutch Cancer Center (US)
The University of Texas MD Anderson Cancer Center (US)
University of Copenhagen (DK)
Corresponding author: Pepe, Margaret S.
ORCID: Fan, Jing
Feng, Ziding
Gerds, Thomas
Venue: Statistics in Biosciences
Publisher: Springer Science and Business Media LLC
Published: 2014-08-23
Volume: 7
Issue: 2
Pages: 282-295
ISSN: 1867-1764 (ISSN-L)
1867-1772
DOI: 10.1007/s12561-014-9118-0
PMC: PMC4615606
OpenAlex: W2014187955
Cited by: 191
References: 27
Citations by year: 2026: 14 · 2025: 18 · 2024: 18 · 2023: 21 · 2022: 20 · 2021: 22 · 2020: 18 · 2019: 15 · 2018: 16 · 2017: 12 · 2016: 12 · 2015: 3 · 2014: 2
FWCI: 10.21
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: www.springer.com/tdm (tdm)
Locations: Statistics in Biosciences (publishedVersion)
PubMed (publishedVersion)
Europe PMC (PubMed Central) (acceptedVersion)
PubMed Central (acceptedVersion)
Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
Funders: National Institutes of Health
National Cancer Institute (R01CA152089, P30CA015704, U24CA086368)
National Institute of General Medical Sciences (R01GM054438)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: net reclassification improvement, risk prediction model, statistical properties, overfitting, risk function, ROC curve, net benefit, Brier score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks