Interpreting Incremental Value of Markers Added to Risk Prediction Models
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The discrimination of a risk prediction model measures that model's ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Despite this modest effect, the inclusion of the GRS into the predictive model consistently improved the assignment of the risk of progression, as expressed by the net reclassification improvement (NRI) index21.”
From New insights into the genetic etiology of Alzheimer’s disease and related dementias (2022) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Self-citationGeneral Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Self-citationPencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
On the generalized distance in statistics
Mahalanobis, P. C. (1936) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 6,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Research works printed but not linked: 1
- D'Agostino RB, Griffith JL, Schmidt CH, et al. Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association, Biometrics Section; 1997. Measures for evaluating model performance; pp. 253–258. [Google Scholar]
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
New insights into the genetic etiology of Alzheimer’s disease and related dementias
Bellenguez, C., Küçükali, F., Jansen, I. E., Kleineidam, L., Moreno-Grau, S., Amin, N., Naj, A. C., Campos-Martin, R., Grenier-Boley, B., Andrade, V., et al. (2022) · Nature Genetics · doi:10.1038/s41588-022-01024-z · Cited by 2,498
Ross, R., Neeland, I. J., Yamashita, S., Shai, I., Seidell, J., Magni, P., Santos, R. D., Arsenault, B., Cuevas, A., Hu, F. B., et al. (2020) · Nature Reviews Endocrinology · doi:10.1038/s41574-019-0310-7 · Cited by 1,966
Oikonomou, E. K., Marwan, M., Desai, M. Y., Mancio, J., Alashi, A., Hutt Centeno, E., Thomas, S., Herdman, L., Kotanidis, C. P., Thomas, K. E., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31114-0 · Cited by 1,172
Selection Criteria for Lung-Cancer Screening
Tammemägi, M. C., Katki, H. A., Hocking, W. G., Church, T. R., Caporaso, N., Kvale, P. A., Chaturvedi, A. K., Silvestri, G. A., Riley, T. L., Commins, J., et al. (2013) · New England Journal of Medicine · doi:10.1056/nejmoa1211776 · Cited by 1,029
Chan, R. H., Maron, B. J., Olivotto, I., Pencina, M. J., Assenza, G. E., Haas, T., Lesser, J. R., Gruner, C., Crean, A. M., Rakowski, H., et al. (2014) · Circulation · doi:10.1161/circulationaha.113.007094 · Cited by 962
“The 3 model performance metrics we used (ie, area under the receiver-operating characteristics curve, integrated discrimination improvement, and NRI) demonstrated an improvement in the SCD risk model after the addition of LGE, substantiating LGE as a risk marker for SCD in HCM and providing information which exceeded…”
MethodSelf-citationKhan, S. S., Coresh, J., Pencina, M. J., Ndumele, C. E., Rangaswami, J., Chow, S. L., Palaniappan, L. P., Sperling, L. S., Virani, S. S., Ho, J. E., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001191 · Cited by 797
“100,101 105 Specifically, biomarkers of cardiac injury (high-sensitivity troponin, natriuretic peptides [eg, B-type natriuretic peptide]) and diagnostic imaging (CT, echocardiography) were discussed by the group in detail given their importance and clinical relevance for ASCVD and HF prediction and…”
BackgroundSelf-citationLeening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594
Self-citationInterpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503
Shah, A. M., Claggett, B., Sweitzer, N. K., Shah, S. J., Anand, I. S., Liu, L., Pitt, B., Pfeffer, M. A., & Solomon, S. D. (2015) · Circulation · doi:10.1161/circulationaha.115.015884 · Cited by 478
“The continuous NRI quantifies the predictive strength of the novel biomarker beyond, and accounting for correlations with, the existing predictors, with values >60 considered strong, those around 40 considered intermediate, and values <20 considered weak.(45) LS was associated with an NRI of ≈40 for CV death, which co…”
Net Reclassification Indices for Evaluating Risk Prediction Instruments: A Critical Review
Kerr, K. F., Wang, Z., Janes, H., McClelland, R. L., Psaty, B. M., & Pepe, M. S. (2014) · Epidemiology · doi:10.1097/ede.0000000000000018 · Cited by 446
Inflammatory Biomarkers and Exacerbations in Chronic Obstructive Pulmonary Disease
Thomsen, M., Ingebrigtsen, T. S., Marott, J. L., Dahl, M., Lange, P., Vestbo, J., & Nordestgaard, B. G. (2013) · European Respiratory Journal · doi:10.1183/13993003/erj.42.suppl_57.4864 · Cited by 452
Inflammatory Biomarkers and Exacerbations in Chronic Obstructive Pulmonary Disease
Thomsen, M., Ingebrigtsen, T. S., Marott, J. L., Dahl, M., Lange, P., Vestbo, J., & Nordestgaard, B. G. (2013) · JAMA · doi:10.1001/jama.2013.5732 · Cited by 401
Ganz, P., Heidecker, B., Hveem, K., Jonasson, C., Kato, S., Segal, M. R., Sterling, D. G., & Williams, S. A. (2016) · JAMA · doi:10.1001/jama.2016.5951 · Cited by 364
Cardiovascular Disease Risk Assessment: Insights from Framingham
D’Agostino Sr., R. B., Pencina, M. J., Massaro, J. M., & Coady, S. (2013) · Global Heart · doi:10.1016/j.gheart.2013.01.001 · Cited by 329
Self-citationThe MEST score provides earlier risk prediction in lgA nephropathy
Barbour, S. J., Espino-Hernandez, G., Reich, H. N., Coppo, R., Roberts, I. S. D., Feehally, J., Herzenberg, A. M., Cattran, D. C., Bavbek, N., Cook, T., et al. (2016) · Kidney International · doi:10.1038/ki.2015.322 · Cited by 292
Poss, A. M., Maschek, J. A., Cox, J. E., Hauner, B. J., Hopkins, P. N., Hunt, S. C., Holland, W. L., Summers, S. A., & Playdon, M. C. (2020) · Journal of Clinical Investigation · doi:10.1172/jci131838 · Cited by 240
Merlo, J., Wagner, P., Ghith, N., & Leckie, G. (2016) · PLOS ONE · doi:10.1371/journal.pone.0153778 · Cited by 230
“There are alternative measures of discriminatory accuracy like net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) [38–40].”
MethodCarotid Stiffness Is Associated With Incident Stroke
van Sloten, T. T., Sedaghat, S., Laurent, S., London, G. M., Pannier, B., Ikram, M. A., Kavousi, M., Mattace-Raso, F., Franco, O. H., Boutouyrie, P., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.08.888 · Cited by 217
Inflammatory and metabolic biomarkers and risk of liver and biliary tract cancer
Aleksandrova, K., Boeing, H., Nöthlings, U., Jenab, M., Fedirko, V., Kaaks, R., Lukanova, A., Trichopoulou, A., Trichopoulos, D., Boffetta, P., et al. (2014) · Hepatology · doi:10.1002/hep.27016 · Cited by 211
“2 weak reclassification improvement.(34) We repeated the analyses after excluding individuals with self-reported diabetes at baseline and those with positive HBsAg/anti-HCV test, high alcohol consumers, and cases that occurred during the first 2 years of follow-up.”
MethodRonit, A., Kirkegaard-Klitbo, D. M., Dohlmann, T. L., Lundgren, J., Sabin, C. A., Phillips, A. N., Nordestgaard, B. G., & Afzal, S. (2020) · Arteriosclerosis, Thrombosis, and Vascular Biology · doi:10.1161/atvbaha.119.313681 · Cited by 204
Quantifying the added value of new biomarkers: how and how not
Cook, N. R. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0037-2 · Cited by 198
Lu, J., He, J., Li, M., Tang, X., Hu, R., Shi, L., Su, Q., Peng, K., Xu, M., Xu, Y., et al. (2019) · Diabetes Care · doi:10.2337/dc18-1390 · Cited by 174
“ference (C statistic) with or without glycemic measures, net reclassification improvement (NRI), and integrated discrimination improvement (IDI) (20).”
BackgroundInfluentialDey, D., Gaur, S., Ovrehus, K. A., Slomka, P. J., Betancur, J., Goeller, M., Hell, M. M., Gransar, H., Berman, D. S., Achenbach, S., et al. (2018) · European Radiology · doi:10.1007/s00330-017-5223-z · Cited by 173
“discrimination over a set of existing predictors [37].”
BackgroundSinner, M. F., Stepas, K. A., Moser, C. B., Krijthe, B. P., Aspelund, T., Sotoodehnia, N., Fontes, J. D., Janssens, A. C. J. W., Kronmal, R. A., Magnani, J. W., et al. (2014) · EP Europace · doi:10.1093/europace/euu175 · Cited by 173
“5%, much greater than the expected average increase beyond variables already included in the model.(26) The IDI of CRP was small (Table 3).”
BackgroundSelf-citationIki, M., Tamaki, J., Kadowaki, E., Sato, Y., Dongmei, N., Winzenrieth, R., Kagamimori, S., Kagawa, Y., & Yoneshima, H. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2048 · Cited by 169
“(45) A strong independent association of a new predictor with the outcome is required to obtain a significant change in the AUC.”
Lo, S. N., Ma, J., Scolyer, R. A., Haydu, L. E., Stretch, J. R., Saw, R. P. M., Nieweg, O. E., Shannon, K. F., Spillane, A. J., Ch’ng, S., et al. (2020) · Journal of Clinical Oncology · doi:10.1200/jco.19.02362 · Cited by 168
Solomon, D. H., Greenberg, J., Curtis, J. R., Liu, M., Farkouh, M. E., Tsao, P., Kremer, J. M., & Etzel, C. J. (2015) · Arthritis & Rheumatology · doi:10.1002/art.39195 · Cited by 167
“validating, and testing the ERS-RA to allow for comparison with other CV risk scores (29,30,31).”
ResultSample size considerations and predictive performance of multinomial logistic prediction models
de Jong, V. M. T., Eijkemans, M. J. C., CALSTER, B. V., Timmerman, D., Moons, K. G. M., Steyerberg, E. W., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8063 · Cited by 184
Pencina, M. J., D’Agostino, R. B., Zdrojewski, T., Williams, K., Thanassoulis, G., Furberg, C. D., Peterson, E. D., Vasan, R. S., & Sniderman, A. D. (2015) · European Journal of Preventive Cardiology · doi:10.1177/2047487315569411 · Cited by 164
“5% increase that could be expected based on the average contribution of the variables already included in the model.(15) The relative contribution was more modest for the residual on non-HDL-C and Table 3.”
BackgroundSelf-citationHan, K., Song, K., & Choi, B. W. (2016) · Korean Journal of Radiology · doi:10.3348/kjr.2016.17.3.339 · Cited by 162
“They therefore recommend reporting all three values together as measures that characterize the performance of the final model (52).”
BackgroundApolinário, D., Lichtenthäler, D. G., Magaldi, R. M., Soares, A. T., Busse, A. L., Amaral, J. R. D. G., Filho, W. J., & Brucki, S. M. D. (2015) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4282 · Cited by 157
“IDI and NRI are relatively new methods for assessing improvement in model performance that have been considered to be more powerful and less influenced by the strength of the baseline model when compared with previously used measures (Pencina et al., 2012).”
MethodNovel biomarkers of cardiovascular disease: Applications in clinical practice
Lyngbakken, M. N., Myhre, P. L., Røsjø, H. R., & Omland, T. (2018) · Critical Reviews in Clinical Laboratory Sciences · doi:10.1080/10408363.2018.1525335 · Cited by 167
“analyses with strong and established baseline models [20].”
MethodHypertensive Disorders of Pregnancy and 10-Year Cardiovascular Risk Prediction
Stuart, J. J., Tanz, L. J., Cook, N. R., Spiegelman, D., Missmer, S. A., Rimm, E. B., Rexrode, K. M., Mukamal, K. J., & Rich-Edwards, J. W. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.05.077 · Cited by 154
Targeted proteomics improves cardiovascular risk prediction in secondary prevention
Nurmohamed, N. S., Belo Pereira, J. P., Hoogeveen, R. M., Kroon, J., Kraaijenhof, J. M., Waissi, F., Timmerman, N., Bom, M. J., Hoefer, I. E., Knaapen, P., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac055 · Cited by 144
“For use in the validation cohort, all three models were recalibrated to allow an equal comparison and avoid miscalibration.(28) All models were constructed using the same machine learning techniques.”
Kavousi, M., Desai, C. S., Ayers, C., Blumenthal, R. S., Budoff, M. J., Mahabadi, A.-A., Ikram, M. A., van der Lugt, A., Hofman, A., Erbel, R., et al. (2016) · JAMA · doi:10.1001/jama.2016.17020 · Cited by 141
Self-citationNovel genetic markers improve measures of atrial fibrillation risk prediction
Everett, B. M., Cook, N. R., Conen, D., Chasman, D. I., Ridker, P. M., & Albert, C. M. (2013) · European Heart Journal · doi:10.1093/eurheartj/eht033 · Cited by 141
“Also, since there is currently no consensus regarding clinically meaningful AF risk categories, the continuous NRI and IDI may be more appropriate measures of model performance since they are not based upon arbitrary risk categories.(18,27,28) Regardless, the data presented here are not yet strong enough to justify wi…”
Koivistoinen, T., Lyytikäinen, L., Aatola, H., Luukkaala, T. H., Juonala, M., Viikari, J. S. A., Lehtimäki, T. J., Raitakari, O. T., Kähönen, M. A. P., & Hutri‐Kähönen, N. (2018) · Hypertension · doi:10.1161/hypertensionaha.117.10368 · Cited by 158
Ferencik, M., Pencina, K. M., Liu, T., Ghemigian, K., Baltrusaitis, K., Massaro, J. M., D’Agostino, R. B., O’Donnell, C. J., & Hoffmann, U. (2017) · Circulation: Cardiovascular Imaging · doi:10.1161/circimaging.117.006592 · Cited by 130
“To assess the ability of the model to discriminate major CHD events after the addition of CAC distribution measures, we calculated area under the receiver-operating characteristics curve and relative integrated discrimination improvement.(20) Analyses were performed with the use of SAS software, version 9.”
Self-citationMcCoy, T. H., Castro, V. M., Roberson, A. M., Snapper, L. A., & Perlis, R. H. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.2172 · Cited by 130
van Doorn, W. P. T. M., Foreman, Y. D., Schaper, N. C., Savelberg, H. H. C. M., Koster, A., van der Kallen, C. J. H., Wesselius, A., Schram, M. T., Henry, R. M. A., Dagnelie, P. C., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0253125 · Cited by 104
Increased plasma trimethylamine- N -oxide is associated with incident atrial fibrillation
Svingen, G. F. T., Zuo, H., Ueland, P. M., Seifert, R., Løland, K. H., Pedersen, E. R., Schuster, P. M., Karlsson, T., Tell, G. S., Schartum-Hansen, H., et al. (2018) · International Journal of Cardiology · doi:10.1016/j.ijcard.2018.04.128 · Cited by 120
Plasma amyloid‐β and risk of Alzheimer's disease in the Framingham Heart Study
Chouraki, V., Beiser, A., Younkin, L., Preis, S. R., Weinstein, G., Hansson, O., Skoog, I., Lambert, J., Au, R., Launer, L., et al. (2015) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2014.07.001 · Cited by 120
“Those measures and their confidence intervals (CIs) were estimated using bootstrap to account for the risk of overfitting and interpreted by comparing their values with simulated data [29].”
MethodMise, K., Hoshino, J., Ueno, T., Hazue, R., Hasegawa, J., Sekine, A., Sumida, K., Hiramatsu, R., Hasegawa, E., Yamanouchi, M., et al. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.04980515 · Cited by 124
Myocardial Fibrosis as a Predictor of Sudden Death in Patients With Coronary Artery Disease
Zegard, A., Okafor, O., de Bono, J. P., Kalla, M., Lencioni, M., Marshall, H. J., Hudsmith, L., Qiu, T., Steeds, R. P., Stegemann, B., et al. (2021) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.10.046 · Cited by 124
Bos, D., Leening, M. J. G., Kavousi, M., Hofman, A., Franco, O. H., van der Lugt, A., Vernooij, M. W., & Ikram, M. A. (2015) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.115.003843 · Cited by 132
“Because there are no established risk categories for mortality, we determined continuous net reclassification,(18,19) and the relative integrated discrimination improvement (ratio of the absolute difference in discrimination slopes of the 2 models over the discrimination slope of the reference model).(20) In 133 (5.”
BackgroundVan Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120
“23).(33) In the setting with 10 true predictors, six had a coefficient of 0.”
BackgroundKhan, S. S., Post, W. S., Guo, X., Tan, J., Zhu, F., Bos, D., Sedaghati-Khayat, B., van Rooij, J., Aday, A., Allen, N. B., et al. (2023) · JAMA · doi:10.1001/jama.2023.7575 · Cited by 117
Self-citationThe chemistry of the Maillard reaction under physiological conditions: a review.
Njoroge, F. G., & Monnier, V. M. (1989) · PubMed · Cited by 163
Scirica, B. M., Mosenzon, O., Bhatt, D. L., Udell, J., Steg, P. G., McGuire, D. K., Im, K., Kanevsky, E., Stahre, C., Sjöstrand, M., et al. (2018) · JAMA Cardiology · doi:10.1001/jamacardio.2017.4228 · Cited by 116
Comorbidity and survival in out-of-hospital cardiac arrest
Hirlekar, G., Jonsson, M. N., Karlsson, T., Hollenberg, J., Albertsson, P., & Herlitz, J. D. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.10.006 · Cited by 119
Charting a Roadmap for Heart Failure Biomarker Studies
Ahmad, T., Fiuzat, M., Pencina, M. J., Geller, N. L., Zannad, F., Cleland, J. G. F., Snider, J. V., Blankenberg, S., Adams, K. F., Redberg, R. F., et al. (2014) · JACC: Heart Failure · doi:10.1016/j.jchf.2014.02.005 · Cited by 109
“(26) proposed heuristic benchmarks based on the relative IDI that put the increment offered by the new biomarker in the context of the average value contributed by each of the predictors already incorporated into the model.”
BackgroundSelf-citationFufaa, G. D., Weil, E. J., Nelson, R. G., Hanson, R. L., Bonventre, J. V., Sabbisetti, V. S., Waikar, S. S., Mifflin, T. E., Zhang, X., Xie, D., et al. (2014) · Diabetologia · doi:10.1007/s00125-014-3389-3 · Cited by 107
“The 95% CIs for the changes in the c-statistic and the relative IDI were computed based on 10,000 bootstrap samples.”
Chen, L. Y., Leening, M. J. G., Norby, F. L., Roetker, N. S., Hofman, A., Franco, O. H., Pan, W., Polak, J. F., Witteman, J. C. M., Kronmal, R. A., et al. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.002907 · Cited by 102
“In addition, we estimated relative integrated discrimination improvement (IDI), which is the ratio of absolute difference in discrimination slopes of the 2 models over the discrimination slope of the model without the arterial index.(36) To test model calibration, we compared the “goodness of fit” of the observed and…”
MethodScirica, B. M., Bhatt, D. L., Braunwald, E., Raz, I., Cavender, M. A., Im, K., Mosenzon, O., Udell, J. A., Hirshberg, B., Pollack, P. S., et al. (2016) · JAMA Cardiology · doi:10.1001/jamacardio.2016.3030 · Cited by 103
Merlo, J., Mulinari, S., Wemrell, M., Subramanian, S. V., & Hedblad, B. (2017) · SSM - Population Health · doi:10.1016/j.ssmph.2017.08.005 · Cited by 126
Lipoprotein(a) and Cardiovascular Risk Prediction Among Women
Cook, N. R., Mora, S., & Ridker, P. M. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.04.060 · Cited by 98
“Although the continuous NRI suggested improvement in the WHI sample, this measure has been shown to reflect association with the outcome rather than model improvement (32).”
ResultLipoprotein(a) and Family History Predict Cardiovascular Disease Risk
Mehta, A., Virani, S. S., Ayers, C. R., Sun, W., Hoogeveen, R. C., Rohatgi, A., Berry, J. D., Joshi, P. H., Ballantyne, C. M., & Khera, A. (2020) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.06.040 · Cited by 99
Marques-Vidal, P., Schmid, R., Bochud, M., Bastardot, F., von Känel, R., Paccaud, F., Glaus, J., Preisig, M., Waeber, G., & Vollenweider, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051768 · Cited by 95
“It should also be noted that, despite their inclusion on recent guidelines [15,16], it has been recently shown that NRI and IDI interpretation should be made with caution [29,30].”
BackgroundMethodNet Risk Reclassification P Values: Valid or Misleading?
PEPE, M. S., Janes, H., & Li, C. I. (2014) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dju041 · Cited by 100
“The simulation scenarios we considered are similar to those used by others evaluating prediction performance metrics (15).”
MethodNauman, J., Nes, B. M., Lavie, C. J., Jackson, A. S., Sui, X., Coombes, J. S., Blair, S. N., & Wisløff, U. (2016) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2016.10.007 · Cited by 103
Udell, J. A., Morrow, D. A., Jarolím, P., Sloan, S., Hoffman, E. B., O’Donnell, T. F. X., Vora, A. N., Omland, T., Solomon, S. D., Pfeffer, M. A., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2014.03.026 · Cited by 98
Huang, C., Li, S.-X., Caraballo, C., Masoudi, F. A., Rumsfeld, J. S., Spertus, J. A., Normand, S.-L. T., Mortazavi, B. J., & Krumholz, H. M. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007526 · Cited by 83
Net reclassification index at event rate: properties and relationships
Pencina, M. J., Steyerberg, E. W., & D'Agostino, R. B. (2017) · Statistics in Medicine · doi:10.1002/sim.7041 · Cited by 91
“It can be shown that the continuous NRI(>0) equals twice the difference in the probability that the risk based on the expanded information x, y is higher than the risk based on just x among subjects with events, minus the same probability among subjects without events [14]:”
BackgroundSelf-citationUrinary L-FABP predicts poor outcomes in critically ill patients with early acute kidney injury
Parr, S. K., Clark, A. J., Bian, A., Shintani, A., Wickersham, N. E., Ware, L. B., Ikizler, T. A., & Siew, E. D. (2014) · Kidney International · doi:10.1038/ki.2014.301 · Cited by 92
“(47) The total IDI provides the difference in mean predicted probabilities, representing the amount by which the addition of a biomarker to a clinical model increases the separation of the mean predicted probabilities for events and non-events.(23) The statistical software package R version 2.”
BackgroundApplication of Risk Prediction Models to Lung Cancer Screening: A Review
Tammemägi, M. C. (2015) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000142 · Cited by 95
Socioeconomic risk-adjustment with the Area Deprivation Index predicts surgical morbidity and cost
Michaels, A. D., Meneveau, M. O., Hawkins, R. B., Charles, E. J., & Mehaffey, J. H. (2021) · Surgery · doi:10.1016/j.surg.2021.02.016 · Cited by 89
Więckowska, B., Kubiak, K. B., Jóźwiak, P., Moryson, W., & Stawińska-Witoszyńska, B. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191610213 · Cited by 93
“Many misunderstandings stem from failure to use the coefficient in accordance with the author’s assumptions and recommendations [21,23,24,33].”
BackgroundMethodSimple Scoring System to Predict In‐Hospital Mortality After Surgery for Infective Endocarditis
Gatti, G., Perrotti, A., Obadia, J., Duval, X., Iung, B., Alla, F., Chirouze, C., Selton‐Suty, C., Hoen, B., Sinagra, G., et al. (2017) · Journal of the American Heart Association · doi:10.1161/jaha.116.004806 · Cited by 91
Structural Predictors of Loss of Renal Function in American Indians with Type 2 Diabetes
Fufaa, G. D., Weil, E. J., Lemley, K. V., Knowler, W. C., Brosius, F. C., Yee, B., Mauer, M., & Nelson, R. G. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.05760515 · Cited by 87
“The magnitude of improvement in discrimination of the 6-year risk for RFL with the addition of the index was assessed by relative integrated discrimination improvement (32,33).”
MethodA New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study
Sharafoddini, A., Dubin, J. A., Maslove, D. M., & Lee, J. (2019) · JMIR Medical Informatics · doi:10.2196/11605 · Cited by 70
“01) in AUROC meaningful and of clinical interest (because of insensitivity of AUROC) [62,64], including the indicators in our study could improve the average AUROC by 0.”
BackgroundMethodCamargo, E. C., Weinstein, G., Beiser, A., Tan, Z. S., DeCarli, C. S., Kelly‐Hayes, M., Kase, C. S., Murabito, J. M., & Seshadri, S. S. (2016) · Journal of Alzheimer s Disease · doi:10.3233/jad-160229 · Cited by 85
“proportional hazards based models [29] to estimate the value of adding handgrip strength or gait speed to a previously validated dementia screening indicator [30].”
MethodXanthakis, V., Larson, M. G., Wollert, K. C., Aragam, J. R., Cheng, S. C., Ho, J. E., Coglianese, E. E., Levy, D. A., Colucci, W. S., Felker, G. M., et al. (2013) · Journal of the American Heart Association · doi:10.1161/jaha.113.000399 · Cited by 79
“60 are considered strong.(19) We applied the final models to assess the predictive ability of these novel biomarkers in the following clinical subgroups: ≥65 and <65 years; with and without hypertension; and with and without prior MI.”
MethodRothenbacher, D., Rehm, M., Iacoviello, L., Costanzo, S., Tunstall‐Pedoe, H., Belch, J. J. F., Söderberg, S., Hultdin, J., Salomaa, V. V., Jousilahti, P., et al. (2020) · BMC Medicine · doi:10.1186/s12916-020-01776-7 · Cited by 79
“The area under the curve (AUC) and net reclassification index (NRI) for events and non-events are also provided (details in Additional file 1, Tables S4 for population- Age at baseline, years Median (Q1, Q3) 50.0 (41.0, 59.0) 63.0 (54.0, 69.0) Proportion ≥ 65 years 13.1% 42.6% Daily smokers, n (”
MethodInfluentialUrinary endotrophin predicts disease progression in patients with chronic kidney disease
Rasmussen, D. G. K., Fenton, A., Jesky, M., Ferro, C. J., Boor, P., Tepel, M., Karsdal, M. A., Genovese, F., & Cockwell, P. (2017) · Scientific Reports · doi:10.1038/s41598-017-17470-3 · Cited by 74
Filion, K. B., Agarwal, S. K., Ballantyne, C. M., Eberg, M., Hoogeveen, R. C., Huxley, R. R., Loehr, L. R., Nambi, V., Soliman, E. Z., & Alonso, A. (2015) · American Heart Journal · doi:10.1016/j.ahj.2014.10.005 · Cited by 73
“In additional analyses, given the arbitrary nature of such categorization, NRI was examined using a continuous (or category-free) approach (16;18).”
MethodThe potential of novel biomarkers to improve risk prediction of type 2 diabetes
Herder, C., Kowall, B., Tabák, Á. G., & Rathmann, W. (2013) · Diabetologia · doi:10.1007/s00125-013-3061-3 · Cited by 81
“In view of the strengths and drawbacks of AROC, NRI and IDI, Pencina et al suggested using all three measures of discrimination to assess the incremental value of a new marker [14].”
Are the Kihon Checklist and the Kaigo-Yobo Checklist Compatible With the Frailty Index?
Kojima, G., Taniguchi, Y., Kitamura, A., & Shinkai, S. (2018) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2018.05.012 · Cited by 73
“Predictive abilities of KCL and KYCL were separately compared with 32-FI using three measures: (1) changes in the area under the receiver operating characteristic (ROC) curve (AUC) and C statistics, (2) categorical and category-free net reclassification improvement (NRI), and (3) integrated discrimination improvement…”
MethodDíaz‐Zamudio, M., Dey, D., Schuhbaeck, A., Nakazato, R., Gransar, H., Slomka, P. J., Narula, J., Berman, D. S., Achenbach, S. S., Min, J. K., et al. (2015) · Radiology · doi:10.1148/radiol.2015141648 · Cited by 72
“Because established risk categories do not exist for quantitative plaque features, patient reclassification was assessed by using the integrated discrimination improvement (IDI) index (18).”
MethodMultilocus Genetic Risk Scores for Venous Thromboembolism Risk Assessment
Soria, J. M., Morange, P., Vila, J., Souto, J. C., Moyano, M., Trégouët, D., Mateo, J., Saut, N., Salas, E., & Elosúa, R. (2014) · Journal of the American Heart Association · doi:10.1161/jaha.114.001060 · Cited by 72
“We used the methods described by Pencina et al.(39,40) Integrated discrimination improvement (IDI) considers changes in the estimated VTE prediction probabilities as a continuous variable.”
MethodCastelnuovo, A. F. D., Veronesi, G., Costanzo, S., Zeller, T., Schnabel, R. B., Curtis, A. D., Salomaa, V. V., Borchini, R., Ferrario, M., Giampaoli, S., et al. (2019) · Stroke · doi:10.1161/strokeaha.118.023218 · Cited by 69
Tanaka, M., Mori, K., Takahashi, S., Higashiura, Y., Ohnishi, H., Hanawa, N., & Furuhashi, M. (2022) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfac188 · Cited by 71
Wu, H.-C., Yang, H., Wang, Q., Chen, C., & Santella, R. M. (2017) · Carcinogenesis · doi:10.1093/carcin/bgx078 · Cited by 65
Predicting Teen Dating Violence Perpetration
Cohen, J. R., Shorey, R. C., Menon, S. V., & Temple, J. R. (2018) · PEDIATRICS · doi:10.1542/peds.2017-2790 · Cited by 67
Lake, D. E., Fairchild, K. D., & Moorman, J. R. (2013) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-013-9530-x · Cited by 65
“We interpret these results to mean that the HeRO score has a medium effect size as a predictor [8].”
MethodWidera, C., Pencina, M. J., Bobadilla, M., Reimann, I., Guba‐Quint, A., Marquardt, I., Bethmann, K., Korf‐Klingebiel, M., Kempf, T., Lichtinghagen, R., et al. (2013) · Clinical Chemistry · doi:10.1373/clinchem.2013.206185 · Cited by 65
Self-citationEstimating 1-Year Mortality for High-Risk Primary Care Patients Using the “Surprise” Question
Lakin, J. R., Robinson, M. G., Bernacki, R., Powers, B. W., Block, S. D., Cunningham, R. J., & Obermeyer, Z. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.5928 · Cited by 63
Primary Care Visit Regularity and Patient Outcomes: an Observational Study
Rose, A. J., Timbie, J. W., Setodji, C. M., Friedberg, M. W., Malsberger, R., & Kahn, K. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4718-x · Cited by 68
Eggers, K. M., Alshakarchi, J. S. H., Berglund, L., Lindahl, B., Siegbahn, A. G., Wallentin, L. C., & Zethelius, B. (2013) · American Heart Journal · doi:10.1016/j.ahj.2013.07.004 · Cited by 62
Serum NOX2 and urinary isoprostanes predict vascular events in patients with atrial fibrillation
Carnevale, R., Farcomeni, A., Cangemi, R., Nocella, C., Bartimoccia, S., Vicario, T., Saliola, M., Lip, G. Y. H., Pignatelli, P., Pastori, D., et al. (2014) · Thrombosis and Haemostasis · doi:10.1160/th14-07-0571 · Cited by 63
Mortensen, M. B., Nordestgaard, B. G., Afzal, S., & Falk, E. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw426 · Cited by 61
Sharouni, M. E., Stodell, M., Ahmed, T., Suijkerbuijk, K. P. M., Cust, A. E., Witkamp, A. J., Sigurdsson, V., van Diest, P. J., Scolyer, R. A., Thompson, J. F., et al. (2020) · Annals of Oncology · doi:10.1016/j.annonc.2020.11.015 · Cited by 60
Barbour, S. J., Cattran, D., Espino-Hernández, G., Hladunewich, M., & Reich, H. N. (2015) · Kidney International · doi:10.1038/ki.2015.241 · Cited by 60
Key Concepts and Limitations of Statistical Methods for Evaluating Biomarkers of Kidney Disease
Parikh, C. R., & Thiessen‐Philbrook, H. (2014) · Journal of the American Society of Nephrology · doi:10.1681/asn.2013121300 · Cited by 68
Blood biomarkers predictive of epilepsy after an acute stroke event
Abraira, L., Santamarina, E., Cazorla, S., Bustamante, A., Quintana, M., Toledo, M., Fonseca, E., Grau‐López, L., Jiménez, M., Ciurans, J., et al. (2020) · Epilepsia · doi:10.1111/epi.16648 · Cited by 61
Brain Imaging and Cognitive Predictors of Stroke and Alzheimer Disease in the Framingham Heart Study
Weinstein, G., Beiser, A. S., DeCarli, C., Au, R., Wolf, P. A., & Seshadri, S. (2013) · Stroke · doi:10.1161/strokeaha.113.000947 · Cited by 56
Kuhn, A. N., van der Giet, M., Kuhlmann, M. K., Martus, P., Mielke, N., Ebert, N., & Schäeffner, E. (2020) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2020.09.015 · Cited by 58
Alternative Performance Measures for Prediction Models
Wu, Y., & Lee, W. (2014) · PLoS ONE · doi:10.1371/journal.pone.0091249 · Cited by 57
“Recently, a number of ‘relative-performance’ indices that are sensitive to performance change have been proposed [12].”
Schmiegelow, M., Hedlin, H., Stefanick, M. L., Mackey, R. H., Allison, M., Martin, L. W., Robinson, J. G., & Hlatky, M. A. (2015) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.114.001563 · Cited by 62
“Finally, although the IDI is less affected by the strength of the baseline model than the c-index, the IDI is less stable and might overestimate the discriminative ability of the predictor in populations with an event rate <5%.(18) Despite the ongoing debate about the NRI and IDI,(35) there is general consensus in the…”
BackgroundMethodBjornstad, P., Maahs, D. M., Rivard, C. J., Pyle, L., Rewers, M., Johnson, R. J., & Snell‐Bergeon, J. K. (2014) · Acta Diabetologica · doi:10.1007/s00592-014-0611-1 · Cited by 56
“NRI estimates correct changes in clinical classification across risk thresholds [23– 25], and IDI and relative IDI use probability differences instead of categories [23–26].”
Metadata
- Authors
- Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P.
- Affiliations
- Boston University (US)
- Corresponding author
- Pencina, Michael J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-08-08
- Volume
- 176
- Issue
- 6
- Pages
- 473-481
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC3530349
- OpenAlex
- W2157026796
- Cited by
- 506
- References
- 37
- Citations by year
- 2026: 36 · 2025: 26 · 2024: 22 · 2023: 30 · 2022: 38 · 2021: 43 · 2020: 45 · 2019: 32 · 2018: 32 · 2017: 30 · 2016: 45 · 2015: 44 · 2014: 50 · 2013: 25 · 2012: 6
- FWCI
- 38.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR000150, n01hc25195-41-0-1)
- National Heart, Lung, and Blood Institute (N01‐HC25195, N01-HC-025195, RC1HL101056)
- National Center for Research Resources (UL1RR025741)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk prediction model, model discrimination, area under the receiver operating characteristic, integrated discrimination improvement, nested model comparison, effect size, Framingham risk score, incident atrial fibrillation
- MeSH terms
- Adult, Aged, Aged, 80 and over, Atrial Fibrillation, Computer Simulation, Data Interpretation, Statistical, Humans, Middle Aged, Prognosis, Risk Factors, ROC Curve, Biomarkers, Logistic Models, Odds Ratio, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.