Interpreting Incremental Value of Markers Added to Risk Prediction Models
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The discrimination of a risk prediction model measures that model's ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Despite this modest effect, the inclusion of the GRS into the predictive model consistently improved the assignment of the risk of progression, as expressed by the net reclassification improvement (NRI) index21.”
From New insights into the genetic etiology of Alzheimer’s disease and related dementias (2022) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Self-citationGeneral Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Self-citationPencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
On the generalized distance in statistics
Mahalanobis, P. C. (1936) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 6,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Research works printed but not linked: 1
- D'Agostino RB, Griffith JL, Schmidt CH, et al. Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association, Biometrics Section; 1997. Measures for evaluating model performance; pp. 253–258. [Google Scholar]
Cited by
Items 301–400 of 517
Hu, H., Kong, W., Yao, N., Qiu, Y., & Yao, R. (2021) · Nursing Open · doi:10.1002/nop2.934 · Cited by 11
“The larger the IDI is, the better the prediction ability of the new model will be indicated (Pencina et al., 2012).”
BackgroundVehviläinen, J., Skrifvars, M. B., Reinikainen, M., Bendel, S., Laitio, R. M., Hoppu, S., Ala-Kokko, T. I., Siironen, J. O., & Raj, R. (2022) · Acta Neurochirurgica · doi:10.1007/s00701-022-05353-0 · Cited by 11
“In addition to the AUC analysis, we calculated the integrated discrimination improvement (IDI) as the AUC might be more insensitive in model comparisons in which the baseline model has performed well [14].”
MethodWong, K. Y., Fan, C., Tanioka, M., Parker, J. S., Nobel, A. B., Zeng, D., Lin, D.-Y., & Perou, C. M. (2019) · Genome biology · doi:10.1186/s13059-019-1640-4 · Cited by 11
Brown, J. R., Stabler, M. E., Parker, D. M., Vricella, L. A., Pasquali, S. K., Leyenaar, J. K., Bohm, A. R., MacKenzie, T. A., Parikh, C. R., Jacobs, M. L., et al. (2019) · Cardiology in the Young · doi:10.1017/s1047951119001471 · Cited by 11
Impact of genetic information on coronary disease risk in Madeira: The GENEMACOR study
Mendonça, M. I., Pereira, A., Monteiro, J. P., Sousa, J., Santos, M., Temtem, M., Borges, S., Henriques, E., Rodrigues, M. F., Sousa, A. C., et al. (2022) · Revista Portuguesa de Cardiologia · doi:10.1016/j.repc.2022.01.009 · Cited by 11
Kurita, D., Sakurai, T., Utsunomiya, D., Kubo, K., Fujii, Y., Kanematsu, K., Ishiyama, K., Oguma, J., & Daiko, H. (2022) · Annals of Surgical Oncology · doi:10.1245/s10434-022-12002-4 · Cited by 10
“Therefore, the NRI and IDI analyses could be used to evaluate betteradded markers newly in risk prediction models.(25,29) In both the EWGSOP2 and AWGS2019 definitions, 5-CST was placed in the same position as HGS to diagnose ‘‘probable sarcopenia’’ and recommended for use in the initial assessment of sarcopenia.”
MethodHartz, S. M., Mozersky, J. T., Schindler, S. E., Linnenbringer, E. L., Wang, J., Gordon, B. A., Raji, C. A., Moulder, K. L., West, T., Benzinger, T. L. S., et al. (2022) · Alzheimer s & Dementia · doi:10.1002/alz.12781 · Cited by 10
“67,68 In addition, the fact that few participants in either sample developed AD dementia within 5 years of the biomarker measurement Regardless of the statistical properties of a risk model, practical implementation requires a user-friendly interface to convey risk information in a meaningful and…”
BackgroundMcGranaghan, P., Düngen, H., Saxena, A., Rubens, M., Salami, J., Radenkovic, J., Bach, D., Apostolovic, S. R., Loncar, G., Zdravkovic, M. D., et al. (2020) · ESC Heart Failure · doi:10.1002/ehf2.12928 · Cited by 12
“The continuous NRI does not depend on the choice of categories, but allocates any change in predicted risk in the correct direction.(39) IDI measures the ability of the new model to increase average sensitivity without reducing average specificity.”
BackgroundZhou, J., Lv, Y., Zhao, F., Yuan, W., Gao, X., Chen, C., Lu, F., Liu, Y., Li, C., Wang, J., et al. (2021) · The Journals of Gerontology Series A · doi:10.1093/gerona/glab339 · Cited by 10
Moons, K. G. M., Altman, D. G., Reitsma, J. B., loannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2022) · Digital Diagnostics · doi:10.17816/dd110794 · Cited by 10
Genetic and clinic predictors of new onset diabetes mellitus after transplantation
Study, T. S. T. C., Saigí-Morgui, N., Quteineh, L., Bochud, P., Crettol, S., Kutalik, Z., Mueller, N. J., Binet, I., van Delden, C., Steiger, J. U., et al. (2017) · The Pharmacogenomics Journal · doi:10.1038/s41397-017-0001-5 · Cited by 10
“As previously described in the literature [25, 26] and in order to compare the genetic and non-genetic model, Likelihood Ratio Tests (LRT) and Integrated Discrimination Improvement (IDI) estimates with their respective p-values were calculated.”
Liu, C., Kuo, C.-Y., Kuo, C.-Y., Wu, I.-C., Chen, P., Hsu, W., Liao, L.-L., Chen, S., TSAO, H., Liu, C., et al. (2025) · npj Aging · doi:10.1038/s41514-025-00198-0 · Cited by 9
CALSTER, B. V., Steyerberg, E. W., D’Agostino, R. B., & Pencina, M. J. (2013) · Medical Decision Making · doi:10.1177/0272989x13513654 · Cited by 12
“Note that the assumption of normality has frequently being used in similar work.(15,18,37) Second, one of the addressed scenarios involved the addition of a binary marker to a prediction model.”
MethodSelf-citationLee, C. J., Ha, J., Kim, J., Kim, I., Ryu, S. K., Rhee, M., Lee, J., Lee, J., Lee, H., Ihm, S., et al. (2020) · Journal of Clinical Hypertension · doi:10.1111/jch.14113 · Cited by 9
Bayesian network models with decision tree analysis for management of childhood malaria in Malawi
Taneja, S. B., Douglas, G. P., Cooper, G. F., Michaels, M. G., Drużdżel, M. J., & Visweswaran, S. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01514-w · Cited by 9
“NRI quantifies how well a new model correctly reclassifies children with and without malaria compared to a baseline model [31].”
MethodInfluentialMise, K., Imamura, M., Yamaguchi, S., Watanabe, M., Higuchi, C., Katayama, A., Miyamoto, S., Uchida, H. A., Nakatsuka, A., Eguchi, J., et al. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.668059 · Cited by 11
“Furthermore, improvement in discriminating the 5-year risk of the study outcome was assessed by analyses of AUROC, category-free NRI, and absolute integrated discrimination improvement (IDI), as reported elsewhere (24, 25).”
MethodMendonça, M. I., Henriques, E., Borges, S., Sousa, A. C., Pereira, A., Santos, M., Temtem, M., Freitas, S., Monteiro, J. P., Sousa, J., et al. (2021) · Genetics and Molecular Biology · doi:10.1590/1678-4685-gmb-2020-0448 · Cited by 9
García‐Osuna, Á., Sans‐Roselló, J., Ferrero‐Gregori, A., Alquézar‐Arbé, A., Sionís, A., & Ordóñez‐Llanos, J. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11051266 · Cited by 11
“[43] suggested interpretations of the continuous NRI: NRI < 20% is weak, NRI 20–40% is intermediate, and NRI > 60% is considered strong.”
BackgroundSiddique, A., Murphy, T. P., Naeem, S. S., Siddiqui, E. U., Pencina, K. M., McEnteggart, G. E., Sellke, F. W., & Dworkin, L. D. (2018) · Catheterization and Cardiovascular Interventions · doi:10.1002/ccd.27890 · Cited by 10
Self-citationToward Personalized Radiation Therapy of Liver Metastasis: Importance of Serial Blood Biomarkers
Ajdari, A., Xie, Y., Richter, C., Niyazi, M., Duda, D. G., Hong, T. S., & Bortfeld, T. R. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00118 · Cited by 9
Dalimunthe, N. N., Alwi, I., Nasution, S. A., & Shatri, H. (2022) · Romanian Journal of Internal Medicine · doi:10.2478/rjim-2022-0012 · Cited by 9
Construction and Analysis of Multiparameter Prognostic Models for Melanoma Outcome
Rothberg, B. E. G., & Rimm, D. L. (2013) · Methods in molecular biology · doi:10.1007/978-1-62703-727-3_13 · Cited by 9
“The incremental value of molecular markers added to clinicopathologic prognostic models is dependent on both the magnitude of the effect size and the inverse of the correlation with the clinicopathologic parameters already in the model [22].”
BackgroundRE: Net Risk Reclassification P Values: Valid or Misleading?
Pencina, M. J., Neely, B., & Steyerberg, E. W. (2014) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dju355 · Cited by 9
“Still, it is essential that those wishing to use continuous NRI follow recently proposed guidelines to avoid spurious claims (6,7).”
BackgroundSelf-citationIs It Time to Employ a Measure of Visit-to-Visit Variability in Clinical Practice?
Whelton, P. K., & Pollock, B. D. (2017) · Hypertension · doi:10.1161/hypertensionaha.117.09455 · Cited by 9
Majcherek, D., Ciesielski, A., & Sobczak, P. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328655 · Cited by 11
“Accuracy, representing the overall correctness of the model, can be misleading in cases of class imbalance, thus necessitating the use of AUC, which evaluates the model’s ability to distinguish between classes across various threshold settings [35].”
Hong, E. P., Heo, S. G., & Park, J. W. (2020) · Metabolites · doi:10.3390/metabo11010006 · Cited by 8
Nan, M. N., García‐Osuna, Á., Móra, J., Trilla, C., Antonijuan-Parés, A., Orantes, V., Cruz-Lemini, M., Blanco‐Vaca, F., & Llurba, E. (2023) · Biomedicines · doi:10.3390/biomedicines11051327 · Cited by 8
Campanile, A., Prota, C., Tedeschi, M., Giano, A., Pianese, B., Cristiano, M., Pompa, A., Sorrentino, R., Vigorito, F., & Ravera, A. (2023) · Journal of Cardiovascular Medicine · doi:10.2459/jcm.0000000000001579 · Cited by 8
Tobias, D. K., Hamaya, R., Clish, C. B., Liang, L., Deik, A., Dennis, C., Bullock, K. H., Zhang, C., Hu, F. B., & Manson, J. E. (2024) · Diabetes/Metabolism Research and Reviews · doi:10.1002/dmrr.3763 · Cited by 10
López-Suárez, A., Bascuñana-Quirell, A., Beltrán-Robles, M., Elvira-González, J., Palacín, F. F., Barroso-Casamitjana, E., & Solino-Ocaña, I. (2014) · Medicine · doi:10.1097/md.0000000000000212 · Cited by 11
“Model performance was assessed with metrics of discrimination, calibration and reclassification.(11,24,25) Discrimination represents the ability to assign higher predicted risk to participants with events than to participants who did not develop events.”
MethodAn, J. N., Ahn, S. V., Lee, J. P., Bae, E., Kang, E., Kim, H., Kim, Y.-J., Oh, Y. K., Kim, Y. H., Kim, Y. H., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0160607 · Cited by 8
“These findings seemed to have relevance to those of our recent study, indicating that post-transplant CV risk factors such as persistent left ventricular hypertrophy and high systolic blood pressure adversely affected allograft survival [24].”
ResultWang, Y., Dai, L., Wang, N., Zhu, Y., Chen, M., & Wang, H. (2019) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2019.06.002 · Cited by 8
Rocha, W. E. M., Oliveira, M. F. R. A., Soares, J. D., L'Armée, V. M. F. S., Martins, M. P. G., Rocha, A. M., Feitosa, A. D. M., Lima, R. D. C., de Oliveira, P. P. M., Silveira-Filho, L. M., et al. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.120.018273 · Cited by 8
“Calcium channel blocker, n (%) 70 (20) 48 (20) 22 (18) 0.”
Yang, M. M. H., Riva-Cambrin, J. K., Cunningham, J., & Casha, S. (2023) · North American Spine Society Journal (NASSJ) · doi:10.1016/j.xnsj.2023.100261 · Cited by 11
Hedbäck, T., Almgren, P., Nilsson, P. M., & Melander, O. (2016) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2016-1736 · Cited by 8
“Regardless of this fact, it has been previously shown that it is problematic to put too much emphasis on the C statistic alone in the evaluation of prediction models and that looking at the continuous net reclassification improvement can be more sensitive in model comparisons in which the traditional model already is…”
BackgroundPredicting Adolescent Substance Use in a Child Welfare Sample: A Multi-Indicator Algorithm
Menon, S. V., Thakur, H., Shorey, R. C., & Cohen, J. R. (2019) · Assessment · doi:10.1177/1073191119880966 · Cited by 8
“The NRI, C statistic, and the Hosmer–Lemeshow goodness of fit statistic were calculated to provide complementary perspectives on our final models as recommended when using reclassification analyses (Pencina et al., 2012).”
MethodComparison of Static and Dynamic Baseline Creatinine Surrogates for Defining Acute Kidney Injury
Warnock, D. G., Neyra, J. A., Macedo, E. M., Miles, A. D., Mehta, R. L., & Wanner, C. (2021) · The Nephron journals/Nephron journals · doi:10.1159/000516953 · Cited by 8
Chen, S., Jiang, J., Su, M., Chen, P., Liu, X., Lei, W., Zhang, S., Wu, Q., Fu, R., Li, X., et al. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08648-4 · Cited by 8
“However, as quantitative analysis methods, NRI and IDI were introduced to further compare the prediction performance between these scoring systems [53].”
MethodMcGeechan, K., Macaskill, P., Irwig, L., & Bossuyt, P. M. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-86 · Cited by 10
“The NRI(>0) has been proposed as a better measure of discrimination than the c-statistic when comparing predictors but how the NRI(>0) then relates to the clinical utility measures has not been examined [5].”
BackgroundResultInfluentialKim, H., Jung, Y., Kim, K. H., & Park, J. M. (2018) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzy225 · Cited by 8
Science and ART—Cardiovascular Disease Risk Assessment in HIV
Feinstein, M. J. (2022) · JAMA Cardiology · doi:10.1001/jamacardio.2022.4880 · Cited by 8
van Giessen, A., Moons, K. G. M., de Wit, G. A., Verschuren, W. M. M., Boer, J. M. A., & Koffijberg, H. (2015) · PLoS ONE · doi:10.1371/journal.pone.0114020 · Cited by 7
“Subsequently, it is essential to evaluate the incremental value of the new model, as compared to the existing model [9].”
MethodResultWaist circumference does not improve established cardiovascular disease risk prediction modeling
Nelms, M. W., Day, A. G., Sui, X., Blair, S. N., & Ross, R. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240214 · Cited by 8
Lin, B., Fu, Z.-Y., & Meng-hua, C. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2022.11.104 · Cited by 7
Krongsut, S., Na-Ek, N., & Khongthon, N. (2025) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2025.108254 · Cited by 7
Aragaki, A. K., Manson, J. E., LeBlanc, E. S., Chlebowski, R. T., Tinker, L. F., Allison, M., Haring, B., Odegaard, A., Wassertheil-Smoller, S. W., Saquib, N., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-00713 · Cited by 8
Generation of co-speech gestures of robot based on morphemic analysis
Chae, Y.-J., Nam, C., Yang, D., Sin, H., Kim, C., & Park, S.-K. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104154 · Cited by 7
Han, C., Wang, L., Liu, C., Qi, W., Zhang, R., Wei, A., Yang, H., Wang, C., Hu, Y., Xu, J., et al. (2023) · Angiology · doi:10.1177/00033197231199228 · Cited by 7
Emerging Translatable Safety Biomarkers
Tarrant, J. M. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-409547-2.12387-x · Cited by 11
Vuilleumier, N., Antiochos, P., Marques-Vidal, P. M., Pagano, S., Virzi, J., Satta, N., Hartley, O., Gaertner, H., Brandt, K. J., Burger, F., et al. (2020) · Clinical & Translational Immunology · doi:10.1002/cti2.1220 · Cited by 7
“Discriminative capacity was assessed by Harrell’s C-statistic.36 Because of the absence of established reclassification cut-offs points for overall mortality and incident CAD, we assessed reclassification according to the category-free integrated discrimination improvement (IDI)36 and the net reclassification improvem…”
MethodPredictors for mortality from respiratory failure in a general population
Kobayashi, M., Shibata, Y., Inoue, S., Igarashi, A., Sato, K., Sato, M., Nemoto, T., Abe, Y., Nunomiya, K., Nishiwaki, M., et al. (2016) · Scientific Reports · doi:10.1038/srep26053 · Cited by 7
Rodríguez‐Girondo, M., Salo, P. P., Burzykowski, T., Perola, M., Houwing‐Duistermaat, J. J., & Mertens, B. J. A. (2018) · The Annals of Applied Statistics · doi:10.1214/17-aoas1125 · Cited by 7
“evaluation of the ability of novel biomolecular markers to improve predictions based on previously established predictive omic sources, often referred as their added or incremental predictive ability [10,11,12]. An illustrative example of these new methodological challenges is given by our motivating problem. We have…”
BackgroundMethodInfluentialPatel, A., Kulkarni, H. R., Kurhe, K., Prakash, A. A., Bhargav, S., Parepalli, S., Fogleman, E., Moore, J., Wallace, D., & Hibberd, P. L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0238315 · Cited by 7
“Incremental values of ANM1 and ANM2 assessments over birth weight as a predictor of preterm birth was determined using integrated discrimination improvement (IDI) and net reclassification index (NRI) [14].”
MethodInfluentialProceedings of 16th International Symposium on IgA Nephropathy
Tesař, V., & Barratt, J. (2021) · Kidney Diseases · doi:10.1159/000519532 · Cited by 7
Crnovrsanin, N., Zeeuw, M. J., Ali, M., Kemna, R., Alipanahi, B., Lumbard, K., Skidmore, Z. L., Rinaldi, L., Erve, I. V. ’., Wesdorp, N. J., et al. (2025) · EBioMedicine · doi:10.1016/j.ebiom.2025.106081 · Cited by 6
“40,55,56 In this light, the observed improvement can be considered meaningful.”
Bi, W., Liu, Y., Wu, W., Lian, S., Pei, Z., Zhu, F., & Meng, Q. (2026) · Journal of Health Population and Nutrition · doi:10.1186/s41043-025-01222-5 · Cited by 6
“This pattern is consistent with the broader prediction literature, in which most biomarkers provide modest refinements rather than large stand-alone improvements [31–34].”
Otomo, S., Hosaka, I., Tanaka, M., Murakami, N., Kokubu, N., Muranaka, A., Nishikawa, R., Hachiro, N., Kawamura, R., Nakata, J., et al. (2025) · Circulation Reports · doi:10.1253/circrep.cr-24-0182 · Cited by 6
“32 , 33 A P value 0.05 was considered the performance with suppressing the overfitting.”
InfluentialKang, X., Liu, X., Li, Y., Yuan, W., Xu, Y., & Yan, H. (2024) · Clinical and Experimental Medicine · doi:10.1007/s10238-024-01296-1 · Cited by 7
“Net Reclassification Index (NRI) was used to assess the model’s predictive performance [20], and Integrated Discrimination Improvement (IDI) [21] was used to compare the model’s overall improvement.”
MethodCordeiro, A. C., de Moraes, A. A. I., Cerutti, V., França, F., Quiroga, B., Amodeo, C., Picotti, J. C., Dutra, L. V., Rodrigues, G. D., Amparo, F. C., et al. (2014) · Journal of the American Society of Hypertension · doi:10.1016/j.jash.2014.02.011 · Cited by 7
Wu, L., Pang, K., Li, B., Le, Y., & Tang, Y. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.886210 · Cited by 6
“The Net Reclassification Index (NRI) and the Integrated Discrimination Improvement Index (IDI) were used to evaluate the risk reclassification ability of HbA1c before the operation (26, 27).”
MethodBaysan, M., Arbous, M. S., Steyerberg, E. W., & van der Bom, J. G. (2022) · Critical Care Explorations · doi:10.1097/cce.0000000000000750 · Cited by 6
“Furthermore, for more insight into the prediction performances, we reported the Net Reclassification Improvement (NRI) of the APACHE IV model with 24-hour lactate concentration compared with the APACHE IV model separately for events (mortality) and nonevents (34, 35).”
MethodPencina, M. J., D’Agostino, R. B., Demler, O., Janssens, A. C. J. W., & Greenland, P. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws206 · Cited by 6
“Finally, we note that the ideas presented in our article (2) should be considered against the backdrop of the current state of the biomarker research, in which the focus is based solely on statistical significance.”
BackgroundMethodSelf-citationEchocardiography in Hypertension in the Era of Risk Stratification and Personalized Medicine
Girerd, N., Huttin, O., & Zannad, F. M. (2016) · American Journal of Hypertension · doi:10.1093/ajh/hpw042 · Cited by 6
External validation of prognostic models for chronic kidney disease among type 2 diabetes
Saputro, S. A., Pattanateepapon, A., Pattanaprateep, O., Aekplakorn, W., McKay, G. J., Attia, J. R., & Thakkinstian, A. (2022) · Journal of Nephrology · doi:10.1007/s40620-021-01220-w · Cited by 6
“01 represents a relevant improvement in model prediction [47, 48].”
BackgroundImproving the analysis and modeling of substance use
Gorelick, D. A., & McPherson, S. (2015) · The American Journal of Drug and Alcohol Abuse · doi:10.3109/00952990.2015.1085264 · Cited by 6
“Calculation of model differences based on AUC, IDI, or NRI is readily done in several available commercial software packages.”
BackgroundInfluentialProstate-specific Antigen Velocity: New Methods, Same Results, Still No Evidence of Clinical Utility
Vickers, A. J., & Pencina, M. J. (2013) · European Urology · doi:10.1016/j.eururo.2013.03.003 · Cited by 6
“[9] Pencina MJ, D’Agostino RB, Pencina KM, Janssens AC, Greenland P.”
BackgroundSelf-citationOgawara, R., Misaka, T., Ohashi, N., Ichimura, S., Tomita, Y., Tani, T., Anzai, F., Sato, Y., Yokokawa, T., Sato, A., et al. (2024) · Clinical Research in Cardiology · doi:10.1007/s00392-024-02539-7 · Cited by 5
“Adding diastolic PTT-BP SD, diastolic PTT-BP VIM, or systolic PTT-BP CV to the base model significantly improved risk reclassification for cardiovascular outcomes (Table 3), suggesting an intermediate improvement in the reclassified prediction model [24].”
Merlo, J., Wagner, P., Juárez, S., Mulinari, S., & Hedblad, B. (2013) · The 20th IEA World Congress of Epidemiology (17-21 August 2014, Anchorage, AK) · Cited by 5
Sheikh, A., Nurmatov, U. B., Al-Katheeri, H. A., & Huneiti, R. A. (2021) · Qatar medical journal · doi:10.5339/qmj.2021.42 · Cited by 5
“The NRI consists of two components, namely, subjects with events and those without events, and then scores subjects on whether their risk was reclassified as higher or lower or not changed using the new prediction model.(20) Outcome selection and optimism bias Some risk prediction models were developed for one cardiov…”
MethodChen, R., Liu, C., Zhou, P., Li, J., Zhou, J., Song, R., Liu, W., Chen, Y., Song, L., Zhao, H., et al. (2022) · Clinical Interventions in Aging · doi:10.2147/cia.s347168 · Cited by 5
“838), which could lead to insensitivity for detecting improvements by adding new biomarkers or predictors.(33,34) Reasons for these improvements could be multiple, but they boiled down to the extra prognostic information provided by D-dimer.”
BackgroundPotier, L., Mohammedi, K., Saulnier, P., Fumeron, F., Halimi, J., Venteclef, N., Marre, M., Hadjadj, S., Roussel, R., & Velho, G. (2022) · Diabetes Care · doi:10.2337/dc21-2638 · Cited by 5
Doney, A. S. F., Nar, A., Huang, Y. S., Trucco, E., MacGillivray, T., Connelly, P., Leese, G., & McKay, G. J. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.945276 · Cited by 6
“Frontiers in Digital Health 03 we determined the Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI) (19).”
MethodInfluentialIncremental Value of Multidomain Risk Factors for Dementia Prediction: A Machine Learning Approach
Tan, W. Y., Hargreaves, C. A., Dawe, G. S., Hsu, W., Lee, M. L., Vipin, A., Kandiah, N., & Hilal, S. (2024) · American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2024.07.016 · Cited by 5
Karaismailoğlu, E., Konar, N. M., Göksülük, D., & Karaağaoğlu, A. E. (2018) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610918.2018.1458135 · Cited by 5
“Pencina et al. (2012) added a continuous and a binary variable to the baseline model separately.”
BackgroundCreatine-kinase and dialysis patients, a helpful tool for stratifying cardiovascular risk?
Quiroga, B., Vega, A., Abad, S., Villaverde, M., Reque, J., & López–Gómez, J. M. (2015) · Nefrología · doi:10.1016/j.nefro.2015.10.004 · Cited by 5
Statistical Considerations in Analysis and Interpretation of Biomarker Studies
Parikh, C. R., & Thiessen‐Philbrook, H. (2010) · Elsevier eBooks · doi:10.1016/b978-0-12-375672-5.10002-7 · Cited by 6
Schousboe, J. T., Langsetmo, L., Taylor, B. C., & Ensrud, K. E. (2017) · Journal of Clinical Densitometry · doi:10.1016/j.jocd.2017.06.012 · Cited by 5
“curve of the base model increases (even assuming the same correlations between all of the covariates in the models) (28).”
BackgroundInfluentialLung Cancer Screening with CT: A Few Steps on a Long Journey
Tammemägi, M. C. (2021) · Radiology · doi:10.1148/radiol.2021210674 · Cited by 5
Ward, A., Kron, B., Lozama, A., Sandhu, A. T., Khandelwal, A., Rodríguez, F., Dash, R., & Nallamshetty, S. (2025) · JACC Advances · doi:10.1016/j.jacadv.2025.102253 · Cited by 4
The prognostic value of the neutrophil-to-lymphocyte ratio in critically ill cirrhotic patients
Giabicani, M., Weiss, E., Chanques, G., Lemaitre, C., JONG, A. D., Grangé, S., Moreau, R., Piton, G., Paugam‐Burtz, C., Jaber, S., et al. (2021) · European Journal of Gastroenterology & Hepatology · doi:10.1097/meg.0000000000002063 · Cited by 4
Matsumori, R., Tanaka, M., Sato, T., Akiyama, Y., Hosaka, I., Nakata, K., Kawaharata, W., Aida, H., Nishizawa, K., Kouzu, H., et al. (2025) · Journal of Atherosclerosis and Thrombosis · doi:10.5551/jat.65939 · Cited by 4
“The addition of sdLDL-C to traditional risk factors (including age, sex, BMI, and current smoking and alcohol drinking habits), medications (including anti-hypertensive drugs, anti-diabetic drugs and anti-dyslipidemic drugs), and the TG level significantly improved the discriminatory capacity for the presence of SLD o…”
InfluentialMaio, M. D., Esposito, L., Silverio, A., Bellino, M., Cancro, F. P., Luca, G. D., Muro, F. M. D., Vassallo, M. G., Vecchione, C., & Galasso, G. (2023) · Catheterization and Cardiovascular Interventions · doi:10.1002/ccd.30842 · Cited by 4
Routinely available inflammation biomarkers to predict stroke and mortality in atrial fibrillation
Wu, L., Yuan, Z., Zeng, Y., Yang, L., Hu, Q., Zhang, H., Li, C., Chen, Y., Zhang, Z., Zhong, L., et al. (2025) · Clinics · doi:10.1016/j.clinsp.2025.100610 · Cited by 4
Association between pentraxin 3 levels and aortic valve calcification
Norimatsu, K., Miura, S., Suematsu, Y., Shiga, Y., Miyase, Y., Nakamura, A., Zhang, B., & Saku, K. (2015) · Journal of Cardiology · doi:10.1016/j.jjcc.2015.08.010 · Cited by 4
“Net reclassification improvement (NRI) [14,15,20] was”
Bruno, G., Barutta, F., Landi, A., Perin, P. C., & Gruden, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114855 · Cited by 4
Chen, C., Tsai, C., Hung, C., Tsai, I., Chiu, Y., Chang, C., Liu, K., Liao, S., Wu, V., & Lin, Y. (2021) · Clinica Chimica Acta · doi:10.1016/j.cca.2021.06.010 · Cited by 4
Huang, X., Zhou, J., Zhang, H., Gao, P., Wang, L., Yang, Y., Jiang, X., Chen, L., & Hua, F. (2022) · Annals of Translational Medicine · doi:10.21037/atm-22-4606 · Cited by 4
“We calculated the C statistics, net reclassification index (NRI), and integrated discrimination improvement (IDI) to evaluate the incremental prognostic values of individual BMI/metabolic status or the combined risk factors beyond conventional risk factors (19).”
MethodShinohara, K., Bryan, T. P., Bartley, C. E., Kelly, M. P., Upasani, V. V., & Newton, P. O. (2024) · Spine · doi:10.1097/brs.0000000000005098 · Cited by 4
Bayesian reclassification statistics for assessing improvements in diagnostic accuracy
Huang, Z., Li, J., Cheng, C., Cheung, C. Y.-L., & Wong, T. Y. (2016) · Statistics in Medicine · doi:10.1002/sim.6899 · Cited by 5
Day, D. E., Shapiro, N. I., & Henning, D. J. (2016) · Shock · doi:10.1097/shk.0000000000000583 · Cited by 4
Clinical Utility in Evaluation of Risk Models
Wacholder, S. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws255 · Cited by 4
“(4) discussed measures for evaluating risk models by purely statistical assessment without consideration of interventions with known efficacy, which is needed for evaluating clinical utility.”
BackgroundInfluentialStatistical Considerations in Analysis and Interpretation of Biomarker Studies
Parikh, C. R., & Philbrook, H. T. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-803014-1.00002-9 · Cited by 3
Models for Predicting Risk of Dementia: Predictive Accuracy and Model Complexity
Stephan, B. C. M. (2014) · International perspectives on aging · doi:10.1007/978-3-319-06650-9_10 · Cited by 4
Common measures of chronic kidney disease improve vascular risk prediction
Stack, A. G. (2017) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(17)30256-5 · Cited by 3
Pang, C. L., Pilkington, N., Wei, Y., Peters, J., Roobottom, C. A., & Hyde, C. (2018) · BMC Cardiovascular Disorders · doi:10.1186/s12872-018-0777-5 · Cited by 3
“Various methods of quantifying incremental prognostic value of an additional test have been described [36].”
Microsimulation model to predict incremental value of biomarkers added to prognostic models
Pencina, K. M., D’Agostino, R. B., Vasan, R. S., & Pencina, M. J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy108 · Cited by 3
Self-citationAhluwalia, M. S., Ali, M. A., Joshi, R. S., Park, E. S., Taha, B. R., McCutcheon, I. E., Chiang, V., Hong, A., Sinclair, G., Bartek, J. J., et al. (2020) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdaa152 · Cited by 3
Kubiak, K. B., Więckowska, B., Jodłowska-Siewert, E., & Guzik, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303276 · Cited by 3
(2014) · PubMed · Cited by 3
Braun, S., Kühn, M., Bergmann, S., Katzke, S., Weise, M., Alakel, N., Stölzel, F., Kroschinsky, F., Ehninger, G., Höffken, G., et al. (2018) · Bone Marrow Transplantation · doi:10.1038/s41409-018-0258-6 · Cited by 3
Metadata
- Authors
- Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P.
- Affiliations
- Boston University (US)
- Corresponding author
- Pencina, Michael J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-08-08
- Volume
- 176
- Issue
- 6
- Pages
- 473-481
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC3530349
- OpenAlex
- W2157026796
- Cited by
- 506
- References
- 37
- Citations by year
- 2026: 36 · 2025: 26 · 2024: 22 · 2023: 30 · 2022: 38 · 2021: 43 · 2020: 45 · 2019: 32 · 2018: 32 · 2017: 30 · 2016: 45 · 2015: 44 · 2014: 50 · 2013: 25 · 2012: 6
- FWCI
- 38.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR000150, n01hc25195-41-0-1)
- National Heart, Lung, and Blood Institute (N01‐HC25195, N01-HC-025195, RC1HL101056)
- National Center for Research Resources (UL1RR025741)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk prediction model, model discrimination, area under the receiver operating characteristic, integrated discrimination improvement, nested model comparison, effect size, Framingham risk score, incident atrial fibrillation
- MeSH terms
- Adult, Aged, Aged, 80 and over, Atrial Fibrillation, Computer Simulation, Data Interpretation, Statistical, Humans, Middle Aged, Prognosis, Risk Factors, ROC Curve, Biomarkers, Logistic Models, Odds Ratio, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.