Interpreting Incremental Value of Markers Added to Risk Prediction Models
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The discrimination of a risk prediction model measures that model's ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Despite this modest effect, the inclusion of the GRS into the predictive model consistently improved the assignment of the risk of progression, as expressed by the net reclassification improvement (NRI) index21.”
From New insights into the genetic etiology of Alzheimer’s disease and related dementias (2022) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Self-citationGeneral Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Self-citationPencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
On the generalized distance in statistics
Mahalanobis, P. C. (1936) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 6,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Research works printed but not linked: 1
- D'Agostino RB, Griffith JL, Schmidt CH, et al. Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association, Biometrics Section; 1997. Measures for evaluating model performance; pp. 253–258. [Google Scholar]
Cited by
Items 201–300 of 517
Kawaharata, W., Tanaka, M., Sato, T., Akiyama, Y., Hosaka, I., Aida, H., Matsumori, R., Nakata, K., Nishizawa, K., Suzuki, T., et al. (2025) · Journal of the American Heart Association · doi:10.1161/jaha.125.043173 · Cited by 24
Cardiovascular Disease Risk Prediction in Women: Is There a Role for Novel Biomarkers?
Paynter, N. P., Everett, B. M., & Cook, N. R. (2013) · Clinical Chemistry · doi:10.1373/clinchem.2013.202796 · Cited by 28
Martens, F. K., Tonk, E. C. M., Kers, J. G., & Janssens, A. C. J. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.07.002 · Cited by 25
Self-citationMills, J. A., Beach, S. R. H., Dogan, M. V., Simons, R., Gibbons, F. X., Long, J. D., & Philibert, R. A. (2019) · Genes · doi:10.3390/genes10010051 · Cited by 25
“We added the alcohol and smoking methylation markers in Models 8–10 and observed that Model 9, with cg05575921, and Model 10, with cg05575921 and cg04987734, both generally had higher values for the model fit indices (C = 0.774 and 0.779; pseudo R2 = 0.130 and 0.135; IDI = 0.0290 and 0.0391; NRI = 0.237 and 0.264, res…”
MethodInfluentialKwan, A. C., McElhinney, P. A., Tamarappoo, B. K., Cadet, S. J., Hurtado, C., Miller, R. J. H., Han, D., Otaki, Y., Eisenberg, E., Ebinger, J. E., et al. (2020) · European Radiology · doi:10.1007/s00330-020-07142-8 · Cited by 25
Prediction value of pericoronary fat attenuation index for coronary in-stent restenosis
Lu, Z.-F., Yin, W., Schoepf, U. J., Abrol, S., Ma, J.-W., Zhao, L., Su, X., An, Y., Xiao, Z.-C., & Lü, B. (2024) · European Radiology · doi:10.1007/s00330-023-10527-0 · Cited by 24
“Bootstrapping with 200 replications were used to calculate 95% CIs for AUC, IDI, and NRI [20].”
Schutte, E., Gansevoort, R. T., Benner, J., Lutgers, H. L., & Heerspink, H. J. L. (2015) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfv119 · Cited by 24
Marcus, L. P., Marshall, D. C., Hirshman, B. R., McCutcheon, B. A., Gonda, D. D., Koiso, T., Hattangadi-Gluth, J. A., Carter, B. S., Yamamoto, M., & Chen, C. C. (2015) · Neurosurgery · doi:10.1227/neu.0000000000001123 · Cited by 24
“2 have been shown to be indicative of a meaningful difference, similar to the Cohen criteria for effect size.(24,25) Moreover, the prognostic value of CITV in our study is superior to that reported for high-density lipoprotein on 10-year cardiac risk estimation (NRI = 0.”
ResultImproved risk stratification in prevention by use of a panel of selected circulating microRNAs
Keller, T., Boeckel, J., Groß, S., Klotsche, J., Palapies, L., Leistner, D. M., Pieper, L., Stalla, G. K., Lehnert, H., Silber, S., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-04040-w · Cited by 25
Bjornstad, P., Pyle, L., Cherney, D. Z. I., Johnson, R. J., Sippl, R., Wong, R., Rewers, M., & Snell‐Bergeon, J. K. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfx255 · Cited by 24
Zhao, B.-C., Zhuang, P., Lei, S.-H., Qiu, S.-D., Yang, X. F., Li, C., Liu, W.-F., & Liu, K. (2021) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000001495 · Cited by 27
“The risk reclassification ability of preoperative NT-proBNP was assessed by the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) indices [32, 33].”
BackgroundMethodWhat to expect from net reclassification improvement with three categories
Pencina, K. M., Pencina, M. J., & D’Agostino, R. B. (2014) · Statistics in Medicine · doi:10.1002/sim.6286 · Cited by 23
Self-citationBeaudoin, C., Jean, S., Moore, L., Gamache, P., Bessette, L., Ste‐Marie, L., & Brown, J. P. (2018) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.3526 · Cited by 24
“Another limitation inherent to this study is the lack of sensitivity of the c-index for detecting small differences in discriminative ability.((49)) To overcome this limitation and confirm the low incremental value of characteristics of prior fractures, measures of net reclassification improvement were calculated.”
BackgroundMethodSuboptimal specificity of Xpert MTB/RIF among treatment-experienced patients
Metcalfe, J. Z., Makumbirofa, S., Makamure, B., Mutetwa, R., Asteria-Peñaloza, R., Sandy, C., Bara, W., Mungofa, S., Hopewell, P. C., & Mason, P. (2015) · European Respiratory Journal · doi:10.1183/09031936.00214114 · Cited by 23
Recent BRCAPRO Upgrades Significantly Improve Calibration
Mazzola, E., Chipman, J., Cheng, S., & Parmigiani, G. (2014) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-13-1364 · Cited by 23
Saarikoski, L. A., Juonala, M., Huupponen, R., Viikari, J. S. A., Lehtimäki, T. J., Jokinen, E. V., Hutri‐Kähönen, N., Taittonen, L. M., Laitinen, T. P., & Raitakari, O. T. (2016) · Annals of Medicine · doi:10.1080/07853890.2016.1226513 · Cited by 29
“(31) have shown that the improvement in the AUC depends strongly on the baseline model.”
BackgroundAleksandrova, K., Drogan, D., Weikert, C., Schulze, M. B., Fritsche, A., Boeing, H., & Pischon, T. (2019) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2019-00477 · Cited by 23
“2 have been considered strong, intermediate, andweak, respectively (28).”
BackgroundMethodWang, M., Shui, A. M., Barry, F., Verna, E. C., Kent, D. S., Yao, F., Seetharaman, S., Berry, K., Grubbs, R. K., George, G., et al. (2023) · American Journal of Transplantation · doi:10.1016/j.ajt.2023.04.012 · Cited by 23
Douville, N. J., Bastarache, L. A., He, J., Wu, K.-H., Vanderwerff, B. R., Bertucci, E. M., Hornsby, W., Lewis, A. J., Jewell, E. S., Kheterpal, S., et al. (2024) · Anesthesiology · doi:10.1097/aln.0000000000005214 · Cited by 24
“52,53 This allows for a category-free approach to net reclassification index, which removes some of the decisions of cutoff values and categories that can result in misleading values.”
Bjornstad, P., Eckel, R. H., Pyle, L., Rewers, M., Maahs, D. M., & Snell‐Bergeon, J. K. (2015) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2015.07.020 · Cited by 22
“Therefore, we also integrated discrimination index (IDI), which uses probability differences to examine prediction performance [11-14].”
BackgroundMethodPaynter, N. P., LaMonte, M. J., Manson, J. E., Martin, L. W., Phillips, L. S., Ridker, P. M., Robinson, J. G., & Cook, N. R. (2014) · Circulation · doi:10.1161/circulationaha.114.012069 · Cited by 24
“BMI, kg/m2 26 (23, 29) 27 (24, 31) 26 (23, 30) 27 (24, 31)”
Dorajoo, S. R., Ng, J. S. L., Goh, J. H. F., Lim, S. C., Yap, C. W., Chan, A., & Lee, J. Y. C. (2017) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2017.02.007 · Cited by 23
Liu, L., Dai, N., Yin, G., Zhang, W., Mohammed, A., Xu, S., Lv, X., Shi, T., Feng, C., Mohammed, A. A. S., et al. (2022) · Journal of Nuclear Cardiology · doi:10.1007/s12350-022-03038-w · Cited by 22
“To assess whether CMD and abnormal MPI had the incremental predictive value for MACE when added to model 1, net reclassification index (NRI) and integrated discrimination improvement (IDI) were calculated to evaluate the improvement in reclassification and discrimination.(31,32) Two-sided P values \ .”
MethodMise, K., Imamura, M., Yamaguchi, S., Teshigawara, S., Tone, A., Uchida, H. A., Eguchi, J., Nakatsuka, A., Ogawa, D., Yoshida, M., et al. (2018) · Diabetes Care · doi:10.2337/dc18-0030 · Cited by 25
Wu, Q., Li, J., Zhao, L., Liu, D., Wen, J., Wang, Y., Wang, Y., Huang, N., Jiang, L., Liu, Q., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62273-0 · Cited by 23
McKeigue, P. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218776989 · Cited by 23
Tanaka, M., Sato, T., Endo, K., Inyaku, M., Mori, K., Hosaka, I., Mikami, T., Akiyama, Y., Ohnishi, H., Hanawa, N., et al. (2023) · Hypertension Research · doi:10.1038/s41440-023-01392-x · Cited by 22
“Moreover, the increased discriminatory value of each lipid fraction was examined by continuous net reclassi fi cation improvement (NRI) and integrated discrimination improvement (IDI) [21, 22].”
MethodCho, H., Kim, M.-H., Kim, H. J., Park, J. Y., Ryu, D., Lee, H., Lee, J. P., Lim, C.-S., Kim, K. H., Oh, K., et al. (2016) · Peritoneal Dialysis International · doi:10.3747/pdi.2015.00201 · Cited by 22
A multimethod screening approach for pediatric depression onset: An incremental validity study.
Cohen, J. R., Thakur, H., Burkhouse, K. L., & Gibb, B. E. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000364 · Cited by 23
Precision Preventive Medicine—Ready for Prime Time?
Greenland, P., & Hassan, S. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.0142 · Cited by 21
Self-citationUsing decision tree algorithms to screen individuals at risk of entry into sexual recidivism
Lussier, P., Deslauriers‐Varin, N., Collin-Santerre, J., & Bélanger, R. (2019) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2019.05.003 · Cited by 22
Andersson, C., & Vasan, R. S. (2014) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.113.000470 · Cited by 21
DeMazumder, D., Limpitikul, W. B., Dorante, M. I., Dey, S., Mukhopadhyay, B., Zhang, Y., Moorman, J. R., Cheng, A., Berger, R. D., Güallar, E., et al. (2016) · EP Europace · doi:10.1093/europace/euv399 · Cited by 21
van Toorenburg, M., van den Berg, V. J., van der Ploeg, T., Heestermans, A. A. C. M., Dirksen, M. T., Hautvast, R. W. M., Drexhage, O., Boersma, E. H., Kardys, I., & Umans, V. A. W. M. (2018) · International Journal of Cardiology · doi:10.1016/j.ijcard.2018.07.100 · Cited by 23
Wada, H., Suzuki, M., Matsuda, M., Ajiro, Y., Shinozaki, T., Sakagami, S., Yonezawa, K., Shimizu, M., Funada, J., Takenaka, T., et al. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.120.018217 · Cited by 20
Sanders, S., Flaws, D., Than, M. P., Pickering, J. W., Doust, J. A., & Glasziou, P. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.006 · Cited by 20
Direct Estimation of HDL-Mediated Cholesterol Efflux Capacity from Serum
Kuusisto, S. M., Holmes, M. V., Ohukainen, P., Kangas, A. J., Karsikas, M., Tiainen, M., Perola, M., Salomaa, V. V., Kettunen, J., & Ala‐Korpela, M. (2019) · Clinical Chemistry · doi:10.1373/clinchem.2018.299222 · Cited by 22
Statistical inference for decision curve analysis, with applications to cataract diagnosis
Sande, S. Z., Li, J., D’Agostino, R. B., Wong, T. Y., & Cheng, C. (2020) · Statistics in Medicine · doi:10.1002/sim.8588 · Cited by 20
“We show in the Appendix that as n → ∞, √ n (( ?̂?1(c) − ?̂?2(c) ) − (φ1(c) − φ2(c)) ) d →N ( 0, η2(c) ) , (23)”
Self-citationSurvival Regression Modeling Strategies in CVD Prediction
Barkhordari, M., Padyab, M., Sardarinia, M., Hadaegh, F., Azizi, F., & Bozorgmanesh, M. (2016) · Intern J Endocrinol Metab · doi:10.5812/ijem.32156 · Cited by 19
“Such algorithms can be utilized to classify people into different (e.g., high, intermediate, or low risk; treat pharmacologically, introduce lifestyle intervention, or do not act, respectively) medical decision (or risk) categories based on predetermined thresholds (35).”
BackgroundEnoksen, I. T., Svistounov, D., Norvik, J. V., Stefansson, V. T. N., Solbu, M. D., Eriksen, B. O., & Melsom, T. (2021) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfab251 · Cited by 19
Wada, H., Shinozaki, T., Suzuki, M., Sakagami, S., Ajiro, Y., Funada, J., Matsuda, M., Shimizu, M., Takenaka, T., Morita, Y., et al. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.121.023464 · Cited by 20
Pezel, T., Garot, P., Toupin, S., Hovasse, T., Sanguineti, F., Champagne, S., Morisset, S., Chițiboi, T., Jacob, A. J., Sharma, P., et al. (2023) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jead100 · Cited by 19
Lund, J. L., Kuo, T., Brookhart, M. A., Meyer, A., Dalton, A. F., Kistler, C. E., Wheeler, S. B., & Lewis, C. L. (2019) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4769 · Cited by 20
“These measures (NRI and IDI) provide complimentary information to the c‐statistic, indicating the marginal strength of the new model to the existing model and the discrimination slope, respectively.(41,42) Calibration of the Gagne score was also plotted.”
BackgroundSmith, E. E., Fang, J., Alibhai, S. M. H., Cram, P. M., Cheung, A. M., Casaubon, L. K., Kapoor, E., Austin, P. C., & Kapral, M. K. (2019) · JAMA Neurology · doi:10.1001/jamaneurol.2019.1114 · Cited by 20
Omarov, M., Zhang, L., Jorshery, S. D., Malik, R., Das, B., Bellomo, T. R., Mansmann, U., Menten, M. J., Natarajan, P., Dichgans, M., et al. (2024) · medRxiv · doi:10.1101/2024.10.17.24315675 · Cited by 21
Heart Rate Variability-Based Prediction of Autonomic Dysreflexia After Spinal Cord Injury
Calderón‐Juárez, M., Miller, T., Samejima, S., Shackleton, C., Malik, R. N., Sachdeva, R., Dorey, T. W., & Krassioukov, A. V. (2024) · Journal of Neurotrauma · doi:10.1089/neu.2023.0583 · Cited by 18
“31 Several factors may influence human HRV, such as body position, pathological conditions (e.g., inflammation), environmental factors (e.g., noise and social stress), and psychological and genetic contributions, 33 as well as the measurement conditions (controlled scenarios vs. ambulatory…”
Development and Validation of a Risk Score Predicting Risk of Colorectal Cancer
Steffen, A., MacInnis, R. J., Joshy, G., Giles, G. G., Banks, E., & Roder, D. M. (2014) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-14-0206 · Cited by 18
Li, Y., Sheu, W. H., & Lee, I. (2020) · Cardiovascular Diabetology · doi:10.1186/s12933-020-01149-7 · Cited by 18
Juárez, S., Wagner, P., & Merlo, J. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005388 · Cited by 21
Biomarkers for the prediction of venous thromboembolism in the community
Puurunen, M. K., Enserro, D. M., Xanthakis, V., Larson, M. G., Benjamin, E. J., Tofler, G. H., Wollert, K. C., O’Donnell, C. J., & Vasan, R. S. (2016) · Thrombosis Research · doi:10.1016/j.thromres.2016.07.006 · Cited by 18
A Preoperative Multimarker Approach to Evaluate Acute Kidney Injury After Cardiac Surgery
Enger, T. B., Pleym, H., Stenseth, R., Greiff, G., Wahba, A., & Videm, V. (2016) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2016.10.005 · Cited by 18
“2 are considered to be strong, intermediate, and weak, respectively.(32)”
BackgroundChoi, J., Lee, H., Kuang, A., Huerta‐Chagoya, A., Scholtens, D. M., Choi, D.-H., Han, M., Lowe, W. L., Manning, A. K., Jang, H. C., et al. (2024) · Diabetes Care · doi:10.2337/dc24-0022 · Cited by 17
Park, M., Her, S., Kim, C. J., SunCho, J., Park, G., Kim, T., Choi, Y., Park, C., Koh, Y., Park, H., et al. (2015) · Genetics in Medicine · doi:10.1038/gim.2015.171 · Cited by 17
Jonsson, M., Åkesson, A., Hommel, A., GRUBB, A. O., & Bentzer, P. (2021) · Scandinavian Journal of Clinical and Laboratory Investigation · doi:10.1080/00365513.2021.1884892 · Cited by 17
“2 have been suggested to reflect a small improvement [45] we believe that the clinical relevance of this finding is uncertain and should be evaluated in future studies.”
ResultWEDEKIND, L. E., Mahajan, A., Hsueh, W., Chen, P., Olaiya, M. T., Kobes, S., Sinha, M., Baier, L. J., Knowler, W. C., McCarthy, M. I., et al. (2021) · Diabetologia · doi:10.1007/s00125-023-05870-2 · Cited by 18
“However, ΔAUC does not fully capture the contribution of a single variable to individual risk [31].”
van Peet, P. G., Drewes, Y. M., de Craen, A. J. M., Gussekloo, J., & de Ruijter, W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0081400 · Cited by 19
“In addition, we calculated the relative integrated discrimination improvement (rIDI) by dividing the integrated discrimination improvement by the discrimination of the reference model [38].”
MethodBertrais, S., Boursier, J., Ducancelle, A., Oberti, F., Fouchard‐Hubert, I., Moal, V., & Calès, P. (2016) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.13668 · Cited by 20
Urinary glycated uromodulin in diabetic kidney disease
Chang, C., Chen, C., Huang, C., Wu, C., Wu, H.-M., Chiu, P.-F., Kor, C., Chen, T.-H., Chang, G.-D., Kuo, C.-C., et al. (2017) · Clinical Science · doi:10.1042/cs20160978 · Cited by 19
Joshi, R. S., Hirshman, B. R., Ali, M. A., Alattar, A. A., Carroll, K. T., Nagano, O., Aiyama, H., Serizawa, T., Yamamoto, M., & Chen, C. C. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2018.09.209 · Cited by 17
Otegbeye, F., Gale, R. P., & Lazarus, H. M. (2014) · Bone Marrow Transplantation · doi:10.1038/bmt.2014.169 · Cited by 18
Krongsut, S., & Piriyakhuntorn, P. (2024) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2024.107874 · Cited by 16
Edmonston, D. L., WOJDYLA, D. M., Mehta, R. C., Cai, X., Lora, C. M., Cohen, D. L., Townsend, R. R., He, J., Go, A. S., Kusek, J. W., et al. (2019) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2019.05.026 · Cited by 16
Self-citationBertók, T., Bertóková, A., Jáné, E., Híreš, M., Aguedo, J., Potočárová, M., Lukáč, Ľ., Vikartovská, A., Kasák, P., Borsig, L., et al. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.735338 · Cited by 16
Genome-wide polygenic risk scores predict risk of glioma and molecular subtypes
Nakase, T. C., Guerra, G., Ostrom, Q. T., Ge, T., Melin, B. S., Wrensch, M., Wiencke, J. K., Jenkins, R. B., Eckel‐Passow, J. E., (GICC), G. I. C.-C. S., et al. (2024) · Neuro-Oncology · doi:10.1093/neuonc/noae112 · Cited by 15
Castaño-León, A. M., Cicuéndez, M., Navarro-Main, B., Paredes, I., Munarriz, P. M., Hilario, A., Ramos, A., Gomez, P. A., & Lagares, A. (2021) · Journal of neurosurgery · doi:10.3171/2020.11.jns203124 · Cited by 16
Arreola, E. V., Hanson, R. L., Bogardus, C., & Knowler, W. C. (2021) · Diabetes · doi:10.2337/db21-0416 · Cited by 15
Day, D. E., Oedorf, K., Kogan, S., Novack, V., Sánchez, L. D., Wolfe, R. E., Shapiro, N. I., & Henning, D. J. (2015) · Shock · doi:10.1097/shk.0000000000000411 · Cited by 15
“of model variables) for each model to give a sense of the relative importance of the new variable (17).”
BackgroundImprovement in the Prediction of Cerebrovascular Events With White Matter Hyperintensity
de Havenon, A., Smith, E. E., Sharma, R., Falcone, G. J., Bangad, A., Prabhakaran, S., & Sheth, K. N. (2023) · Journal of the American Heart Association · doi:10.1161/jaha.123.029374 · Cited by 17
Pua, Y., Clark, R. A., & Ong, P. (2015) · PLoS ONE · doi:10.1371/journal.pone.0117124 · Cited by 15
“01) – a large effect size[23] with a net gain of 38%(12/32; 95%CI, 5 to 70%) in patients who required walking aids with a large base-of-support (broadbase quadstick or walking frame) and 22%(13/57; 95%CI, −2 to 48%) net gain in patients who required walking aids with a small base-of-support (narrow-base quadstick or w…”
BackgroundDaza, J. F., Bartoszko, J. M., van Klei, W. A., Ladha, K. S., McCluskey, S. A., & Wijeysundera, D. N. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005509 · Cited by 15
Prognostic value of serum α-HBDH levels in patients with lung cancer
Yuan, Z.-M., Wang, L., & Chen, C. (2023) · World Journal of Surgical Oncology · doi:10.1186/s12957-023-02965-3 · Cited by 14
“However, NRI and IDI are also not without problems [35].”
BackgroundGregório, T., Albuquerque, I., Neves, V., Reinas, R. P. V., Pipa, S., Azevedo, L. F., & Castro‐Chaves, P. (2019) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2019.02.014 · Cited by 14
Nakada, N., Miyazaki, T., Ueda, T., Yamagishi, Y., Yamada, K., Kawamura, H., Kakeya, H., Mukae, H., Mikamo, H., Takesue, Y., et al. (2021) · Mycoses · doi:10.1111/myc.13380 · Cited by 14
Skali, H., Carli, M. F. D., Blankstein, R., Chow, B. J. W., Beanlands, R. S., Berman, D. S., Germano, G., Min, J. K., Merhige, M. E., Williams, B. A., et al. (2019) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2019180018 · Cited by 14
Quantifying the value of biomarkers for predicting mortality
Goldman, N., & Glei, D. A. (2015) · Annals of Epidemiology · doi:10.1016/j.annepidem.2015.08.008 · Cited by 14
“point out that, if there is some minimum change in risk considered to be clinically meaningful, it may be preferable to calculate NRI(>x), where x represents that minimal change [1].”
BackgroundMethodInfluentialKawakita, C., Mise, K., Onishi, Y., Sugiyama, H., Yoshida, M., Yamada, M., & Wada, J. (2021) · Scientific Reports · doi:10.1038/s41598-020-77736-1 · Cited by 14
Ke, J., Ruan, X., Liu, W., Liu, X., Wu, K., Qiu, H., Wang, X., Ding, Y., Tan, X., Li, Z., et al. (2024) · iScience · doi:10.1016/j.isci.2024.110233 · Cited by 13
for the fullPIERS Group, Ukah, U. V., Payne, B. A., Hutcheon, J. A., Chappell, L. C., Seed, P. T., Conti-Ramsden, F. I., Ansermino, J. M., Magee, L. A., & von Dadelszen, P. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-03332-w · Cited by 16
Simple screening models for cognitive impairment in community settings: The IRIDE Cohort Study
Abe, T., Kitamura, A., Yamashita, M., Kim, H., Obuchi, S. P., Ishizaki, T., Fujiwara, Y., Awata, S., Toba, K., & IRIDE Cohort Study investigators (2022) · Geriatrics and gerontology international/Geriatrics & gerontology international · doi:10.1111/ggi.14360 · Cited by 13
Coughlan, J. J., Aytekin, A., Lahu, S., Scalamogna, M., Wiebe, J., Pinieck, S., Kufner, S., Xhepa, E., Joner, M., Kuna, C., et al. (2023) · EuroIntervention · doi:10.4244/eij-d-22-00860 · Cited by 13
Tian, X.-X., Luo, J., Liu, F., Qiu, Y., Luo, F., Zeng, L., Zhang, Z., Yang, Y., & Li, X. (2023) · BMC Cardiovascular Disorders · doi:10.1186/s12872-023-03193-z · Cited by 13
Risk stratification in critically ill patients: GDF-15 scores in adult respiratory distress syndrome
Kempf, T., & Wollert, K. C. (2013) · Critical Care · doi:10.1186/cc12765 · Cited by 13
“77, which represents a considerable improvement in discriminatory precision [2].”
Predictive Models in Heart Failure: Who Cares?
Califf, R. M., & Pencina, M. J. (2013) · Circulation Heart Failure · doi:10.1161/circheartfailure.113.000659 · Cited by 13
Self-citationXiao, C.-H., Liu, P., Zhang, H.-H., Yang, F., Chen, X., huang, F., Liu, J., & Tan, X.-Z. (2024) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2024.02.004 · Cited by 12
Tanaka, M., Akiyama, Y., Mori, K., Hosaka, I., Endo, K., Ogawa, T., Sato, T., Suzuki, T., Yano, T., Ohnishi, H., et al. (2025) · Clinical and Experimental Hypertension · doi:10.1080/10641963.2025.2449613 · Cited by 12
“Moreover, the different discriminatory values were examined by continuous net reclassification improvement (NRI) and integrated discrimination improvement (IDI) (52,53).”
Statistically Significant Association Does not Imply Improvement in Prediction of Clinical Outcomes
Jiang, S., ROSNER, B. A., & Colditz, G. A. (2025) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-25-0056 · Cited by 12
Lim, L., Fu, A. W. C., Lau, E. S. H., Ozaki, R., Cheung, K. K.-T., Ronald, C. W., Luk, A. O. Y., Chan, J. C. N., & Kong, A. P. S. (2018) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfy154 · Cited by 12
“6 is considered weak, intermediate and strong, respectively [27].”
BackgroundThiruganasambandamoorthy, V., McRae, A. D., Rowe, B. H., Sivilotti, M. L. A., Mukarram, M. A., Nemnom, M., Booth, R. A., Calder, L. A., Stiell, I. G., Wells, G. A., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m19-3515 · Cited by 17
“We assessed improvement in model performance for addition of NT-proBNP values to the CSRS predictors by calculating the change in the area under the curve (AUC), relative integrated discrimination improvement (IDI), net reclassification improvement (NRI), and net benefit in the form of decision curve analysis, as reco…”
MethodVaes, B., Indestege, P., Serneels, T., Hegendörfer, E., van Peet, P. G., Poortvliet, R. K. E., Wallemacq, P. E., Gussekloo, J., & Degryse, J. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035809 · Cited by 12
Wu, L., Shi, R., Wu, C., Jiang, M., Guo, Q., Zhu, Y., Tang, L., Xu, J., Pu, J., Zhou, Y., et al. (2022) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.28070 · Cited by 12
Giménez, M. R., Pagano, S., Virzi, J., Montecucco, F., Twerenbold, R., Reichlin, T., Wildi, K. S., Grueter, D., Jaeger, C., Haaf, P., et al. (2015) · European Journal of Clinical Investigation · doi:10.1111/eci.12411 · Cited by 12
Weighted Genetic Risk Scores and Prediction of Weight Gain in Solid Organ Transplant Populations
Saigí-Morgui, N., Quteineh, L., Bochud, P., Crettol, S., Kutalik, Z., Wojtowicz, A., Bibert, S., Beckmann, S., Mueller, N. J., Binet, I., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0164443 · Cited by 13
“As previously described, [45, 46] in order to assess the added value of selected SNPs in predicting a 10%WG one year after transplantation (i.”
BackgroundAbouleka, Y., Mohammedi, K., Carpentier, C., Dubois, S. G., Gourdy, P., Gautier, J., Roussel, R., Scheen, A. J., Alhenc‐Gelas, F., Hadjadj, S., et al. (2021) · Diabetes Care · doi:10.2337/dc20-3036 · Cited by 14
“It was significant in NRI and IDI tests but not in the Harrell C statistic, which is less sensitive to changes of small magnitude (34).”
BackgroundThe 82-plex plasma protein signature that predicts increasing inflammation
Tepel, M., Beck, H. C., Tan, Q., Borst, C., & Rasmussen, L. M. (2015) · Scientific Reports · doi:10.1038/srep14882 · Cited by 12
Statistical methods for building better biomarkers of chronic kidney disease
Pencina, M. J., Parikh, C. R., Kimmel, P. L., Cook, N. R., Coresh, J., Feldman, H. I., Foulkes, A. S., Gimotty, P. A., Hsu, C., Lemley, K. V., et al. (2019) · Statistics in Medicine · doi:10.1002/sim.8091 · Cited by 12
Self-citationZhang, S., Chen, H., Xu, H.-W., Yi, Y., Fang, X., & Wang, S. (2022) · Aging Clinical and Experimental Research · doi:10.1007/s40520-022-02218-5 · Cited by 11
“Furthermore, the addition of L3 PMD to the model allows for better categorization and discrimination of SVF risk than incorporating L1 BMD alone (NRI = 53.97%, p = 0.001; IDI = 0.092, p = 0.001).”
MethodInfluentialChen, C., Li, X., Wang, J., Zhou, J., Wei, Y., Luo, Y., Xu, L., Liu, Z., Lv, Y., & Shi, X. (2023) · Innovation in Aging · doi:10.1093/geroni/igad114 · Cited by 11
“…could be more predictive for mortality, we calculated the C -statistics and integrated discrimination improvement (IDI) along with net reclassification improvement (NRI) in fully adjusted model 3 with alternative measures (Kang et al., 2015; Pencina et al., 2012; details in Supplementary Material).”
MethodArm and ankle blood pressure indices, and peripheral artery disease, and mortality: a cohort study
Mohammedi, K., Pigeyre, M., Bosch, J., Yusuf, S., & Gerstein, H. C. (2024) · European Heart Journal · doi:10.1093/eurheartj/ehae087 · Cited by 12
Net reclassification improvement: a link between statistics and clinical practice
Leening, M. J. G., & Cook, N. R. (2013) · European Journal of Epidemiology · doi:10.1007/s10654-012-9759-6 · Cited by 11
New Risk Markers for Cardiovascular Prevention
Backer, G. D. (2014) · Current Atherosclerosis Reports · doi:10.1007/s11883-014-0427-z · Cited by 15
Metadata
- Authors
- Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P.
- Affiliations
- Boston University (US)
- Corresponding author
- Pencina, Michael J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-08-08
- Volume
- 176
- Issue
- 6
- Pages
- 473-481
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC3530349
- OpenAlex
- W2157026796
- Cited by
- 506
- References
- 37
- Citations by year
- 2026: 36 · 2025: 26 · 2024: 22 · 2023: 30 · 2022: 38 · 2021: 43 · 2020: 45 · 2019: 32 · 2018: 32 · 2017: 30 · 2016: 45 · 2015: 44 · 2014: 50 · 2013: 25 · 2012: 6
- FWCI
- 38.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR000150, n01hc25195-41-0-1)
- National Heart, Lung, and Blood Institute (N01‐HC25195, N01-HC-025195, RC1HL101056)
- National Center for Research Resources (UL1RR025741)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk prediction model, model discrimination, area under the receiver operating characteristic, integrated discrimination improvement, nested model comparison, effect size, Framingham risk score, incident atrial fibrillation
- MeSH terms
- Adult, Aged, Aged, 80 and over, Atrial Fibrillation, Computer Simulation, Data Interpretation, Statistical, Humans, Middle Aged, Prognosis, Risk Factors, ROC Curve, Biomarkers, Logistic Models, Odds Ratio, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.