Interpreting Incremental Value of Markers Added to Risk Prediction Models
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The discrimination of a risk prediction model measures that model's ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Despite this modest effect, the inclusion of the GRS into the predictive model consistently improved the assignment of the risk of progression, as expressed by the net reclassification improvement (NRI) index21.”
From New insights into the genetic etiology of Alzheimer’s disease and related dementias (2022) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Self-citationGeneral Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491
Self-citationPencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Self-citationDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
On the generalized distance in statistics
Mahalanobis, P. C. (1936) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 6,068
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Research works printed but not linked: 1
- D'Agostino RB, Griffith JL, Schmidt CH, et al. Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association, Biometrics Section; 1997. Measures for evaluating model performance; pp. 253–258. [Google Scholar]
Cited by
Items 101–200 of 517
Hamdine, O., Eijkemans, M. J. C., Lentjes, E. G. W. M., Torrance, H. L., Macklon, N., Fauser, B. C. J. M., & Broekmans, F. J. (2015) · Fertility and Sterility · doi:10.1016/j.fertnstert.2015.06.030 · Cited by 55
Prediction of Cardiovascular Events by Type I Central Systolic Blood Pressure: A Prospective Study
Lamarche, F., Agharazii, M., Madore, F., & Goupil, R. (2020) · Hypertension · doi:10.1161/hypertensionaha.120.16163 · Cited by 58
the CRIC Study Investigators, Orlandi, P., Fujii, N., Roy, J. A., Chen, H., Hamm, L. L., Sondheimer, J., He, J., Fischer, M. J., Rincon-Choles, H., et al. (2018) · BMC Nephrology · doi:10.1186/s12882-018-0951-0 · Cited by 54
“Both estimates may vary from − 1 (assignment in the wrong direction) to + 1(correct direction) [20].”
BackgroundVultaggio, A., Virgili, G., Gaeta, F., Romano, A. G., Maggi, E., & Matucci, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0121857 · Cited by 52
“The Net Reclassification Improvement was used to calculate the difference in the number of correctly classified patients using a new criterion as an addition to a conventional one [11].”
Sex differences in impact of coronary artery calcification to predict coronary artery disease
Nakao, Y. M., Miyamoto, Y., Higashi, M., Noguchi, T., Ohishi, M., Kubota, I., Tsutsui, H., Kawasaki, T., Furukawa, Y., Yoshimura, M., et al. (2018) · Heart · doi:10.1136/heartjnl-2017-312151 · Cited by 52
Sharouni, M. E., Ahmed, T., Varey, A. H. R., Elias, S. G., Witkamp, A. J., Sigurdsson, V., Suijkerbuijk, K. P. M., van Diest, P. J., Scolyer, R. A., van Gils, C. H., et al. (2021) · Journal of Clinical Oncology · doi:10.1200/jco.20.02446 · Cited by 52
Urinary excretion of epidermal growth factor and rapid loss of kidney function
Norvik, J. V., Harskamp, L. R., Nair, V., Shedden, K., Solbu, M. D., Eriksen, B. O., Kretzler, M., Gansevoort, R. T., Ju, W., & Melsom, T. (2020) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfaa208 · Cited by 53
“Although the use of continuous NRI has been criticized, in the current setting it is expected to be more sensitive than C-statistics and able to capture the enhanced detection of individuals at risk of a specific outcome where natural categories do not exist [28].”
MethodForsblom, C. M., Moran, J. L., Harjutsalo, V., Loughman, T., Wadén, J., Tolonen, N., Thorn, L. M., Saraheimo, M., Gordin, D., Groop, P., et al. (2014) · Diabetes Care · doi:10.2337/dc14-0225 · Cited by 54
Pezel, T., Sanguineti, F., Kinnel, M., Landon, V., Bonnet, G., Garot, P., Hovasse, T., Unterseeh, T., Champagne, S., Louvard, Y., et al. (2020) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.120.010599 · Cited by 50
Stamatelopoulos, K. S., Mueller‐Hennessen, M., Georgiopoulos, G. A., Sachse, M., Boeddinghaus, J., Sopova, K., Gatsiou, A., Amrhein, C., Biener, M., Vafaie, M., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m17-1540 · Cited by 50
Qu, R., Hu, L., Ling, Y., Hou, Y., Fang, H., Zhang, H., Liang, S., He, Z., Fang, M., Li, J., et al. (2020) · BMC Anesthesiology · doi:10.1186/s12871-020-01207-3 · Cited by 50
Scirica, B. M. (2016) · Clinical Chemistry · doi:10.1373/clinchem.2016.255539 · Cited by 55
Urine YKL-40 is associated with progressive acute kidney injury or death in hospitalized patients
Hall, I. E., Stern, E. P., Cantley, L. G., Elias, J. A., & Parikh, C. R. (2014) · BMC Nephrology · doi:10.1186/1471-2369-15-133 · Cited by 46
“We also determined the continuous net reclassification index (cNRI) and integrated discrimination index (IDI) as suggested by Pencina and colleagues [19,20].”
MethodSoluble ST2 predicts outcome and hemorrhagic transformation after acute stroke
Wolcott, Z. C., Batra, A., Bevers, M. B., Sastre, C., Khoury, J. C., Sperling, M. I., Meyer, B. C., Walsh, K. B., Adeoye, O. M., Broderick, J. P., et al. (2017) · Annals of Clinical and Translational Neurology · doi:10.1002/acn3.435 · Cited by 45
“We calculated continuous (category-free) NRI values in groups, since there were no prespecified, externally validated risk categories.(18,19) Kaplan–Meier survival curves were implemented to further investigate the ability of sST2 to predict mortality.”
MethodEsmeijer, K., Schoe, A., Ruhaak, L. R., Hoogeveen, E. K., Soonawala, D., Romijn, F. P. H. T. M., Shirzada, M. R., van Dissel, J. T., Cobbaert, C. M., & de Fijter, J. W. (2021) · Scientific Reports · doi:10.1038/s41598-020-80196-2 · Cited by 46
Clinically Relevant Measures of Fit? A Note of Caution
Cook, N. R. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws208 · Cited by 44
“(4) focus on effect size, it is useful to also consider power.”
BackgroundInfluentialBiomarkers for incident CKD: a new framework for interpreting the literature
Shlipak, M. G., & Day, E. C. (2013) · Nature Reviews Nephrology · doi:10.1038/nrneph.2013.108 · Cited by 43
Díaz‐Zamudio, M., Fuchs, T. A., Slomka, P. J., Otaki, Y., Arsanjani, R., Gransar, H., Germano, G., Berman, D. S., Kaufmann, P. A., & Dey, D. (2016) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jew274 · Cited by 43
VEGF‐C and Mortality in Patients With Suspected or Known Coronary Artery Disease
Wada, H., Suzuki, M., Matsuda, M., Ajiro, Y., Shinozaki, T., Sakagami, S., Yonezawa, K., Shimizu, M., Funada, J., Takenaka, T., et al. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.010355 · Cited by 43
Marston, N., Bonaca, M. P., Jarolím, P., Goodrich, E. L., Bhatt, D. L., Steg, P. G., Cohen, M. C., Storey, R. F., Johanson, P., Wiviott, S. D., et al. (2020) · JAMA Cardiology · doi:10.1001/jamacardio.2020.2981 · Cited by 43
Graversen, P. L., Abildstrom, S. Z., Jespersen, L., Borglykke, A., & Prescott, E. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316638201 · Cited by 43
“In the original SCORE population AUC was 0.81 (95% CI: 0.80–0.82) in the high-risk population and AUC of the low-risk population was 0.74 (95% CI: 0.72–0.76).3 A prediction model with an AUC level of 0.85 is considered a very good prediction model and it requires a very strong predictor of CVD to significantly increas…”
BackgroundInfluentialWada, H., Suzuki, M., Matsuda, M., Ajiro, Y., Shinozaki, T., Sakagami, S., Yonezawa, K., Shimizu, M., Funada, J., Takenaka, T., et al. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.015761 · Cited by 42
Outcome Implication of Aortic Valve Area Normalized to Body Size in Asymptomatic Aortic Stenosis
Tribouilloy, C. M., Bohbot, Y., Maréchaux, S., Debry, N., Delpierre, Q., Peltier, M., Diouf, M., Slama, M., Messika–Zeitoun, D., & Ruşinaru, D. (2016) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.116.005121 · Cited by 42
“The continuous NRI has limitations because its value depends only on the strength of its association with the outcome and not on the strength of the baseline model so that it may be considered a test of association rather than reclassification or model improvement.(28)”
BackgroundBjornstad, P., Wiromrat, P., Johnson, R. J., Sippl, R., Cherney, D. Z. I., Wong, R., Rewers, M., & Snell‐Bergeon, J. K. (2018) · Diabetes Care · doi:10.2337/dc18-1527 · Cited by 41
“The C-statistic has been criticized for insensitivities to changes in clinical decisions yielded for information gained (16,17).”
MethodHjortebjerg, R., Lindberg, S. Ø., Pedersen, S. H., Møgelvang, R., Jensen, J. S., Oxvig, C., Frystyk, J., & Bjerre, M. (2017) · Journal of the American Heart Association · doi:10.1161/jaha.116.005358 · Cited by 42
Prognostic Value of BNP Reduction During Hospitalization in Patients With Acute Heart Failure
Kagiyama, N., Kitai, T., Hayashida, A., Yamaguchi, T., Okumura, T., Kida, K., Mizuno, A., Oishi, S., Inuzuka, Y., Akiyama, E., et al. (2019) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2019.04.004 · Cited by 45
Qazi, S., Massaro, J. M., Chuang, M. L., D’Agostino, R. B., Hoffmann, U., & O’Donnell, C. J. (2017) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.117.006776 · Cited by 45
Self-citationHafezi‐Nejad, N., Zikria, B. A., Eng, J., Carrino, J. A., & Demehri, S. (2015) · Skeletal Radiology · doi:10.1007/s00256-015-2217-2 · Cited by 45
Roubinian, N. H., Murphy, E. L., Swain, B. E., Gardner, M. N., Liu, V. X., & Escobar, G. J. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-213 · Cited by 40
“These two indices have been shown to be useful in quantifying the effect size of an added predictor to a model, as they rely on the strength of a predictor’s association with the outcome and less on the strength of the baseline model [24].”
Bianco, M., D’Ascenzo, F., Raposeiras‐Roubín, S., Kinnaird, T. D., Peyracchia, M., Ariza‐Solé, A., Cerrato, E., Manzano‐Fernández, S., Gravinese, C., Templin, C., et al. (2019) · International Journal of Cardiology · doi:10.1016/j.ijcard.2019.11.132 · Cited by 39
Hu, W., Wu, X., Dong, D., Cui, L., Jiang, M., Zhang, J., Wang, Y., Wang, X., Gao, L., Tian, J., et al. (2020) · International journal of cardiac imaging · doi:10.1007/s10554-020-01896-4 · Cited by 39
“reported before [19], and the variance of the NRI was calculated using a more exact formula proposed by Duprez et al.”
MethodHanon, O., Vidal, J., Lehmann, S., Bombois, S., Allinquant, B., Rose, C., Tréluyer, J., Abdoul, H., Gelé, P., Delmaire, C., et al. (2022) · Alzheimer s & Dementia · doi:10.1002/alz.12613 · Cited by 39
Park, J. Y., Kim, M., Han, S. S., Cho, H., Kim, H., Ryu, D., Kim, H., Lee, H., Lee, J. P., Lim, C.-S., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127240 · Cited by 38
“Furthermore, this metric can capture the degree of improvement in discrimination after accounting for correlation with other variables in the risk prediction model [32].”
BackgroundResultMarinovich, M. L., Houssami, N., Macaskill, P., von Minckwitz, G., Blohmer, J., & Irwig, L. (2014) · International Journal of Cancer · doi:10.1002/ijc.29323 · Cited by 38
“studies have shown that the contribution of additional variables depends on the baseline model, with the magnitude of improvement to the AUC decreasing with increasing size of the baseline AUC.(23,24) Thus, even strong predictors may contribute only small improvements when the baseline model is “good.”
BackgroundGaibazzi, N., Porter, T. R., Agricola, E., Cioffi, G., Mazzone, C., Lorenzoni, V., Albertini, L., Faden, G., Chamsi‐Pasha, M. A., Biabhav, B., et al. (2015) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2014.10.014 · Cited by 40
“MAC >0 318 (24) AVC >0 330 (27) SE result”
ResultModel for End-Stage Liver Disease score predicts mortality in critically ill cirrhotic patients
Boone, M. D., Celi, L. A., Ho, B. G., Pencina, M. J., Curry, M. P., Lior, Y., Talmor, D. S., & Novack, V. (2014) · Journal of Critical Care · doi:10.1016/j.jcrc.2014.05.013 · Cited by 41
“A model is concluded to be improved if the relative IDI was found to be greater than 1/the number of variables in the model [16].”
BackgroundSelf-citationAntiochos, P., Ge, Y., Steel, K. E., Chen, Y.-Y., Bingham, S. E., Abdullah, S. M., Mikolich, J. R., Arai, A. E., Bandettini, W. P., Patel, A. R., et al. (2020) · JAMA Cardiology · doi:10.1001/jamacardio.2020.2834 · Cited by 39
Brain Natriuretic Peptide Usefulness in Very Elderly Dyspnoeic Patients: The BED Study
Plichart, M., Orvoën, G., Jourdain, P., Quinquis, L., Coste, J., Escande, M., Friocourt, P., Paillaud, É., Chedhomme, F.-X., Labouree, F., et al. (2016) · European Journal of Heart Failure · doi:10.1002/ejhf.699 · Cited by 39
Gaibazzi, N., Rigo, F., Facchetti, R., Carerj, S., Giannattasio, C., Moreo, A. M., Mureddu, G. F., Salvetti, M., Grolla, E., Faden, G., et al. (2015) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jev222 · Cited by 37
Miyamori, D., Tanaka, M., Sato, T., Endo, K., Mori, K., Mikami, T., Hosaka, I., Hanawa, N., Ohnishi, H., & Furuhashi, M. (2023) · Journal of the American Heart Association · doi:10.1161/jaha.123.030269 · Cited by 39
Gurunathan, U., Rapchuk, I. L., Dickfos, M. J., Larsen, P. J., Forbes, A. B., Martin, C., Leslie, K., & Myles, P. S. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.16345 · Cited by 36
Bjornstad, P., Maahs, D. M., Wadwa, R. P., Pyle, L. L., Rewers, M., Eckel, R. H., & Snell‐Bergeon, J. K. (2014) · Journal of clinical lipidology · doi:10.1016/j.jacl.2014.08.008 · Cited by 38
“The C-statistic has been criticized for insensitivities to changes in clinical decisions yielded for information gained.(15,16) Therefore, we used integrated discrimination index, category-free, and categorical NRI.”
MethodWang, Y., Chen, Y., Chen, R., Xu, Y., Zheng, H., Xu, J., Xia, J., Cai, Y., Xu, H., & Wang, X. (2023) · BMC Geriatrics · doi:10.1186/s12877-023-04310-5 · Cited by 36
“…be the optimal measure for evaluating models aimed at predicting future risk or stratifying individuals into risk categories [30] and not sensitive in comparing models with a well-performed baseline model [31], we used NRI and IDI as complement measures to test the discrimination ability [32, 33].”
Sabatino, J., Rosa, S. D., Leo, I., Strangio, A., Spaccarotella, C., Polimeni, A., Sorrentino, S., Salvo, G. D., & Indolfi, C. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.719603 · Cited by 35
Pan, L., Zou, H., Meng, X., Li, D., Li, W., Chen, X., Yang, Y., & Yu, X. (2023) · Journal of Diabetes Investigation · doi:10.1111/jdi.14024 · Cited by 35
Veronesi, G., Gianfagna, F., Giampaoli, S., Chambless, L. E., Mancia, G., Cesana, G., & Ferrario, M. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.04.007 · Cited by 38
Kerr, K. F., Bansal, A., & Pepe, M. S. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws210 · Cited by 35
“present an interesting study of the behavior of measures of incremental value (1).”
BackgroundLi, Q., Reeves, M. J., Owen, J., & Keith, L. G. (2014) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2014.07.003 · Cited by 34
Payne, B. A., Hutcheon, J. A., Dunsmuir, D. T., Cloete, G., Dumont, G. A., Hall, D. R., Lim, J. M.-Y., Magee, L. A., Sikandar, R., Qureshi, R. N., et al. (2015) · Journal of Obstetrics and Gynaecology Canada · doi:10.1016/s1701-2163(15)30358-3 · Cited by 37
Inclusion of Smoking Data in Cardiovascular Disease Risk Estimation
Duncan, M. S., Greevy, R. A., Tindle, H. A., Vasan, R. S., Lipworth, L. P., Aldrich, M. C., Lloyd‐Jones, D. M., & Freiberg, M. S. (2021) · JAMA Cardiology · doi:10.1001/jamacardio.2021.4990 · Cited by 36
Polygenic risk prediction models for colorectal cancer: a systematic review
Sassano, M., Mariani, M., Quaranta, G., Pastorino, R., & Boccia, S. (2022) · BMC Cancer · doi:10.1186/s12885-021-09143-2 · Cited by 38
LACY, M. E., Wellenius, G. A., Carnethon, M. R., Loucks, E. B., Carson, A. P., Luo, X., Kiefe, C. I., Gjelsvik, A., Gunderson, E. P., Eaton, C. B., et al. (2015) · Diabetes Care · doi:10.2337/dc15-0509 · Cited by 33
Halon, D. A., Azencot, M., Rubinshtein, R., Zafrir, B., Flugelman, M. Y., & Lewis, B. S. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.116.003226 · Cited by 33
Corrales‐Medina, V., Taljaard, M., Fine, M. J., Dwivedi, G., Perry, J. J., Musher, D. M., & Chirinos, J. A. (2014) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2013.09.015 · Cited by 33
Huang, Z., Klarić, L., Krasauskaite, J., Khalid, W., Strachan, M. W. J., Wilson, J. F., & Price, J. F. (2023) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwad160 · Cited by 34
Reconsidering lactate as a sepsis risk biomarker
Moran, J. L., & Santamaria, J. D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185320 · Cited by 34
Cell-free DNA as a biomarker in stroke: Current status, problems and perspectives
Glebova, K. V., Вейко, Н. Н., Nikonov, A. A., Porokhovnik, L. N., & Kostuyk, S. V. (2018) · Critical Reviews in Clinical Laboratory Sciences · doi:10.1080/10408363.2017.1420032 · Cited by 32
Ng, A. C. T., Bertini, M., Ewe, S. H., van der Velde, E. T., Leung, D. Y., Delgado, V., & Bax, J. J. (2019) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2019.06.011 · Cited by 32
White Matter Microstructure Improves Stroke Risk Prediction in the General Population
Evans, T. E., O'Sullivan, M. J., de Groot, M., Niessen, W. J., Albert Hofman, D., Krestin, G. P., van der Lugt, A., Portegies, M. L. P., Koudstaal, P. J., Bos, D., et al. (2016) · Stroke · doi:10.1161/strokeaha.116.014651 · Cited by 37
“Given that there are no established risk categories for stroke, we calculated continuous net reclassification (cNRI), event and nonevent net reclassification, and the relative integrated discrimination improvement (ie, the ratio of the absolute difference in discrimination slopes of the 2 models over the discriminatio…”
MethodClinical risk reclassification at 10 years
Cook, N. R., Demler, O., & Paynter, N. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7340 · Cited by 32
“In addition, some work has focused on interpreting levels of these metrics and deciding what values may be high enough to be clinically important [8,13].”
BackgroundZorzi, A., Brunetti, G., Cardaioli, F., D’Arcangelo, F., Fabris, T., Gambato, M., Iliceto, S., Martini, A., Mattesi, G., Peluso, C., et al. (2021) · International Journal of Cardiology · doi:10.1016/j.ijcard.2021.06.046 · Cited by 31
Serum prealbumin and its changes over time are associated with mortality in acute kidney injury
Wang, W., Pan, Y., Tang, X., Hao, G., Xie, Y., Ma, S., Luo, J., Guo, D., & Ding, F. H. (2017) · Scientific Reports · doi:10.1038/srep41493 · Cited by 31
Baysan, M., Baroni, G. D., van Boekel, A. M., Steyerberg, E. W., Arbous, M. S., & van der Bom, J. G. (2020) · Critical Care Explorations · doi:10.1097/cce.0000000000000087 · Cited by 35
“In line with recent recommendations (23), we reported the NRI for events (in-hospital mortality) and the NRI for non-events separately, thereby quantifying the effect of an added predictor as it depends more on the effect of the predictor rather than on the strength of the baseline APACHE IV model (24).”
BackgroundLi, L., Hu, X., Tao, X., Shi, X., Zhou, W., Hu, H., & Hu, X. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109769 · Cited by 30
van der Weijden, J., Mahesh, S. V. K., van Londen, M., Bakker, S. J. L., Sanders, J., Navis, G. J., Pol, R. A., Roodnat, J. I., Kho, M. M. L., Yakar, D., et al. (2022) · Kidney International · doi:10.1016/j.kint.2022.01.034 · Cited by 30
Multivariable Quantitative US Parameters for Assessing Hepatic Steatosis
Kuroda, H., Oguri, T., Kamiyama, N., Toyoda, H., Yasuda, S., Imajo, K., Suzuki, Y., Sugimoto, K., Akita, T., Tanaka, J., et al. (2023) · Radiology · doi:10.1148/radiol.230341 · Cited by 30
Using Self‐Reported Health Measures to Predict High‐Need Cases among Medicaid‐Eligible Adults
Wherry, L., Burns, M. E., & Leininger, L. J. (2014) · Health Services Research · doi:10.1111/1475-6773.12222 · Cited by 32
“Equivalent to the improvement in discrimination slopes between a new and existing model, the IDI is a more sensitive measure than the c-statistic and offers a direct interpretation—an increase in the difference in mean predicted probabilities of events and nonevents (Pencina et al. 2012).”
Carotid Stiffness: A Novel Cerebrovascular Disease Risk Factor
van Sloten, T. T., & Stehouwer, C. D. A. (2016) · Pulse · doi:10.1159/000445354 · Cited by 31
Hirshman, B. R., Wilson, B. R., Ali, M. A., Schüpper, A. J., Proudfoot, J., Goetsch, S. J., Carter, B. S., Sinclair, G., Bartek, J. J., Chiang, V., et al. (2017) · Neurosurgery · doi:10.1093/neuros/nyx380 · Cited by 30
Czyz, E. K., Yap, J. R. T., King, C. A., & Nahum-Shani, I. B. (2020) · Assessment · doi:10.1177/1073191120939168 · Cited by 31
“For models including the mean of daily ratings as the only feature in the model, risk factors were selected for further investigation if the odds ratio (OR) per 1 standard deviation of mean daily ratings was at least medium in magnitude (i.e., OR ≥ 1.65 or OR ≤ 0.61; see Pencina et al., 2012).”
BackgroundMethodInfluentialBeyond Self‐Reports: Changes in Biomarkers as Predictors of Mortality
Glei, D. A., Goldman, N., Rodríguez, G., & Weinstein, M. (2014) · Population and Development Review · doi:10.1111/j.1728-4457.2014.00676.x · Cited by 29
Steyerberg, E. W., Vedder, M. M., Leening, M. J. G., Postmus, D., D’Agostino, R. B., CALSTER, B. V., & Pencina, M. J. (2014) · Biometrical Journal · doi:10.1002/bimj.201300260 · Cited by 35
“The increase in AUC depends on the AUC of the baseline model, e.g. Cohen’s D of 0.5 corresponds to +0.05 for a baseline model with AUC of 0.65, whereas only to +0.02 for a baseline model with AUC of 0.80 (Pencina et al., 2012a).”
BackgroundMethodInfluentialSelf-citationCamacho, C., Straatmann, V. S., Day, J., & Taylor-Robinson, D. C. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024851 · Cited by 30
“The IDI using the complete case sample from model 1 was calculated to assess difference in performance between models as the percentage change in individuals being correctly assigned by the model.(40) The AUROC and its 95% CI was used to measure discriminatory power of the models.”
MethodKhan, S. S., Page, C. B., WOJDYLA, D. M., Schwartz, Y. Y., Greenland, P., & Pencina, M. J. (2022) · Circulation · doi:10.1161/circulationaha.121.058426 · Cited by 30
Self-citationMechanisms of urban blue-green infrastructure on winter microclimate using artificial neural network
Fei, F., Wang, Y., Wang, L., Fukuda, H., Yao, W., Zhou, Y., & Dong, X. (2023) · Energy and Buildings · doi:10.1016/j.enbuild.2023.113188 · Cited by 28
de Vries, S., Jeffe, D. B., Davidson, N. O., Deshpande, A. D., & Schootman, M. (2014) · Cancer Causes & Control · doi:10.1007/s10552-014-0451-x · Cited by 32
“Because the effect sizes of race/ethnicity, neighborhood, and hospital characteristics were not very large and larger effects may raise the AUC more than predictors with only moderate or weak effects [35], this may indicate why we did not observe large improvements in the AUC when adding these predictors to the SEER-M…”
Otaki, Y., Hell, M. M., Slomka, P. J., Schuhbaeck, A., Gransar, H., Huber, B., Nakazato, R., Germano, G., Hayes, S. W., Thomson, L., et al. (2015) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2015.03.005 · Cited by 28
Cywiński, J. B., Mascha, E. J., Kurz, A. M., & Sessler, D. I. (2015) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-015-0398-8 · Cited by 28
“The NRI has possible limitations: it is sensitive to the calibration of the model; it may be significantly different from zero for very subtle improvements in prediction, and it is dependent on the number of reclassification categories used.(21,22) Nevertheless, in our study, the NRI showed a clear benefit of eGFR ove…”
MethodLingman, M., Hartford, M., Karlsson, T., Herlitz, J. D., Rubulis, A., Caidahl, K., & Bergfeldt, L. (2016) · International Journal of Cardiology · doi:10.1016/j.ijcard.2016.05.005 · Cited by 28
“The incremental predictive value of our VCG measures was at least “medium” sized [14], indicating that further evaluation of the clinical utility, effect on clinical outcomes and cost-effectiveness is warranted [15].”
MethodInfluentialVerwijmeren, L., Peelen, L. M., van Klei, W. A., Daeter, E. J., van Dongen, E. P. A., & Noordzij, P. G. (2020) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.12.042 · Cited by 27
An, X.-N., Wei, Z.-N., Yao, X.-Y., Xu, J., Qian, W.-T., Pan, X., Shen, P., Shi, H., Zhang, W., Chen, X., et al. (2021) · Clinical and Experimental Rheumatology · doi:10.55563/clinexprheumatol/24ep0c · Cited by 27
Iki, M., Winzenrieth, R., Tamaki, J., Sato, Y., Dongmei, N., Kajita, E., Kouda, K., Yura, A., Tachiki, T., Kamiya, K., et al. (2021) · Osteoporosis International · doi:10.1007/s00198-021-06013-2 · Cited by 28
“obtain a significant improvement in AUC relative to the reference model [22].”
ResultMehta, R. C., Ning, H., Bansal, N., Cohen, J. B., Srivastava, A., Dobre, M. A., Michos, E. D., Rahman, M., Townsend, R. R., Seliger, S. L., et al. (2021) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2021.10.007 · Cited by 27
Dynamic Risk Prediction for Hospital-Acquired Pressure Injury in Adult Critical Care Patients
Shui, A. M., Kim, P., Aribindi, V. K., Huang, C., Kim, M.-O., Rangarajan, S., Schorger, K., Aldrich, J. M., & Lee, H. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000580 · Cited by 27
Stamatelopoulos, K., Mueller‐Hennessen, M., Georgiopoulos, G. A., López‐Ayala, P., Sachse, M., Vlachogiannis, N. I., Sopova, K., Delialis, D., Bonini, F., Patras, R., et al. (2022) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2022.05.055 · Cited by 29
Hirshman, B. R., Wilson, B. R., Ali, M. A., Proudfoot, J., Koiso, T., Nagano, O., Carter, B. S., Serizawa, T., Yamamoto, M., & Chen, C. C. (2017) · Neurosurgery · doi:10.1093/neuros/nyx225 · Cited by 28
Ali, M. A., Hirshman, B. R., Wilson, B. R., Schüpper, A. J., Joshi, R. S., Proudfoot, J. A., Goetsch, S. J., Alksne, J. F., Ott, K. H., Aiyama, H., et al. (2017) · World Neurosurgery · doi:10.1016/j.wneu.2017.07.109 · Cited by 27
“NRI> 0 and IDI are two established and generally accepted measures for quantifying classif ication improvement.[15-17] The NRI”
Latronico, N., Piva, S., Fagoni, N., Pinelli, L., Frigerio, M., Tintori, D., Berardino, M., Bottazzi, A., Carnevale, L., Casalicchio, T., et al. (2020) · Critical Care · doi:10.1186/s13054-020-2746-5 · Cited by 26
Kähönen, E., Lyytikäinen, L., Aatola, H., Koivistoinen, T., Haarala, A., Sipilä, K., Juonala, M., Lehtimäki, T. J., Raitakari, O. T., Kähönen, M. A. P., et al. (2020) · Blood Pressure · doi:10.1080/08037051.2020.1783992 · Cited by 26
“which is a good improvement in reclassification [20].”
BackgroundCoughlin, L. N., Nahum-Shani, I. B., Bonar, E. E., Philyaw-Kotov, M. L., Rabbi, M., Klasnja, P., & Walton, M. A. (2021) · Substance Use & Misuse · doi:10.1080/10826084.2021.1972314 · Cited by 26
“AUCs range from 0 to 1 with 0.56, 0.64, and 0.71 denoting small, medium, and large AUC effect sizes, respectively (Pencina et al., 2012; Rice & Harris, 2005).”
InfluentialPrediction of Cardiovascular Events by Pulse Waveform Parameters: Analysis of CARTaGENE
Desbiens, L., Fortier, C., Nadeau‐Fredette, A., Madore, F., Hametner, B., Wassertheurer, S., Agharazii, M., & Goupil, R. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.122.026603 · Cited by 26
“47 Although the predictive value of waveform parameters may be small in magnitude, their addition to the ASCVD score did re-classify a clinically significant proportion of individuals.”
Ørsted, D. D., Bojesen, S. E., Kamstrup, P. R., & Nordestgaard, B. G. (2013) · European Urology · doi:10.1016/j.eururo.2013.01.028 · Cited by 26
“However, obtaining increases in Harrell C index for a new model is very difficult when a strong risk factor is already included in the old model [27].”
BackgroundMcKearnan, S. B., Wolfson, J. A., Vock, D. M., Vazquez‐Benitez, G., & O’Connor, P. J. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx374 · Cited by 26
“When used with clinically relevant categories, the results of the NRI can be a valuable addition to analysis, consistent with prior recommendations (27).”
ResultLai, E. L., Huang, W., Chen, H., Chen, J.-P., Chen, D.-Y., Hsieh, T., Hung, W., Lai, K., Lin, C.-T., Tang, K., et al. (2020) · Archives of Osteoporosis · doi:10.1007/s11657-020-00726-3 · Cited by 26
Serum Levels of 1,5-Anhydroglucitol and Risk of Incident End-Stage Renal Disease
Rebholz, C. M., Grams, M. E., Chen, Y., Gross, A. L., Sang, Y., Coresh, J., & Selvin, E. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx167 · Cited by 26
Louis, G. M. B., Druschel, C. M., Bell, E. M., Stern, J. E., Luke, B., McLain, A. C., Sundaram, R., & Yeung, E. H. (2015) · Fertility and Sterility · doi:10.1016/j.fertnstert.2015.02.033 · Cited by 25
Wen, D., Xu, Z., An, R. F., Ren, J., Jia, Y., Li, J., & Zheng, M. (2021) · Clinical Radiology · doi:10.1016/j.crad.2021.10.019 · Cited by 26
The distress thermometer as a prognostic tool for one-year survival among patients with lung cancer
Geerse, O. P., Brandenbarg, D., Kerstjens, H. A. M., Berendsen, A. J., Duijts, S. F. A., Burger, H., Holtman, G. A., Hoekstra‐Weebers, J. E. H. M., & Hiltermann, T. J. N. (2019) · Lung Cancer · doi:10.1016/j.lungcan.2019.02.008 · Cited by 25
Lin, X.-J., Qiao, N., Shen, Y., Fang, H., Xue, Q., Cui, B., Chen, L., Zhu, H., Zhang, S., Chen, Y., et al. (2021) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-20-4375 · Cited by 26
Estimated Glomerular Filtration Rate: Fit for What Purpose?
Warnock, D. G. (2016) · The Nephron journals/Nephron journals · doi:10.1159/000444062 · Cited by 28
Metadata
- Authors
- Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P.
- Affiliations
- Boston University (US)
- Corresponding author
- Pencina, Michael J.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-08-08
- Volume
- 176
- Issue
- 6
- Pages
- 473-481
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- PMC
- PMC3530349
- OpenAlex
- W2157026796
- Cited by
- 506
- References
- 37
- Citations by year
- 2026: 36 · 2025: 26 · 2024: 22 · 2023: 30 · 2022: 38 · 2021: 43 · 2020: 45 · 2019: 32 · 2018: 32 · 2017: 30 · 2016: 45 · 2015: 44 · 2014: 50 · 2013: 25 · 2012: 6
- FWCI
- 38.21
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- American Journal of Epidemiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR000150, n01hc25195-41-0-1)
- National Heart, Lung, and Blood Institute (N01‐HC25195, N01-HC-025195, RC1HL101056)
- National Center for Research Resources (UL1RR025741)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk prediction model, model discrimination, area under the receiver operating characteristic, integrated discrimination improvement, nested model comparison, effect size, Framingham risk score, incident atrial fibrillation
- MeSH terms
- Adult, Aged, Aged, 80 and over, Atrial Fibrillation, Computer Simulation, Data Interpretation, Statistical, Humans, Middle Aged, Prognosis, Risk Factors, ROC Curve, Biomarkers, Logistic Models, Odds Ratio, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.