Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
Open accessAbstract
Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver‐operating‐characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers. Copyright © 2007 John Wiley & Sons, Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Net reclassification and AUC were calculated from Cox models adjusted for traditional risk factors with and without TMAO.
“Net reclassification and area under Receiver Operator Characteristic curve were calculated according to mortality risk estimated using Cox models adjusted for above mentioned traditional risk factors with versus without TMAO as previously described22.”
The net reclassification improvement is improper.
“Dedicated measures, such as the widely used but improper net reclassification improvement, are available for evaluating competing models.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [118]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · see citing works
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,944 · see citing works
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581 · see citing works
Self-citationKupper, L. L., Hosmer, D. W., & Lemeshow, S. A. (1990) · Journal of the American Statistical Association · doi:10.2307/2290035 · Cited by 7,690 · see citing works
Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project
Conroy, R. (2003) · European Heart Journal · doi:10.1016/s0195-668x(03)00114-3 · Cited by 5,290 · see citing works
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
D'Agostino, R. B., Grundy, S., Sullivan, L. M., Wilson, P., & for the CHD Risk Prediction Group (2001) · JAMA · doi:10.1001/jama.286.2.180 · Cited by 2,108 · see citing works
Self-citationAn updated coronary risk profile. A statement for health professionals.
Anderson, K. M., Wilson, P. W., Odell, P. M., & Kannel, W. B. (1991) · Circulation · doi:10.1161/01.cir.83.1.356 · Cited by 1,834 · see citing works
Ridker, P. M., Buring, J. E., Rifai, N., & Cook, N. R. (2007) · JAMA · doi:10.1001/jama.297.6.611 · Cited by 1,814 · see citing works
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
Pencina, M. J., & D’Agostino, R. B. (2004) · Statistics in Medicine · doi:10.1002/sim.1802 · Cited by 1,446 · see citing works
Self-citationMultiple Biomarkers for the Prediction of First Major Cardiovascular Events and Death
Wang, T. J., Gona, P. N., Larson, M. G., Tofler, G. H., Levy, D. A., Newton‐Cheh, C., Jacques, P. F., Rifai, N., Selhub, J., Robins, S. J., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa055373 · Cited by 1,342 · see citing works
Self-citationPepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276 · see citing works
A general cardiovascular risk profile: The Framingham study
Kannel, W. B., McGee, D., & Gordon, T. (1976) · The American Journal of Cardiology · doi:10.1016/0002-9149(76)90061-8 · Cited by 1,189 · see citing works
Encyclopedia of Biostatistics.
Kemp, C. D., Armitage, P., & Coulton, T. (1998) · Biometrics · doi:10.2307/2533871 · Cited by 674 · see citing works
Primary and subsequent coronary risk appraisal: New results from the Framingham study
D’Agostino, R. B., Russell, M. W., Huse, D. M., Ellison, R. C., Silbershatz, H., Wilson, P. W. F., & Hartz, S. C. (2000) · American Heart Journal · doi:10.1016/s0002-8703(00)90236-9 · Cited by 588 · see citing works
Self-citationGamma Glutamyl Transferase and Metabolic Syndrome, Cardiovascular Disease, and Mortality Risk
Lee, D. S., Evans, J. C., Robins, S. J., Wilson, P. W. F., Albano, I., Fox, C. S., Wang, T. J., Benjamin, E. J., D’Agostino, R. B., & Vasan, R. S. (2006) · Arteriosclerosis Thrombosis and Vascular Biology · doi:10.1161/01.atv.0000251993.20372.40 · Cited by 528 · see citing works
Self-citationThe effect of including C-reactive protein in cardiovascular risk prediction models for women
Cook, N. R., Buring, J. E., & Ridker, P. M. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-1-200607040-00128 · Cited by 504 · see citing works
Lee, K. L., & Mark, D. B. (1996) · Cited by 485 · see citing works
The Limitations of Risk Factors as Prognostic Tools
Ware, J. H. (2006) · New England Journal of Medicine · doi:10.1056/nejmp068249 · Cited by 466 · see citing works
External correspondence: Decompositions of the mean probability score
Yates, J. (1982) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(82)90237-9 · Cited by 466 · see citing works
An adaptation of the Framingham coronary heart disease risk function to European Mediterranean areas
Marrugat, J., D’Agostino, R. B., SULLIVAN, L. M., Elosúa, R., Wilson, P., Ordovás, J. M., Solanas, P., Cordón, F., Ramos, R., Sala, J., et al. (2003) · Journal of Epidemiology & Community Health · doi:10.1136/jech.57.8.634 · Cited by 369 · see citing works
Self-citationEvaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures
D’Agostino, R. B., & Nam, B. (2003) · Handbook of statistics · doi:10.1016/s0169-7161(03)23001-7 · Cited by 340 · see citing works
Self-citationEstimation of time‐dependent area under the ROC curve for long‐term risk prediction
Chambless, L. E., & Diao, G. (2005) · Statistics in Medicine · doi:10.1002/sim.2299 · Cited by 309 · see citing works
Integrating the Predictiveness of a Marker with Its Performance as a Classifier
Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R. L., Thompson, I. M., & Zheng, Y. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm305 · Cited by 291 · see citing works
When Is a New Prediction Marker Useful?
Greenland, P., & O’Malley, P. G. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.21.2454 · Cited by 123 · see citing works
Risk prediction and finding new independent prognostic factors
D’Agostino, R. B. (2006) · Journal of Hypertension · doi:10.1097/01.hjh.0000217845.57466.cc · Cited by 16 · see citing works
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Research works printed but not linked: 36
- Zhou X-H, Obuchowski NA, McClish DK. Statistical Methods in Diagnostic Medicine. New York, NY: John Wiley & Sons; 2002.
- Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford, UK: Oxford University Press; 2003.
- Li J, Jiang B, Fine JP. Multicategory reclassification statistics for assessing improvements in diagnostic accuracy. Biostatistics. 2013;14(2):382-394.
- Li J, Fine JP. ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics. 2008;9(3):566-576.
- Van Calster B, Vergouwe Y, Looman CWN, Van Belle V, Timmerman D, Steyerberg EW. Assessing the discriminative ability of risk models for more than two outcome categories. Eur J Epidemiol. 2012;27(10):761-770.
- Kwak C, Clayton-Matthews A. Multinomial logistic regression. Nursing Research. 2002;51(6):404-410.
- Vapnik V. Statistical Learning Theory. New York, NY: Wiley; 1998.
- Suykens JAK, Vandewalle J. Least squares support vector machine classifiers. Neural Process Lett. 1999;9(3):293-300.
- Crammer K, Singer Y. On the algorithmic implementation of multiclass kernel-based vector machines. J Mach Learn Res. 2001;2:265-292.
- Lee Y, Lin Y, Wahba G. Multicategory support vector machines, theory, and application to the classification of microarray data and satellite radiance data. J Am Stat Assoc. 2004;99(465):67-81.
- Liaw A, Wiener M. Classification and regression by randomforest. R News. 2002;2(3):18-22.
- Mika S, Ratsch G, Weston J, Scholkopf B, Mullers KR. Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop; 1999; Madison, WI.
- Schalkoff RJ. Artificial Neural Networks. Vol 1. New York, NY: McGraw-Hill; 1997.
- Shore J, Johnson R. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans Inf Theory. 1980;26(1):26-37.
- Gardner MW, Dorling SR. Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences. Atmospheric Environment. 1998;32(14-15):2627-2636.
- Zhao K, Huang L. Minibatch and parallelization for online large margin structured learning. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2013; Atlanta, GA.
- Hecht-Nielsen R. Theory of the backpropagation neural network. Neural Networks. 1988;1(1):445-448.
- Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NIPS Proceedings); 2012.
- Li J, Chow Y, Wong WK, Wong TY. Sorting multiple classes in multi-dimensional ROC analysis: parametric and nonparametric approaches. Biomarkers. 2014;19(1):1-8.
- Novoselova N, Beffa CD, Wang J, et al. HUM calculator and HUM package for R: easy-to-use software tools for multicategory receiver operating characteristic analysis. Bioinformatics. 2014;30(11):1635.
- Golub TR, Slonim DK, Tamayo P, et al. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science. 1999;286:531-537.
- Nakas CT, Yiannoutsos CT. Ordered multiple-class ROC analysis with continuous measurements. Statist Med. 2004;23(22):3437-3449.
- Cox DR, Wermuth N. A comment on the coefficient of determination for binary response. Am Stat. 1992;46(1):1-4.
- Menard S. Coefficients of determination for multiple logistic regression analysis. Am Stat. 2000;54(1):17-24.
- Hu B, Palta M, Shao J. Properties of R 2 statistics for logistic regression. Statist Med. 2006;25(8):1383-1395.
- Tjur T. Coefficients of determination in logistic regression models-a new proposal: the coefficient of discrimination. Am Stat. 2009;63(4):366-372.
- Van Calster B, Van Belle V, Vergouwe Y, Timmerman D, Van Huffel S, Steyerberg EW. Extending the c-statistic to nominal polytomous outcomes: The polytomous discrimination index. Statist Med. 2012;31(23):2610-2626.
- Pencina MJ, D'Agostino RB Sr, D'Agostino RB, Vasan RS Jr. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statist Med. 2008;27(2):157-172.
- Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models, a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138.
- Pencina MJ, D'Agostino RB Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statist Med. 2011;30(1):11-21.
- Pencina MJ, D'Agostino RB Sr, Demler OV. Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvements for normal variables and nested models. Statist Med. 2012;31(2):101-113.
- Austin PC, Steyerberg EW. Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Statist Med. 2013;32(4):661-672.
- Pepe MS, Feng Z, Gu JW. Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine. Statist Med. 2008;27(2):173-181. https://doi.org/10.1002/sim. 2929 · DOI
- Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Statist Med. 2014;33(19):3405-3414.
- Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2014;25(1):114-121.
- Hilden J. Commentary: on NRI, IDI, and "good-looking" statistics with nothing underneath. Epidemiology. 2014;25(2):265-267.
Cited by
Sorted by citations
General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study
D’Agostino, R. B., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008) · Circulation · doi:10.1161/circulationaha.107.699579 · Cited by 7,491 · see citing works
Self-citationMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citation2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945 · see citing works
Self-citationEstimating glomerular filtration rate from serum creatinine and cystatin C
Inker, L. A., Schmid, C. H., Tighiouart, H., Eckfeldt, J. H., Feldman, H. I., Greene, T., Kusek, J. W., Manzi, J., Van Lente, F., Zhang, Y. L., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1114248 · Cited by 4,102 · see citing works
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437741.48606.98 · Cited by 3,697 · see citing works
Self-citationRoss, R., Blair, S. N., Arena, R., Church, T. S., Després, J.-P., Franklin, B. A., Haskell, W. L., Kaminsky, L. A., Levine, B. D., Lavie, C. J., et al. (2016) · Circulation · doi:10.1161/cir.0000000000000461 · Cited by 2,486 · see citing works
Arterial Stiffness and Cardiovascular Events: The Framingham Heart Study
Mitchell, G. F., Hwang, S.-J., Vasan, R. S., Larson, M. G., Pencina, M. J., Hamburg, N. M., Vita, J. A., Levy, D., & Benjamin, E. J. (2010) · Circulation · doi:10.1161/circulationaha.109.886655 · Cited by 2,225 · see citing works
Self-citationGeneral and abdominal adiposity and risk of death in Europe
Pischon, T., Boeing, H., Hoffmann, K., Bergmann, M., Schulze, M. B., Overvad, K., van der Schouw, Y. T., Spencer, E., Moons, K. G. M., Tjønneland, A., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0801891 · Cited by 2,155 · see citing works
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Ross, R., Neeland, I. J., Yamashita, S., Shai, I., Seidell, J., Magni, P., Santos, R. D., Arsenault, B., Cuevas, A., Hu, F. B., et al. (2020) · Nature Reviews Endocrinology · doi:10.1038/s41574-019-0310-7 · Cited by 1,966 · see citing works
Cholesterol efflux capacity, high-density lipoprotein function, and atherosclerosis
Khera, A. V., Cuchel, M., de la Llera-Moya, M., Rodrigues, A., Burke, M. F., Jafri, K., French, B. C., Phillips, J. A., Mucksavage, M. L., Wilensky, R. L., et al. (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1001689 · Cited by 1,943 · see citing works
Aortic Pulse Wave Velocity Improves Cardiovascular Event Prediction
Ben-Shlomo, Y., Spears, M., Boustred, C., May, M., Anderson, S. G., Benjamin, E. J., Boutouyrie, P., Cameron, J., Chen, C.-H., Cruickshank, J. K., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.09.063 · Cited by 1,835 · see citing works
Self-citationAlba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
High performance plasma amyloid-β biomarkers for Alzheimer's disease
Nakamura, A., Kaneko, N., Villemagne, V. L., Kato, T., Doecke, J., Doré, V., Fowler, C., Li, Q.-X., Martins, R., Rowe, C., et al. (2018) · Nature · doi:10.1038/nature25456 · Cited by 1,681 · see citing works
Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Glycated hemoglobin, diabetes, and cardiovascular risk in nondiabetic adults
Selvin, E., Steffes, M. W., Zhu, H., Matsushita, K., Wagenknecht, L., Pankow, J., Coresh, J., & Brancati, F. L. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa0908359 · Cited by 1,544 · see citing works
Uno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · see citing works
Self-citationSevere hypoglycemia and risks of vascular events and death
Zoungas, S., Patel, A., Chalmers, J., de Galan, B. E., Li, Q., Billot, L., Woodward, M., Ninomiya, T., Neal, B., MacMahon, S., et al. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa1003795 · Cited by 1,471 · see citing works
HDL cholesterol efflux capacity and incident cardiovascular events
Rohatgi, A., Khera, A., Berry, J. D., Givens, E. G., Ayers, C. R., Wedin, K. E., Neeland, I. J., Yuhanna, I. S., Rader, D. R., de Lemos, J. A., et al. (2014) · New England Journal of Medicine · doi:10.1056/nejmoa1409065 · Cited by 1,390 · see citing works
Jiménez, D. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.199 · Cited by 1,354 · see citing works
The Emerging Risk Factors Collaboration (2011) · The Lancet · doi:10.1016/s0140-6736(11)60105-0 · Cited by 1,216 · see citing works
Tang, W. H. W., Wang, Z., Kennedy, D. J., Wu, Y., Buffa, J. A., Agatisa-Boyle, B., Li, X. S., Levison, B. S., & Hazen, S. L. (2015) · Circulation Research · doi:10.1161/circresaha.116.305360 · Cited by 1,223 · see citing works
Friberg, L., Rosenqvist, M., & Lip, G. Y. H. (2012) · European Heart Journal · doi:10.1093/eurheartj/ehr488 · Cited by 1,192 · see citing works
C-reactive protein, fibrinogen, and cardiovascular disease prediction
The Emerging Risk Factors Collaboration (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1107477 · Cited by 1,150 · see citing works
Prognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172 · see citing works
A combined comorbidity score predicted mortality in elderly patients better than existing scores
Gagne, J. J., Glynn, R. J., Avorn, J., Levin, R., & Schneeweiss, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.10.004 · Cited by 995 · see citing works
Clinical risk factors, DNA variants, and the development of type 2 diabetes
Lyssenko, V., Jonsson, A., Almgren, P., Pulizzi, N., Isomaa, B., Tuomi, T., Berglund, G., Altshuler, D., Nilsson, P., & Groop, L. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0801869 · Cited by 938 · see citing works
Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889 · see citing works
Yeboah, J., Folsom, A. R., Burke, G. L., Johnson, C., Polak, J. F., Post, W., Lima, J. A., Crouse, J. R., & Herrington, D. M. (2009) · Circulation · doi:10.1161/circulationaha.109.864801 · Cited by 888 · see citing works
Ridker, P. M., Paynter, N. P., Rifai, N., Gaziano, J. M., & Cook, N. R. (2008) · Circulation · doi:10.1161/circulationaha.108.814251 · Cited by 841 · see citing works
Genotype score in addition to common risk factors for prediction of type 2 diabetes
Meigs, J. B., Shrader, P., Sullivan, L. M., McAteer, J. B., Fox, C. S., Dupuis, J., Manning, A. K., Florez, J. C., Wilson, P. W. F., D'Agostino, R. B., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0804742 · Cited by 759 · see citing works
Self-citationAfilalo, J., Eisenberg, M. J., Morin, J.-F., Bergman, H., Monette, J., Noiseux, N., Perrault, L. P., Alexander, K. P., Langlois, Y., Dendukuri, N., et al. (2010) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.06.039 · Cited by 746 · see citing works
Iron deficiency in chronic heart failure: an international pooled analysis
Klip, I. T., Comin-Colet, J., Voors, A. A., Ponikowski, P., Enjuanes, C., Banasiak, W., Lok, D. J., Rosentryt, P., Torrens, A., Polonski, L., et al. (2013) · American Heart Journal · doi:10.1016/j.ahj.2013.01.017 · Cited by 736 · see citing works
Polymorphisms Associated with Cholesterol and Risk of Cardiovascular Events
Kathiresan, S., Melander, O., Anevski, D., Guiducci, C., Burtt, N. P., Roos, C., Hirschhorn, J. N., Berglund, G., Hedblad, B., Groop, L., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0706728 · Cited by 696 · see citing works
Cardiovascular Risk Prediction: Basic Concepts, Current Status, and Future Directions
Lloyd-Jones, D. M. (2010) · Circulation · doi:10.1161/circulationaha.109.849166 · Cited by 705 · see citing works
Tang, W. H. W., Wang, Z., Fan, Y., Levison, B., Hazen, J. E., Donahue, L. M., Wu, Y., & Hazen, S. L. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2014.02.617 · Cited by 681 · see citing works
Red blood cell distribution width and the risk of death in middle-aged and older adults
Patel, K. V., Ferrucci, L., Ershler, W. B., Longo, D. L., & Guralnik, J. M. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.11 · Cited by 618 · see citing works
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594 · see citing works
Self-citationRipatti, S., Tikkanen, E., Orho-Melander, M., Havulinna, A. S., Silander, K., Sharma, A., Guiducci, C., Perola, M., Jula, A., Sinisalo, J., et al. (2010) · The Lancet · doi:10.1016/s0140-6736(10)61267-6 · Cited by 562 · see citing works
Interpreting Incremental Value of Markers Added to Risk Prediction Models
Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws207 · Cited by 506 · see citing works
Self-citationPlasma protein patterns as comprehensive indicators of health
Williams, S. A., Kivimaki, M., Langenberg, C., Hingorani, A. D., Casas, J. P., Bouchard, C., Jonasson, C., Sarzynski, M. A., Shipley, M. J., Alexander, L., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0665-2 · Cited by 506 · see citing works
Mora, S., Otvos, J. D., Rifai, N., Rosenson, R. S., Buring, J. E., & Ridker, P. M. (2009) · Circulation · doi:10.1161/circulationaha.108.816181 · Cited by 490 · see citing works
Prognostic Utility of Novel Biomarkers of Cardiovascular Stress: The Framingham Heart Study
Wang, T. J., Wollert, K. C., Larson, M. G., Coglianese, E., McCabe, E. L., Cheng, S., Ho, J. E., Fradley, M. G., Ghorbani, A., Xanthakis, V., et al. (2012) · Circulation · doi:10.1161/circulationaha.112.129437 · Cited by 482 · see citing works
Li, X. S., Obeid, S., Klingenberg, R., Gencer, B., Mach, F., Räber, L., Windecker, S., Rodondi, N., Nanchen, D., Muller, O., et al. (2017) · European Heart Journal · doi:10.1093/eurheartj/ehw582 · Cited by 479 · see citing works
Ndrepepa, G., Tiroch, K., Fusaro, M., Keta, D., Seyfarth, M., Byrne, R. A., Pache, J., Alger, P., Mehilli, J., Schömig, A., et al. (2010) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2009.12.054 · Cited by 475 · see citing works
Green, P., Woglom, A. E., Genereux, P., Daneault, B., Paradis, J.-M., Schnell, S., Hawkey, M., Maurer, M. S., Kirtane, A. J., Kodali, S., et al. (2012) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2012.06.011 · Cited by 473 · see citing works
Baber, U., Mehran, R., Sartori, S., Schoos, M. M., Sillesen, H., Muntendam, P., Garcia, M. J., Gregson, J., Pocock, S., Falk, E., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.01.017 · Cited by 461 · see citing works
Lammers, W. J., Hirschfield, G. M., Corpechot, C., Nevens, F., Lindor, K. D., Janssen, H. L. A., Floreani, A., Ponsioen, C. Y., Mayo, M. J., Invernizzi, P., et al. (2015) · Gastroenterology · doi:10.1053/j.gastro.2015.07.061 · Cited by 441 · see citing works
Value and Limitations of Existing Scores for the Assessment of Cardiovascular Risk
Cooney, M. T., Dudina, A. L., & Graham, I. M. (2009) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2009.07.020 · Cited by 436 · see citing works
Hijazi, Z., Oldgren, J., Andersson, U., Connolly, S. J., Ezekowitz, M. D., Hohnloser, S. H., Reilly, P. A., Vinereanu, D., Siegbahn, A., Yusuf, S., et al. (2012) · Circulation · doi:10.1161/circulationaha.111.038729 · Cited by 402 · see citing works
Principles for high-quality, high-value testing
Power, M., Fell, G., & Wright, M. (2013) · Evidence Based Medicine · doi:10.1136/eb-2012-100645 · Cited by 394 · see citing works
Pirracchio, R., Petersen, M. L., Carone, M., Rigon, M. R., Chevret, S., & van der Laan, M. J. (2015) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(14)70239-5 · Cited by 374 · see citing works
Drug-Induced Death Signaling Strategy Rapidly Predicts Cancer Response to Chemotherapy
Montero, J., Sarosiek, K. A., DeAngelo, J. D., Maertens, O., Ryan, J., Ercan, D., Piao, H., Horowitz, N. S., Berkowitz, R. S., Matulonis, U., et al. (2015) · Cell · doi:10.1016/j.cell.2015.01.042 · Cited by 378 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Association between a literature-based genetic risk score and cardiovascular events in women
Paynter, N. P. (2010) · JAMA · doi:10.1001/jama.2010.119 · Cited by 361 · see citing works
Chien, K.-L., Chen, P.-C., Hsu, H.-C., Su, T.-C., Sung, F.-C., Chen, M.-F., & Lee, Y.-T. (2010) · SLEEP · doi:10.1093/sleep/33.2.177 · Cited by 350 · see citing works
de Ruijter, W., Westendorp, R. G. J., Assendelft, W. J. J., den Elzen, W. P. J., de Craen, A. J. M., le Cessie, S., & Gussekloo, J. (2009) · BMJ · doi:10.1136/bmj.a3083 · Cited by 339 · see citing works
Wang, D. D., Toledo, E., Hruby, A., Rosner, B. A., Willett, W. C., Sun, Q., Razquin, C., Zheng, Y., Ruiz-Canela, M., Guasch-Ferré, M., et al. (2017) · Circulation · doi:10.1161/circulationaha.116.024261 · Cited by 327 · see citing works
Risk prediction is improved by adding markers of subclinical organ damage to SCORE
Sehestedt, T., Jeppesen, J., Hansen, T. W., Wachtell, K., Ibsen, H., Torp-Petersen, C., Hildebrandt, P., & Olsen, M. H. (2010) · European Heart Journal · doi:10.1093/eurheartj/ehp546 · Cited by 315 · see citing works
Comparisons of established risk prediction models for cardiovascular disease: systematic review
Siontis, G. C. M., Tzoulaki, I., Siontis, K. C., & Ioannidis, J. P. A. (2012) · BMJ · doi:10.1136/bmj.e3318 · Cited by 300 · see citing works
Association Between Familial Atrial Fibrillation and Risk of New-Onset Atrial Fibrillation
Lubitz, S. A., Yin, X., Fontes, J. D., Magnani, J. W., Rienstra, M., Pai, M., Villalon, M. L., Vasan, R. S., Pencina, M. J., Levy, D., et al. (2010) · JAMA · doi:10.1001/jama.2010.1690 · Cited by 313 · see citing works
Self-citationC-Reactive Protein and Reclassification of Cardiovascular Risk in the Framingham Heart Study
Wilson, P. W. F., Pencina, M., Jacques, P., Selhub, J., D'Agostino, R., & O'Donnell, C. J. (2008) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.108.831198 · Cited by 299 · see citing works
Self-citationPlasma Copeptin and the Risk of Diabetes Mellitus
Enhörning, S., Wang, T. J., Nilsson, P. M., Almgren, P., Hedblad, B., Berglund, G., Struck, J., Morgenthaler, N. G., Bergmann, A., Lindholm, E., et al. (2010) · Circulation · doi:10.1161/circulationaha.109.909663 · Cited by 299 · see citing works
Risk assessment tools for identifying individuals at risk of developing type 2 diabetes
Buijsse, B., Simmons, R. K., Griffin, S. J., & Schulze, M. B. (2011) · Epidemiologic Reviews · doi:10.1093/epirev/mxq019 · Cited by 297 · see citing works
Xanthakis, V., Enserro, D. M., Murabito, J. M., Polak, J. F., Wollert, K. C., Januzzi, J. L., Wang, T. J., Tofler, G., & Vasan, R. S. (2014) · Circulation · doi:10.1161/circulationaha.114.009273 · Cited by 287 · see citing works
Self-citationExtreme lipoprotein(a) levels and improved cardiovascular risk prediction
Kamstrup, P. R., Tybjærg-Hansen, A., & Nordestgaard, B. G. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2012.12.023 · Cited by 284 · see citing works
Discrimination and Net Reclassification of Cardiovascular Risk With Lipoprotein(a)
Willeit, P., Kiechl, S., Kronenberg, F., Witztum, J. L., Santer, P., Mayr, M., Xu, Q., Mayr, A., Willeit, J., & Tsimikas, S. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2014.03.061 · Cited by 286 · see citing works
Oxidative stress, inflammation, endothelial dysfunction and incidence of type 2 diabetes
Odegaard, A. O., Jacobs, D. R., Sanchez, O. A., Goff, D. C., Reiner, A. P., & Gross, M. D. (2016) · Cardiovascular Diabetology · doi:10.1186/s12933-016-0369-6 · Cited by 286 · see citing works
Selvin, E., Rawlings, A. M., Grams, M., Klein, R., Sharrett, A. R., Steffes, M., & Coresh, J. (2014) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(13)70199-2 · Cited by 262 · see citing works
Brownrigg, J. R. W., Hughes, C. O., Burleigh, D., Karthikesalingam, A., Patterson, B. O., Holt, P. J., Thompson, M. M., de Lusignan, S., Ray, K. K., & Hinchliffe, R. J. (2016) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(16)30057-2 · Cited by 242 · see citing works
Trimethylamine <i>N</i>-oxide and prognosis in acute heart failure
Suzuki, T., Heaney, L. M., Bhandari, S. S., Jones, D. J. L., & Ng, L. L. (2016) · Heart · doi:10.1136/heartjnl-2015-308826 · Cited by 253 · see citing works
Quantifying importance of major risk factors for coronary heart disease
Pencina, M. J., Navar, A. M., Wojdyla, D., Sanchez, R. J., Khan, I., Elassal, J., D’Agostino, R. B., Peterson, E. D., & Sniderman, A. D. (2019) · Circulation · doi:10.1161/circulationaha.117.031855 · Cited by 247 · see citing works
Self-citationBinuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Hoffmann, U., Massaro, J. M., D'Agostino, R. B., Kathiresan, S., Fox, C. S., & O'Donnell, C. J. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.003144 · Cited by 239 · see citing works
Self-citationRim, T. H., Lee, C. J., Tham, Y.-C., Cheung, N., Yu, M., Lee, G., Kim, Y., Ting, D. S. W., Chong, C. C. Y., Choi, Y. S., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00043-1 · Cited by 234 · see citing works
Hicks, G. E., Shardell, M., Alley, D. E., Miller, R. R., Bandinelli, S., Guralnik, J., Lauretani, F., Simonsick, E. M., & Ferrucci, L. (2012) · The Journals of Gerontology Series A: Biological Sciences and Medical Sciences · doi:10.1093/gerona/glr055 · Cited by 229 · see citing works
Lorenz, M. W., Schaefer, C., Steinmetz, H., & Sitzer, M. (2010) · European Heart Journal · doi:10.1093/eurheartj/ehq189 · Cited by 231 · see citing works
N-Terminal Pro–B-Type Natriuretic Peptide for Risk Assessment in Patients With Atrial Fibrillation
Hijazi, Z., Wallentin, L., Siegbahn, A., Andersson, U., Christersson, C., Ezekowitz, J., Gersh, B. J., Hanna, M., Hohnloser, S., Horowitz, J., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2012.11.082 · Cited by 231 · see citing works
Diagnostic and prognostic prediction models
Hendriksen, J. M. T., Geersing, G. J., Moons, K. G. M., & de Groot, J. A. H. (2013) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12262 · Cited by 229 · see citing works
Cooney, M. T., Dudina, A., D'Agostino, R., & Graham, I. M. (2010) · Circulation · doi:10.1161/circulationaha.109.852756 · Cited by 222 · see citing works
Self-citationAssessing the incremental value of diagnostic and prognostic markers: a review and illustration
Steyerberg, E. W., Pencina, M. J., Lingsma, H. F., Kattan, M. W., Vickers, A. J., & Van Calster, B. (2012) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2011.02562.x · Cited by 219 · see citing works
Self-citationAutomated and interpretable patient ECG profiles for disease detection, tracking, and discovery
Tison, G. H., Zhang, J., Delling, F. N., & Deo, R. C. (2019) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.118.005289 · Cited by 208 · see citing works
Thanassoulis, G., Peloso, G. M., Pencina, M. J., Hoffmann, U., Fox, C. S., Cupples, L. A., Levy, D., D'Agostino, R. B., Hwang, S.-J., & O'Donnell, C. J. (2012) · Circulation: Cardiovascular Genetics · doi:10.1161/circgenetics.111.961342 · Cited by 211 · see citing works
Self-citationRidker, P. M. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.119214 · Cited by 206 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
A metabolome-wide association study of kidney function and disease in the general population
Sekula, P., Goek, O.-N., Quaye, L., Barrios, C., Levey, A. S., Römisch-Margl, W., Menni, C., Yet, I., Gieger, C., Inker, L. A., et al. (2016) · Journal of the American Society of Nephrology · doi:10.1681/asn.2014111099 · Cited by 203 · see citing works
Reporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195 · see citing works
Comparison of different measures of urinary protein excretion for prediction of renal events
Lambers Heerspink, H. J., Gansevoort, R. T., Brenner, B. M., Cooper, M. E., Parving, H. H., Shahinfar, S., & de Zeeuw, D. (2010) · Journal of the American Society of Nephrology · doi:10.1681/asn.2010010063 · Cited by 193 · see citing works
Hivert, M.-F., Jablonski, K. A., Perreault, L., Saxena, R., McAteer, J. B., Franks, P. W., Hamman, R. F., Kahn, S. E., Haffner, S., the DIAGRAM Consortium, et al. (2011) · Diabetes · doi:10.2337/db10-1119 · Cited by 191 · see citing works
Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J. (2015) · Statistics in Biosciences · doi:10.1007/s12561-014-9118-0 · Cited by 191 · see citing works
“large portion of simulations the NRI statistic was positive and statistically significant in the test dataset using the standard test based on the empirically estimated NRI divided by the standard error provided in [11,12].”
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185 · see citing works
Kim, R. B., Morse, B. L., Djurdjev, O., Tang, M., Muirhead, N., Barrett, B., Holmes, D. T., Madore, F., Clase, C. M., Rigatto, C., et al. (2016) · Kidney International · doi:10.1016/j.kint.2016.01.014 · Cited by 179 · see citing works
Beatty, A. L., Schiller, N. B., & Whooley, M. A. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2012.2198 · Cited by 174 · see citing works
Cardiovascular risk assessment scores for people with diabetes: a systematic review
Chamnan, P., Simmons, R. K., Sharp, S. J., Griffin, S. J., & Wareham, N. J. (2009) · Diabetologia · doi:10.1007/s00125-009-1454-0 · Cited by 176 · see citing works
Gait Speed and Operative Mortality in Older Adults Following Cardiac Surgery.
Afilalo, J., Kim, S., O’Brien, S., Brennan, J. M., Edwards, F. H., Mack, M. J., McClurken, J. B., Cleveland, J. C., Smith, P. K., Shahian, D. M., et al. (2016) · JAMA Cardiology · doi:10.1001/jamacardio.2016.0316 · Cited by 175 · see citing works
Extracellular matrix proteomics identifies molecular signature of symptomatic carotid plaques
Langley, S. R., Willeit, K., Didangelos, A., Matic, L. P., Skroblin, P., Barallobre-Barreiro, J., Lengquist, M., Rungger, G., Kapustin, A., Kedenko, L., et al. (2017) · Journal of Clinical Investigation · doi:10.1172/jci86924 · Cited by 174 · see citing works
Trimethylamine <i>N</i> ‐Oxide and Mortality Risk in Patients With Peripheral Artery Disease
Senthong, V., Wang, Z., Fan, Y., Wu, Y., Hazen, S. L., & Tang, W. H. W. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.116.004237 · Cited by 175 · see citing works
Chapman, B. P., Hampson, S., & Clarkin, J. (2014) · Developmental Psychology · doi:10.1037/a0034135 · Cited by 166 · see citing works
- Authors
- Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S.
- Affiliations
- Boston University (US)
- Corresponding author
- Pencina, Michael J.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2007-06-13
- Volume
- 27
- Issue
- 2
- Pages
- 157-172
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2112316706
- Cited by
- 6,493
- References
- 35
- Citations by year
- 2026: 421 · 2025: 235 · 2024: 258 · 2023: 290 · 2022: 329 · 2021: 401 · 2020: 379 · 2019: 336 · 2018: 319 · 2017: 375 · 2016: 433 · 2015: 433 · 2014: 452 · 2013: 516 · 2012: 436
- FWCI
- 150.62
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Education, India
- National Institutes of Health (n01hc25195-41-0-1)
- National Heart, Lung, and Blood Institute (2K24 HL 04334, N01‐HC25195, K24HL004334)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- cardiovascular risk prediction, biomarker analysis, area under the ROC curve, risk reclassification, Framingham Heart Study, preventive cardiology
- MeSH terms
- Cardiovascular Diseases, Humans, Risk Factors, ROC Curve, United States, Models, Statistical, Risk Assessment, Area Under Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
