The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Open accessAbstract
Background: Diagnostic and prognostic or predictive models serve different purposes. Whereas diagnostic models are usually used for classification, prognostic models incorporate the dimension of time, adding a stochastic element. Content: The ROC curve is typically used to evaluate clinical utility for both diagnostic and prognostic models. This curve assesses how well a test or model discriminates, or separates individuals into two classes, such as diseased and nondiseased. A strong risk predictor, such as lipids for cardiovascular disease, may have limited impact on the area under the curve, called the AUC or c-statistic, even if it alters predicted values. Calibration, measuring whether predicted probabilities agree with observed proportions, is another component of model accuracy important to assess. Reclassification can directly compare the clinical impact of two models by determining how many individuals would be reclassified into clinically relevant risk strata. For example, adding high-sensitivity C-reactive protein and family history to prediction models for cardiovascular disease using traditional risk factors moves approximately 30% of those at intermediate risk levels, such as 5%–10% or 10%–20% 10-year risk, into higher or lower risk categories, despite little change in the c-statistic. A calibration statistic can asses how well the new predicted values agree with those observed in the cross-classified data. Summary: Although it is useful for classification, evaluation of prognostic models should not rely solely on the ROC curve, but should assess both discrimination and calibration. Risk reclassification can aid in comparing the clinical impact of two models on risk for the individual, as well as the population.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grundy, S. M., Bilheimer, D. W., Chait, A., Clark, L. T., Denke, M. A., Havel, R. J., Hazzard, W. R., Hulley, S. B., Hunninghake, D. B., Kreisberg, R. A., et al. (1993) · JAMA · doi:10.1001/jama.269.23.3015 · Cited by 27,947
Expert Panel on Detection, E. (2001) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.285.19.2486 · Cited by 26,011
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,112
Self-citationThe Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Ridker, P. M., Buring, J. E., Rifai, N., & Cook, N. R. (2007) · JAMA · doi:10.1001/jama.297.6.611 · Cited by 1,814
Self-citationReceiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473
Cited by
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Self-citationMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
“The c-statistic should not be used as the only performance measure, however, since it is influenced by the distribution of predictor values and is often insensitive to inclusion of an additional predictor in the model [59,122–125].”
Ensuring fairness in machine learning to advance health equity.
Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018) · Annals of Internal Medicine · doi:10.7326/m18-1990 · Cited by 1,341
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Adiponectin Levels and Risk of Type 2 Diabetes: A Systematic Review and Meta-analysis
Li, S., Shin, H. J., Ding, E. L., & van Dam, R. M. (2009) · JAMA · doi:10.1001/jama.2009.976 · Cited by 991
Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Ridker, P. M., Paynter, N. P., Rifai, N., Gaziano, J. M., & Cook, N. R. (2008) · Circulation · doi:10.1161/circulationaha.108.814251 · Cited by 841
Self-citationPostoperative Biomarkers Predict Acute Kidney Injury and Poor Outcomes after Adult Cardiac Surgery
Parikh, C. R., Coca, S. G., Thiessen-Philbrook, H., Shlipak, M. G., Koyner, J. L., Wang, Z., Edelstein, C. L., Devarajan, P., Patel, U. D., Zappitelli, M., et al. (2011) · Journal of the American Society of Nephrology · doi:10.1681/asn.2010121302 · Cited by 754
Prognostic Models With Competing Risks: Methods and Application to Coronary Risk Prediction
Wolbers, M., Koller, M. T., Witteman, J. C. M., & Steyerberg, E. W. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a39056 · Cited by 625
Red blood cell distribution width and the risk of death in middle-aged and older adults
Patel, K. V., Ferrucci, L., Ershler, W. B., Longo, D. L., & Guralnik, J. M. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.11 · Cited by 618
Good and poor adherence: optimal cut-point for adherence measures using administrative claims data.
Karve, S., Cleves, M. A., Helm, M., Hudson, T. J., West, D. S., & Martin, B. C. (2009) · Current Medical Research and Opinion · doi:10.1185/03007990903126833 · Cited by 620
Buckley, D. I., Fu, R., Freeman, M., Rogers, K., & Helfand, M. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-7-200910060-00009 · Cited by 559
Rangaka, M. X., Wilkinson, K. A., Glynn, J. R., Ling, D., Menzies, D., Mwansa-Kambafwile, J., Fielding, K., Wilkinson, R. J., & Pai, M. (2012) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(11)70210-9 · Cited by 476
Impact of Platelet Reactivity on Clinical Outcomes After Percutaneous Coronary Intervention
Brar, S. S., ten Berg, J., Marcucci, R., Price, M. J., Valgimigli, M., Kim, H.-S., Patti, G., Breet, N. J., DiSciascio, G., Cuisset, T., et al. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2011.06.059 · Cited by 422
Pirracchio, R., Petersen, M. L., Carone, M., Rigon, M. R., Chevret, S., & van der Laan, M. J. (2015) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(14)70239-5 · Cited by 374
Helfand, M., Buckley, D. I., Freeman, M., Fu, R., Rogers, K., Fleming, C., & Humphrey, L. L. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-7-200910060-00010 · Cited by 382
Guthrie, W., Swineford, L. B., Nottke, C., & Wetherby, A. M. (2013) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12008 · Cited by 366
“Predictive Value of Time 1 ADOS-T Classifications Logistic regressions were run to examine the predictive value of T1 ADOS-T scores rather than ROC curves, as they provide an index of calibration (rather than discrimination), a critical component of prognostic models (see Cook, 2008).”
MethodParikh, C. R., Devarajan, P., Zappitelli, M., Sint, K., Thiessen-Philbrook, H., Li, S., Kim, R. W., Koyner, J. L., Coca, S. G., Edelstein, C. L., et al. (2011) · Journal of the American Society of Nephrology · doi:10.1681/asn.2010111163 · Cited by 351
Masson, S., Latini, R., Anand, I. S., Barlera, S., Angelici, L., Vago, T., Tognoni, G., & Cohn, J. N. (2008) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2008.04.069 · Cited by 348
Carrington, A. M., Manuel, D. G., Fieguth, P. W., Ramsay, T., Osmani, V., Wernly, B., Bennett, C., Hawken, S., Magwood, O., Sheikh, Y., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3145392 · Cited by 339
Comparisons of established risk prediction models for cardiovascular disease: systematic review
Siontis, G. C. M., Tzoulaki, I., Siontis, K. C., & Ioannidis, J. P. A. (2012) · BMJ · doi:10.1136/bmj.e3318 · Cited by 300
Søreide, K. (2009) · Journal of Clinical Pathology · doi:10.1136/jcp.2008.061010 · Cited by 300
Risk assessment tools for identifying individuals at risk of developing type 2 diabetes
Buijsse, B., Simmons, R. K., Griffin, S. J., & Schulze, M. B. (2011) · Epidemiologic Reviews · doi:10.1093/epirev/mxq019 · Cited by 297
“It has further been suggested that the contribution of new markers to the performance of prediction models should also be evaluated based on risk stratification (17, 19, 20).”
BackgroundMcGorrian, C., Yusuf, S., Islam, S., Jung, H., Rangarajan, S., Avezum, A., Prabhakaran, D., Almahmeed, W., Rumboldt, Z., Budaj, A., et al. (2011) · European Heart Journal · doi:10.1093/eurheartj/ehq448 · Cited by 281
Evaluation of Biomarkers and Surrogate Endpoints in Chronic Disease
(2010) · National Academies Press eBooks · doi:10.17226/12869 · Cited by 276
Copeptin: A novel, independent prognostic marker in patients with ischemic stroke
Katan, M., Fluri, F., Morgenthaler, N. G., Schuetz, P., Zweifel, C., Bingisser, R., Müller, K., Meckel, S., Gass, A., Kappos, L., et al. (2009) · Annals of Neurology · doi:10.1002/ana.21783 · Cited by 267
Paynter, N. P., Chasman, D. I., Buring, J. E., Shiffman, D., Cook, N. R., & Ridker, P. M. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-150-2-200901200-00003 · Cited by 262
Self-citationExalto, L. G., Biessels, G. J., Karter, A. J., Huang, E. S., Katon, W. J., Minkoff, J. R., & Whitmer, R. A. (2013) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(13)70048-2 · Cited by 244
Lorenz, M. W., Schaefer, C., Steinmetz, H., & Sitzer, M. (2010) · European Heart Journal · doi:10.1093/eurheartj/ehq189 · Cited by 231
Nishimura, K., Okamura, T., Watanabe, M., Nakai, M., Takegami, M., Higashiyama, A., Kokubo, Y., Okayama, A., & Miyamoto, Y. (2014) · Journal of Atherosclerosis and Thrombosis · doi:10.5551/jat.19356 · Cited by 228
Ramjee, V., Sperling, L. S., & Jacobson, T. A. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2011.05.009 · Cited by 157
“With clinical prognostic models, the mpact that a given marker may have on outcomes can be ccurately assessed by calculating the percentage of reclasified individuals expressed as the net reclassification index NRI) (44).”
InfluentialBaiden, P., LaBrenz, C. A., Asiedua-Baiden, G., & Muehlenkamp, J. J. (2020) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2020.02.029 · Cited by 118
Sato, K. K., Hayashi, T., Harita, N., Yoneda, T., Nakamura, Y., Endo, G., & Kambe, H. (2009) · Diabetes Care · doi:10.2337/dc08-1631 · Cited by 112
Søreide, K., Kørner, H., & Søreide, J. A. (2011) · Annals of Surgery · doi:10.1097/sla.0b013e318204a892 · Cited by 109
Rapid identification of slow healing wounds
Jung, K., Covington, S., Sen, C. K., Januszyk, M., Kirsner, R. S., Gurtner, G. C., & Shah, N. H. (2016) · Wound Repair and Regeneration · doi:10.1111/wrr.12384 · Cited by 100
“A prognostic model for delayed wound healing should discriminate between normal and delayed healing wounds, and provide accurate probabilities for those outcomes.(26,27) We evaluated discrimination by AUC, and calibration by Brier reliability (Figure 4).”
MethodRichard, E., Schmand, B. A., Eikelenboom, P., & Van Gool, W. A. (2013) · BMJ Open · doi:10.1136/bmjopen-2012-002541 · Cited by 84
Baiden, P., Graaf, G., Zaami, M., Acolatse, C. K., & Adeku, Y. (2019) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2019.02.018 · Cited by 77
“5 indicate that the model is no better than that which could have been obtained by chance (Cook, 2008).”
BackgroundMethodUpdate on the laboratory investigation of dyslipidemias
Ramasamy, I. (2018) · Clinica Chimica Acta · doi:10.1016/j.cca.2018.01.015 · Cited by 50
Wang, J. L., Patten, S., Sareen, J., Bolton, J., Schmitz, N., & MacQueen, G. (2014) · Depression and Anxiety · doi:10.1002/da.22215 · Cited by 43
Assessing the incremental role of novel and emerging risk factors.
Cook, N. R. (2010) · Current Cardiovascular Risk Reports · doi:10.1007/s12170-010-0084-x · Cited by 39
The impact of functional dependence and related surgical complications on postoperative mortality
Clifton, J. C., Engoren, M., Shotwell, M. S., Martin, B. J., Clemens, E. M., Guillamondegui, O. D., & Freundlich, R. E. (2022) · Journal of Medical Systems · doi:10.1007/s10916-021-01779-8 · Cited by 16
“The receiver operating characteristic (ROC) curve and concordance statistic assessed discriminative ability of the model, while a calibration plot measured the predictive accuracy [41, 42].”
MethodGonzález-Martin, A. M., Limón-Villegas, E. S., Reyes-Castillo, Z., Esparza-Ros, F., Hernández-Palma, L. A., Santillán-Rivera, M. S., Herrera-Amante, C. A., Ramos-García, C. O., & Righini, N. (2025) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk10030276 · Cited by 4
Metadata
- Authors
- Cook, N. R.
- Venue
- Clinical Chemistry
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-01-01
- Volume
- 54
- Issue
- 1
- Pages
- 17-23
- ISSN
- 0009-9147 (ISSN-L)
- 1530-8561
- OpenAlex
- W2130644475
- Cited by
- 691
- References
- 22
- FWCI
- 28.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.