Using Relative Utility Curves to Evaluate Risk Prediction
Open accessAbstract
Summary: Because many medical decisions are based on risk prediction models that are constructed from medical history and results of tests, the evaluation of these prediction models is important. This paper makes five contributions to this evaluation: the relative utility curve which gauges the potential for better prediction in terms of utilities, without the need for a reference level for one utility, while providing a sensitivity analysis for misspecification of utilities, the relevant region, which is the set of values of prediction performance that are consistent with the recommended treatment status in the absence of prediction, the test threshold, which is the minimum number of tests that would be traded for a true positive prediction in order for the expected utility to be non-negative, the evaluation of two-stage predictions that reduce test costs and connections between various measures of performance of prediction. An application involving the risk of cardiovascular disease is discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The utility of an AI model-guided workflow is highly dependent on the benefits and costs of possible outcomes, and thus the inputs to the APLUS simulation tool are specific to the use case.
“The utility of the workflow is highly dependent on the benefits and costs of possible outcomes 10-12 , and thus the inputs to APLUS are specific to the use case.”
From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 15
- Briggs WM Zaretski R The skill plot: a graphical technique for evaluating continuous diagnostic tests Biometrics 2008 64 250 256 18304288
- Cook NR Use and misuse of the receiver operating characteristic curve in risk prediction Circulation 2007 115 928 935 17309939
- Gail MH Pfeiffer RM On criteria for evaluating models for absolute risk Biostatistics 2005 6 227 239 15772102
- Greenhouse SW Mantel N The evaluation of diagnostic tests Biometrics 1950 6 399 412 14791576
- Greenland S The need for reorientation toward cost-effective prediction: Comments on “Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond” by M. J. Pencina et al., Statistics in Medicine Statistics in Medicine 2008 27 199 206 Correction. Statistics in Medicine 27, 316 17729377
- Heagerty PJ, Lumley T Pepe MS Time-dependent ROC curve for censored survival data and a diagnostic marker Biometrics 2000 56 337 334 10877287
- Huang Y Pepe MS Feng Z Evaluating predictiveness of a continuous marker Biometrics 2007 63 1181 1188 17489968 PMC3059154
- Michiels S Koscielny S Hill C Prediction of cancer outcome with microarrays: a multiple random validation strategy Lancet 2005 365 488 92 15705458
- Pauker SG Kassirer JP Therapeutic decision making: a cost-benefit analysis New England Journal of Medicine 1975 293 229 234 1143303
- Peirce CS The numerical measure of the success of predictions Science 1884 4 453 454 17795531
- Pencina MJ D’Agostino RB D’Agostino RB Jr. Vasan RS Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond Statistics in Medicine 2008 27 157 172 17569110
- Ransahoff DF Feinstein AR Problems of spectrum and bias in evaluating the efficacy of diagnostic tests New England Journal of Medicine 1978 299 926 930 692598
- Ridker PM Cook NR Lee IM Gordon D Gaziano JM Manson JE Hennekens CH Buring JE A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women New England Journal of Medicine 2005 352 1293 1304 15753114
- Vickers AJ Elkin EB Decision curve analysis: a novel method for evaluating prediction models Medical Decision Making 2006 26 565 574 17099194 PMC2577036
- Weinstein MC Fineberg HV Elstein AS Frazier NS Neuhauser D Neutra RR McNeil BJ Clinical Decision Analysis 1980 W. B. Saunders Philadelphia
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationA simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174 · see citing works
Self-citationCollins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
The genetic interpretation of area under the ROC curve in genomic profiling
Wray, N. R., Yang, J., Goddard, M. E., & Visscher, P. M. (2010) · PLoS Genetics · doi:10.1371/journal.pgen.1000864 · Cited by 334 · see citing works
Three myths about risk thresholds for prediction models
on behalf of the Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Wynants, L., van Smeden, M., McLernon, D. J., Timmerman, D., Steyerberg, E. W., & Van Calster, B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1425-3 · Cited by 213 · see citing works
Pepe, M. S., Fan, J., Feng, Z., Gerds, T., & Hilden, J. (2015) · Statistics in Biosciences · doi:10.1007/s12561-014-9118-0 · Cited by 191 · see citing works
“This is a measure closely related to the decision curve [2,22] and the relative utility [1].”
Verbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.028 · Cited by 138 · see citing works
Callahan, A., McElfresh, D., Banda, J. M., Bunney, G., Char, D., Chen, J., Corbin, C. K., Dash, D., Downing, N. L., Jain, S. S., et al. (2024) · NEJM Catalyst · doi:10.1056/cat.24.0131 · Cited by 61 · see citing works
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42 · see citing works
Assessing the clinical impact of risk models for opting out of treatment
Kerr, K. F., Brown, M. D., Marsh, T. L., & Janes, H. (2019) · Medical Decision Making · doi:10.1177/0272989x18819479 · Cited by 23 · see citing works
- Authors
- Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S.
- Affiliations
- National Cancer Institute (US)
- Brigham and Women's Hospital (US)
- Memorial Sloan Kettering Cancer Center (US)
- National Institutes of Health (US)
- Corresponding author
- Baker, Stuart G.
- Venue
- Journal of the Royal Statistical Society Series A: Statistics in Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2009-04-03
- Volume
- 172
- Issue
- 4
- Pages
- 729-748
- ISSN
- 0964-1998 (ISSN-L)
- 1467-985X
- PMC
- PMC2804257
- OpenAlex
- W2070808305
- Cited by
- 142
- References
- 0
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 5 · 2023: 5 · 2022: 3 · 2021: 13 · 2020: 7 · 2019: 17 · 2018: 6 · 2017: 4 · 2016: 10 · 2015: 8 · 2014: 14 · 2013: 11 · 2012: 12
- FWCI
- 7.49
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Heart, Lung and Blood Institute
- National Cancer Institute (HL043851, CA047988)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- risk prediction model, expected utility, sensitivity analysis, test threshold, two-stage prediction, test cost reduction, cardiovascular risk, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
