Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests
Open accessAbstract
Many decisions in medicine involve trade-offs, such as between diagnosing patients with disease versus unnecessary additional testing for those who are healthy. Net benefit is an increasingly reported decision analytic measure that puts benefits and harms on the same scale. This is achieved by specifying an exchange rate, a clinical judgment of the relative value of benefits (such as detecting a cancer) and harms (such as unnecessary biopsy) associated with models, markers, and tests. The exchange rate can be derived by asking simple questions, such as the maximum number of patients a doctor would recommend for biopsy to find one cancer. As the answers to these sorts of questions are subjective, it is possible to plot net benefit for a range of reasonable exchange rates in a “decision curve.” For clinical prediction models, the exchange rate is related to the probability threshold to determine whether a patient is classified as being positive or negative for a disease. Net benefit is useful for determining whether basing clinical decisions on a model, marker, or test would do more good than harm. This is in contrast to traditional measures such as sensitivity, specificity, or area under the curve, which are statistical abstractions not directly informative about clinical value. Recent years have seen an increase in practical applications of net benefit analysis to research data. This is a welcome development, since decision analytic techniques are of particular value when the purpose of a model, marker, or test is to help doctors make better clinical decisions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Balancing sensitivity with false alarms and lead time for alerts remains a persistent challenge, and the optimal balance varies by use case.
“Balancing sensitivity with false alarms and lead time for alerts remains a persistent challenge, and the optimal balance varies by use case [41].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [41]
The paper’s “Decision making and net benefit” section describes the trade-off between identifying disease and subjecting disease-free patients to unnecessary interventions. “Choice of threshold probabilities” explains that acceptable thresholds vary with clinical judgments and patient preferences. Our copy does not address lead time for alerts, so the citation supports the trade-off but not that part of the claim. (LLM, conf=0.87)
Experimental — accuracy not yet measuredDecision analysis methods can assess whether a prediction model should be used in practice by incorporating and quantifying its clinical impact, considering anticipated benefits, risks, and costs.
“Decision analysis methods can be used to assess whether a prediction model should be used in practice by incorporating and quantifying its clinical impact, considering the anticipated benefits, risks, and costs.104”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [104]
Net benefit brings the benefits and harms of a decision strategy on the same scale so they can be compared.
“Net benefit brings the benefits and harms of a decision strategy (eg, to decide for or against treatment based on a prediction model) on the same scale so they can be compared.104”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [104]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 24,015 · see citing works
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Self-citationAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationVickers, A. J., Cronin, A. M., Elkin, E. B., & Gönen, M. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-53 · Cited by 1,377 · see citing works
Self-citationCalibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Collins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 245 · see citing works
Vickers, A. J., Cronin, A. M., Roobol, M. J., Savage, C. J., Peltola, M. T., Pettersson, K. S. I., Scardino, P. T., Schröder, F. H., & Lilja, H. G. (2010) · Journal of Clinical Oncology · doi:10.1200/jco.2009.24.1968 · Cited by 237 · see citing works
Self-citationAnkerst, D. P., Hoefler, J., Bock, S., Goodman, P. J., Vickers, A., Hernandez, J., Sokoll, L. J., Sanda, M. G., Wei, J. T., Leach, R. J., et al. (2014) · Urology · doi:10.1016/j.urology.2014.02.035 · Cited by 236 · see citing works
Self-citationAssessing the incremental value of diagnostic and prognostic markers: a review and illustration
Steyerberg, E. W., Pencina, M. J., Lingsma, H. F., Kattan, M. W., Vickers, A. J., & Van Calster, B. (2012) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2011.02562.x · Cited by 219 · see citing works
Self-citationDecision Analysis for the Evaluation of Diagnostic Tests, Prediction Models, and Molecular Markers
Vickers, A. J. (2008) · The American Statistician · doi:10.1198/000313008x370302 · Cited by 141 · see citing works
Self-citationDecision Curve Analysis: A Discussion
Steyerberg, E. W., & Vickers, A. J. (2007) · Medical Decision Making · doi:10.1177/0272989x07312725 · Cited by 135 · see citing works
Self-citationEvaluation of Markers and Risk Prediction Models
CALSTER, B. V., Vickers, A. J., Pencina, M. J., Baker, S. G., Timmerman, D., & Steyerberg, E. W. (2013) · Medical Decision Making · doi:10.1177/0272989x12470757 · Cited by 76 · see citing works
Self-citationProspective Evaluation of Operating Characteristics of Prostate Cancer Detection Biomarkers
Liang, Y., Ankerst, D. P., Ketchum, N. S., Ercole, B., Shah, G. V., Shaughnessy, J. D., Leach, R. J., & Thompson, I. M. (2010) · The Journal of Urology · doi:10.1016/j.juro.2010.08.088 · Cited by 28 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
A simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174 · see citing works
Self-citationA guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657 · see citing works
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Interpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503 · see citing works
Self-citationToward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety
Huckvale, K., Venkatesh, S., & Christensen, H. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0166-1 · Cited by 433 · see citing works
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400 · see citing works
Fusar-Poli, P., Rutigliano, G., Stahl, D., Davies, C., Bonoldi, I., Reilly, T., & McGuire, P. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0284 · Cited by 246 · see citing works
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0531-0 · Cited by 234 · see citing works
Evaluating Machine Learning Models and Their Diagnostic Value
Varoquaux, G., & Colliot, O. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_20 · Cited by 242 · see citing works
Bellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197 · see citing works
Three myths about risk thresholds for prediction models
on behalf of the Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Wynants, L., van Smeden, M., McLernon, D. J., Timmerman, D., Steyerberg, E. W., & Van Calster, B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1425-3 · Cited by 213 · see citing works
Self-citationMinimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197 · see citing works
Self-citationCollins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6986 · Cited by 185 · see citing works
Use of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160 · see citing works
“Balancing sensitivity with false alarms and lead time for alerts remains a persistent challenge, and the optimal balance varies by use case [41].”
BackgroundVerbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.028 · Cited by 138 · see citing works
Self-citationValidation of prediction models in the presence of competing risks: a guide through modern methods
van Geloven, N., Giardiello, D., Bonneville, E. F., Teece, L., Ramspek, C. L., van Smeden, M., Snell, K. I. E., van Calster, B., Pohar-Perme, M., Riley, R. D., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069249 · Cited by 106 · see citing works
Self-citationFusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., McGuire, P., & Osborn, D. (2019) · Schizophrenia Bulletin · doi:10.1093/schbul/sby070 · Cited by 95 · see citing works
Fusar-Poli, P., Sullivan, S. A., Shah, J. L., & Uhlhaas, P. J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00774 · Cited by 92 · see citing works
Predicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71 · see citing works
Artificial Intelligence–Enabled Prediction of Heart Failure Risk From Single-Lead Electrocardiograms
Dhingra, L. S., Aminorroaya, A., Pedroso, A. F., Khunte, A., Sangha, V., McIntyre, D., Chow, C. K., Asselbergs, F. W., Brant, L. C. C., Barreto, S. M., et al. (2025) · JAMA Cardiology · doi:10.1001/jamacardio.2025.0492 · Cited by 65 · see citing works
Raket, L. L., Jaskolowski, J., Kinon, B. J., Brasen, J. C., Jönsson, L., Wehnert, A., & Fusar-Poli, P. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30024-8 · Cited by 60 · see citing works
Deep Learning to Estimate Cardiovascular Risk From Chest Radiographs
Weiss, J., Raghu, V. K., Paruchuri, K., Zinzuwadia, A., Natarajan, P., Aerts, H. J. W. L., & Lu, M. T. (2024) · Annals of Internal Medicine · doi:10.7326/m23-1898 · Cited by 50 · see citing works
Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go
Kessler, R. C. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1238 · Cited by 41 · see citing works
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42 · see citing works
Self-citationUbink, I., Elias, S. G., Moelans, C. B., Laclé, M. M., van Grevenstein, W. M. U., van Diest, P. J., Borel Rinkes, I. H. M., & Kranenburg, O. (2017) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djw303 · Cited by 40 · see citing works
Pfohl, S., Xu, Y., Foryciarz, A., Ignatiadis, N., Genkins, J., & Shah, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533166 · Cited by 30 · see citing works
Eriksson, L. S. E., Epstein, E., Testa, A. C., Fischerova, D., Valentin, L., Sladkevicius, P., Franchi, D., Frühauf, F., Fruscio, R., Haak, L. A., et al. (2020) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.21950 · Cited by 24 · see citing works
Self-citationClinical prediction models using machine learning in oncology: challenges and recommendations
Collins, G. S., Chester-Jones, M., Gerry, S., Ma, J., Matos, J., Sehjal, J., Tsegaye, B., & Dhiman, P. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000914 · Cited by 27 · see citing works
Assessing the clinical impact of risk models for opting out of treatment
Kerr, K. F., Brown, M. D., Marsh, T. L., & Janes, H. (2019) · Medical Decision Making · doi:10.1177/0272989x18819479 · Cited by 23 · see citing works
Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference
Rhon, D. I., Teyhen, D. S., Collins, G. S., & Bullock, G. S. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2022-001388 · Cited by 20 · see citing works
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19 · see citing works
Self-citationLandolfo, C., Ceusters, J., Valentin, L., Froyman, W., Van Gorp, T., Heremans, R., Baert, T., Wouters, R., Vankerckhoven, A., Van Rompuy, A.-S., et al. (2024) · British Journal of Cancer · doi:10.1038/s41416-024-02578-x · Cited by 14 · see citing works
Self-citationTeuho, J., Schultz, J., Klén, R., Juarez-Orozco, L. E., Knuuti, J., Saraste, A., Ono, N., & Kanaya, S. (2024) · Journal of Nuclear Cardiology · doi:10.1016/j.nuclcard.2024.101889 · Cited by 7 · see citing works
Tang, Q., Deng, K., Cui, Y., Liu, H., & Qian, B. (2026) · International Journal of General Medicine · doi:10.2147/ijgm.s610298
“50,51 Workflow and the Action Gap The transition from retrospective algorithm validation to clinical deployment requires that AI-ECG outputs be incorporated into EHR-based clinical workflows rather than remain stand-alone analytic results.”
- Authors
- Vickers, A. J., Van Calster, B., & Steyerberg, E. W.
- Affiliations
- Memorial Sloan Kettering Cancer Center (US)
- Erasmus MC (NL)
- Corresponding author
- Vickers, Andrew J
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2016-01-25
- Volume
- 352
- Pages
- i6
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC4724785
- OpenAlex
- W2226880313
- Cited by
- 1,163
- References
- 15
- Citations by year
- 2026: 239 · 2025: 144 · 2024: 134 · 2023: 143 · 2022: 126 · 2021: 116 · 2020: 100 · 2019: 74 · 2018: 46 · 2017: 31 · 2016: 9 · 2012: 1
- FWCI
- 54.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, cc-by-nc)
- Lirias (KU Leuven) (submittedVersion, cc-by-nc)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Funders
- Prostate Cancer Foundation (P30 CA008748, P50-CA92629)
- Memorial Sloan-Kettering Cancer Center
- Sidney Kimmel Center for Prostate and Urologic Cancers
- National Cancer Institute (P50 CA92629, R01CA179115, P30CA008748, P50CA092629)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- net benefit, decision curve analysis, clinical prediction model, molecular markers, diagnostic testing, exchange rates, clinical decision-making, benefit-harm trade-off, threshold probability, diagnostic accuracy measures
- MeSH terms
- Decision Support Techniques, Diagnostic Tests, Routine, Humans, Predictive Value of Tests, ROC Curve, Models, Statistical, Biomarkers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
