The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Open accessAbstract
Decision-analytic measures to assess clinical utility of prediction models and diagnostic tests incorporate the relative clinical consequences of true and false positives without the need for external information such as monetary costs. Net Benefit is a commonly used metric that weights the relative consequences in terms of the risk threshold at which a patient would opt for treatment. Theoretical results demonstrate that clinical utility is affected by a model';s calibration, the extent to which estimated risks correspond to observed event rates. We analyzed the effects of different types of miscalibration on Net Benefit and investigated whether and under what circumstances miscalibration can make a model clinically harmful. Clinical harm is defined as a lower Net Benefit compared with classifying all patients as positive or negative by default. We used simulated data to investigate the effect of overestimation, underestimation, overfitting (estimated risks too extreme), and underfitting (estimated risks too close to baseline risk) on Net Benefit for different choices of the risk threshold. In accordance with theory, we observed that miscalibration always reduced Net Benefit. Harm was sometimes observed when models underestimated risk at a threshold below the event rate (as in underestimation and overfitting) or overestimated risk at a threshold above event rate (as in overestimation and overfitting). Underfitting never resulted in a harmful model. The impact of miscalibration decreased with increasing discrimination. Net Benefit was less sensitive to miscalibration for risk thresholds close to the event rate than for other thresholds. We illustrate these findings with examples from the literature and with a case study on testicular cancer diagnosis. Our findings strengthen the importance of obtaining calibrated risk models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In contrast to net benefit, expected cost searches for the decision threshold that minimises cost given the misclassification costs.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
In the citing paper
“Miscalibration of the model might thus be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fi xes the threshold such that miscalibration reduces the net benefit value.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
In the citing paper
“Miscalibration can compromise predictive AI application by leading to systematic overtreatment or undertreatment.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
In the citing paper
“Previous work has shown that miscalibration decreases clinical utility compared to moderate calibration [18].”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
In the citing paper
“it is an important observation that calibration in a weak sense may still result in harmful decisionmaking [18].”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
In the citing paper
“This suggests that moderate calibration prevents the model from becoming harmful [18] but that strong calibration may further enhance the model's clinical utility.”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Criteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M. A., Greenland, P., Arnett, D. K., Ballantyne, C. M., Criqui, M. H., Elkind, M. S. V., Go, A. S., Harrell, F. E., Hong, Y., Howard, B. V., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,146
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362
Research works printed but not linked: 1
- Vickers AJ, 2010, Semin Oncol, 76, 1298
Cited by
Sorted by citations
Items 301–371 of 371
Wells, J. S., Wang, C., Dolgin, K., & Kayyali, R. (2023) · Patient Preference and Adherence · doi:10.2147/ppa.s397424 · Cited by 1
Moons, K. G. M., Altman, D. G., Reitsma, J. B., loannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, P. G. S. (2023) · Вопросы современной педиатрии · doi:10.15690/vsp.v22i2.2557 · Cited by 1
Sud, M. K., Sivaswamy, A., Austin, P. C., Anderson, T. J., Naimark, D. M. J., Farkouh, M. E., Lee, D. S., Roifman, I., Thanassoulis, G., Tu, K., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m23-1345 · Cited by 2
Liao, T., Su, T., Yang, L., Huang, L., & Feng, L.-H. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e39494 · Cited by 1
Feng, L.-H., Su, T., Huang, L., Liao, T., Yang, L., & Wu, L. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1544024 · Cited by 2
Validation of the prostate health index in a predictive model of prostate cancer
Bonet, A., Barrionuevo-González, M., Bajo, A. M., Pulido-Fonseca, L., Polledo, L. E. O., Ruíz, J. C. T., & Sánchez‐Chapado, M. (2017) · Actas Urológicas Españolas (English Edition) · doi:10.1016/j.acuroe.2017.11.006 · Cited by 1
Stahl, J. E. (2014) · Medical Decision Making · doi:10.1177/0272989x14563080 · Cited by 1
Real-world usage diminishes validity of Artificial Intelligence toolsPreprint
Vaid, A., Sawant, A. S., Suárez‐Fariñas, M., Lee, J., Kaul, S. K., Kovatch, P., Freeman, R., Jiang, J. J., Jayaraman, P., Fayad, Z. A., et al. (2022) · medRxiv · doi:10.1101/2022.11.17.22282440 · Cited by 1
Sidorova, J., & Lozano, J. J. (2023) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2023.08.001 · Cited by 1
The Dilemma of Incomplete Reperfusion After Thrombectomy for Ischemic Stroke
Dippel, D. W. J., & Klaveren, D. V. (2024) · Neurology · doi:10.1212/wnl.0000000000209646 · Cited by 1
Agovi, A. M., Thompson, C. T., Meadows, R. J., Lu, Y., & Ojha, R. P. (2025) · medRxiv · doi:10.1101/2025.01.17.25320732 · Cited by 1
Biological aging markers and polygenic risk scores for mortality prediction: a multicohort studyPreprint
Mostafaei, S., Lindh, J., Qin, C., Mak, J. K. L., & Hägg, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9600666/v1 · Cited by 1
Reply to Akinosoglou and Gogos: Sepsis-3 in Community-acquired Pneumonia: How Reliable Is It?
Ranzani, O., & Torres, A. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201708-1598le
“It is important to note that even with certain mis-calibration (6), qSOFA outperformed SIRS in all scenarios for risk-stratification in our study, including the decision-curve analysis.”
MethodZheng, J., Zheng, Y., & Hsu, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01850
“Therefore, model recalibration is critically needed for accurate risk projection in the target population, but it is frequently under-appreciated (Van Calster and Vickers 2015).”
BackgroundWessler, B. S., Nelson, J., Park, J. G., McGinnes, H. L., Upshaw, J. N., CALSTER, B. V., Klaveren, D. V., Steyerberg, E. W., & Kent, D. M. (2021) · medRxiv · doi:10.1101/2021.01.21.21250234
Promotion, Integration, Management and Processing of Critical Inpatients’ Open Big Data Repositories
de la Hoz, M. Á. A. (2020) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.63196
Zhang, S., Sun, S., Zhai, Y., Wang, X., Zhang, Q., Shi, Z., Ge, P., & Zhang, D. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.979014
Biziaev, T., Aktary, M. L., Wang, Q., Chekouo, T., Bhatti, P., Shack, L., Robson, P. J., & Kopciuk, K. A. (2023) · Cancers · doi:10.3390/cancers15143545 · Cited by 1
王, 靖. (2023) · Advances in Clinical Medicine · doi:10.12677/acm.2023.1392108
Yeramosu, T., Krivicich, L. M., Puzzitiello, R. N., Guenthner, G., & Salzler, M. J. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.07.019
Salim, O., Draz, M. S., Bligh, E. R., & Mathieson, C. S. (2024) · Seminars in Spine Surgery · doi:10.1016/j.semss.2024.101132 · Cited by 1
Vorhersage von Psoriasis‐Arthritis bei Patienten mit Plaque‐Psoriasis durch neuartiges Nomogramm
Tan, M., Chen, J., Cheng, J., Hu, J., Hu, K., Yang, J. X., Li, X., Zhang, M., Zhu, W., Liao, L., et al. (2024) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.15446_g
He, X., Long, Y., Huang, J., Liao, J., & Zhang, L. (2025) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2025.110117
Zhang, L., Song, H., Patel, A., Pollack, M. M., & Watson, L. T. (2025) · Research Square · doi:10.21203/rs.3.rs-6787167/v1
“In the clinical setting, this calibration enables reliable risk stratification, supporting decisions such as anticipatory monitoring, medication tapering, or early intervention 27,28 , all of which require nuanced risk awareness rather than binary prediction alone.”
Lin, Y., Yang, L., Chen, H., Zhang, L., Chen, J., Zheng, Y., Tang, F., & Lin, J. (2025) · Research Square · doi:10.21203/rs.3.rs-7079505/v1
Huang, C., Chen, Z., Chen, S., Wu, Z., Zhao, X., & Miao, W. (2025) · Research Square · doi:10.21203/rs.3.rs-7389512/v1
Feng, L.-H., Liang, H., Hu, B., Wei, L., Li, Q., Su, T., Yin, Q., Feng, Y., Huang, L., Liang, H., et al. (2025) · JMIR Medical Informatics · doi:10.2196/78354
Chen, C., Liu, L., Liu, X., & Ya, T. (2025) · Frontiers in Neurology · doi:10.3389/fneur.2025.1625236
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Li, B., Zhao, Y., 蔡望喜, 明安萍, & Li, H. (2021) · Figshare · doi:10.6084/m9.figshare.15567433
van Klaveren, D., Zanos, T. P., Nelson, J., Levy, T. J., Park, J. G., Retel Helmrich, I. R. A., Rietjens, J. A. C., Basile, M. J., Hajizadeh, N., Lingsma, H. F., et al. (2022) · Figshare · doi:10.6084/m9.figshare.21622824
Li, B., Zhao, Y., 蔡望喜, 明安萍, & Li, H. (2021) · Figshare · doi:10.6084/m9.figshare.15567343
Zhong, X., Li, L., Jiang, H., Yin, J., Lu, B., Han, W. G., Li, J., & Zhang, J. (2020) · Figshare · doi:10.6084/m9.figshare.12904778
Plaisy, M. K., Riebensahm, C., Moreira, R., Samala, N., Borse, R. T., Kuniholm, M. H., Minga, A. K., Wandeler, G., López-Íñiguez, Á., Michael, D., et al. (2025) · AIDS · doi:10.1097/qad.0000000000004434
Yang, C., Ma, H., Zeng, X., Yang, J., Xiao, B., Tan, R., Liu, Y., & Zeng, Q. (2026) · Clinical and Applied Thrombosis/Hemostasis · doi:10.1177/10760296251408357
Prediction models for primary graft dysfunction after lung transplantation: Systematic review
Zhao, R., Bu, X., & fan, R. (2026) · Transplantation Reviews · doi:10.1016/j.trre.2026.101017
Zhan, Z., Cao, W., Huang, S., Gui, X., & Li, J. (2026) · Journal of International Medical Research · doi:10.1177/03000605261431643
Multi-omics gut microbiome signatures for treat-to-target management in inflammatory bowel disease
Cai, X. T., Yao, Y., Zheng, Y., & Zhao, X. (2026) · Microbiological Research · doi:10.1016/j.micres.2026.128511
Wang, Y., Chen, C., Jiang, G., & Xuan, C. (2026) · Frontiers in Pediatrics · doi:10.3389/fped.2026.1834603
Guillen-Ramirez, H., Lucas, K. L., Wintsch, Y. M., Blatter, T. U., Triep, K., Endrich, O., & Beldi, G. (2026) · Journal of the American Medical Informatics Association · doi:10.64898/2026.06.15.26355656
Yu, H., Li, P., Gong, S., Zhao, J., & Jiang, K. (2026) · Medicine · doi:10.1097/md.0000000000049606
Chen, X., Wang, L., Qin, W., Yang, M., Yan, Y., Luo, X. P., & Yuan, X. (2026) · Digital Health · doi:10.1177/20552076261466804
Huang, Y.-C., Huang, S.-M., Yeh, J.-H., & Tu, S.-J. (2026) · Strahlentherapie und Onkologie (Print) · doi:10.1007/s00066-026-02569-2
Gao, Y., Liu, C. P., Xiong, X., Yang, C., Tang, S., Wan, J., Chen, Q., Jin, L., Wang, L. Q., Xia, H., et al. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1825792
Kuijten, R. H., Groot, T. M., van Weezenbeek, M. A., Doornberg, J. N., Verlaan, J., & Groot, O. Q. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112438
Yoon, H., Lee, H., & Lee, H. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02995-7
Schöpper, I., Dohna, M., & Bansmann, P. M. (2026) · Insights into Imaging · doi:10.1186/s13244-026-02371-9
Zhong, X., Wang, Y. F., Chai, W., Lin, Y., Zheng, N., & Huang, J. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1875678
Sposito, A. C. (2026) · Journal of the American Heart Association · doi:10.1161/jaha.125.050232
Yeramosu, T., Khorram, R., Phal, A., Beredjiklian, P. K., Eygendaal, D. E., Colaris, J. W., Doornberg, J. N., & Kachooei, A. R. (2026) · Journal of Hand Surgery Global Online · doi:10.1016/j.jhsg.2026.101118
Liu, W., Li, S., Zeng, Y., Dai, W., Peng, J., Wang, X., Chen, Q., Li, W., Du, D., Lu, S., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10772120/v1
Zhao, Q., Hu, W., Xi, X., Wu, L., Hu, S., Liu, Y., & Chen, Y. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1907985
Nguyen, N. H., Lý, K. N., & Le, N. Q. K. (2026) · Computers · doi:10.3390/computers15090626
Patil, D. D., & Kamlakar, M. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11689059
Multimodal prediction of MCI-to-dementia conversion over a two-year window
Simkhada, B., Liang, T. Y., Cui, X., Adeyosoye, M., Simkhada, B., Cabrerizo, M., Cid, R. C., Burke, S. L., Barreto, A., Rishe, N., et al. (2026) · GeroScience · doi:10.1007/s11357-026-02551-x
Calibration Shrinkage and Microvascular Quantification in a Renal Fibrosis Nomogram
Shao, J., & Gu, J. (2026) · Journal of Ultrasound in Medicine · doi:10.1002/jum.70492
Gokhale, S., Belsti, Y., Taylor, D., Zeps, N., Nagpurkar, M., Teede, H., & Enticott, J. (2026) · Archives of Gerontology and Geriatrics Plus · doi:10.1016/j.aggp.2026.100338
Predicting Inflammatory Adaptation to Exercise via Circulating miR-146a and Machine Learning
Mourgan, F. H. A. E. (2026) · Biomedical and Biotechnology Research Journal (BBRJ) · doi:10.4103/bbrj.bbrj_83_26
What would universal screening for cerebral palsy be worth? The case for net benefit analysis
Briganti, G., & McNally, R. J. (2026) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.70553
Kwon, H., Kim, J., Yun, H., & Park, S. Y. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106740
From discovery to proof: a Translation Evidence Map for precision psychiatry
Baune, B. T. (2026) · The Lancet Psychiatry · doi:10.1016/s2215-0366(26)00267-1
Why decision curves cross treat-all and treat-none: A PPV and calibration perspective
Hoessly, L. (2026) · Global Epidemiology · doi:10.1016/j.gloepi.2026.100290
島内, 淳., Nishino, T., Shigeta, K., Mase, H., Yamamoto, T., Yokobori, S., & Ishikawa, M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0360511
Xiao, F., Guo, H., Wang, L., Qu, Y., & Dai, C. (2026) · Medicine · doi:10.1097/md.0000000000051089
Li, J., Zhu, T., & Kashiwazaki, H. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15208138
Metadata
- Authors
- Van Calster, B., & Vickers, A. J.
- Affiliations
- KU Leuven, Department of Development and Regeneration, Leuven, Belgium (BVC)
- Memorial Sloan Kettering Cancer Center (US)
- ORCID
- Vickers, Andrew J.
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2014-08-25
- Volume
- 35
- Issue
- 2
- Pages
- 162-169
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- OpenAlex
- W2054193584
- Cited by
- 371
- References
- 29
- Citations by year
- 2026: 44 · 2025: 39 · 2024: 48 · 2023: 40 · 2022: 57 · 2021: 40 · 2020: 31 · 2019: 29 · 2018: 10 · 2017: 19 · 2016: 7 · 2015: 6 · 2014: 1
- FWCI
- 11.61
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Medical Decision Making (publishedVersion)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Fonds Wetenschappelijk Onderzoek
- Vlaamse regering
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model calibration, net benefit, clinical utility, calibration error, overfitting, underfitting, risk overestimation, risk underestimation, risk thresholds, model discrimination, testicular cancer diagnosis
- MeSH terms
- Computer Simulation, Decision Support Techniques, Humans, Male, Models, Theoretical, Norway, Predictive Value of Tests, ROC Curve, Testicular Neoplasms, Bias, Linear Models, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review