Cited by 371Statistical Methods in EpidemiologyClinical Reasoning and Diagnostic SkillsMeta-analysis and systematic reviews

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance

Open access

Van Calster, Ben, Vickers, Andrew J.·2015·Medical Decision Making

DOI 10.1177/0272989x14547233

Open this paper in CoChat — free account Open PDF

Abstract

Decision-analytic measures to assess clinical utility of prediction models and diagnostic tests incorporate the relative clinical consequences of true and false positives without the need for external information such as monetary costs. Net Benefit is a commonly used metric that weights the relative consequences in terms of the risk threshold at which a patient would opt for treatment. Theoretical results demonstrate that clinical utility is affected by a model';s calibration, the extent to which estimated risks correspond to observed event rates. We analyzed the effects of different types of miscalibration on Net Benefit and investigated whether and under what circumstances miscalibration can make a model clinically harmful. Clinical harm is defined as a lower Net Benefit compared with classifying all patients as positive or negative by default. We used simulated data to investigate the effect of overestimation, underestimation, overfitting (estimated risks too extreme), and underfitting (estimated risks too close to baseline risk) on Net Benefit for different choices of the risk threshold. In accordance with theory, we observed that miscalibration always reduced Net Benefit. Harm was sometimes observed when models underestimated risk at a threshold below the event rate (as in underestimation and overfitting) or overestimated risk at a threshold above event rate (as in overestimation and overfitting). Underfitting never resulted in a harmful model. The impact of miscalibration decreased with increasing discrimination. Net Benefit was less sensitive to miscalibration for risk thresholds close to the event rate than for other thresholds. We illustrate these findings with examples from the literature and with a case study on testicular cancer diagnosis. Our findings strengthen the importance of obtaining calibrated risk models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1177/0272989x14547233

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,483
Clinical Prediction Models
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Criteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M. A., Greenland, P., Arnett, D. K., Ballantyne, C. M., Criqui, M. H., Elkind, M. S. V., Go, A. S., Harrell, F. E., Hong, Y., Howard, B. V., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,146
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507
Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362

Show all 29 references

Research works printed but not linked: 1

Vickers AJ, 2010, Semin Oncol, 76, 1298

Sorted by citations

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citation
Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer
Huang, Y., Liang, C., He, L., Tian, J., Liang, C., Chen, X., Ma, Z., & Liu, Z. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.9128 · Cited by 1,753
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
CALSTER, B. V., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y. J., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,306
Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163
A simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514
Interpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503
The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 427
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312
“Errors in individual predicted probabilities can lead to overconfidence, inappropriate treatment, or poor allocation of resources.(1,8,39,40) Although the importance of calibration is emphasized in current recommendations, many studies focus solely on discrimination, and the few existing evaluations of calibration dri…”
Background
The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models
Austin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286
Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 271
Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Big Data and Predictive Analytics
Shah, N. D., Steyerberg, E. W., & Kent, D. M. (2018) · JAMA · doi:10.1001/jama.2018.5602 · Cited by 244
A systematic review of machine learning models for predicting outcomes of stroke with structured data
Wang, W., Kiik, M., Peek, N. B., Ćurčin, V., Marshall, I. J., Rudd, A. G., Wang, Y., Douiri, A., Wolfe, C. D. A., & Bray, B. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234722 · Cited by 205
Quantifying the added value of new biomarkers: how and how not
Cook, N. R. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0037-2 · Cited by 198
New Sepsis Definition (Sepsis-3) and Community-acquired Pneumonia Mortality. A Validation and Clinical Decision-Making Study
Ranzani, O., Prina, E., Menéndez, R., Ceccato, A., Cillóniz, C., Méndez, R., Gabarrús, A., Barbeta, E., Bassi, G. L., Ferrer, M., et al. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201611-2262oc · Cited by 182
Poor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 173
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160
Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models
McLernon, D. J., Giardiello, D., CALSTER, B. V., Wynants, L., van Geloven, N., van Smeden, M., Therneau, T. M., Steyerberg, E. W., Initiative, T. G. 6. A. 8. O. T. S., McLernon, D. J., et al. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0844 · Cited by 166
Prognosis in Moderate and Severe Traumatic Brain Injury: A Systematic Review of Contemporary Models and Validation Studies
Dijkland, S. A., Foks, K. A., Polinder, S., Dippel, D. W. J., Maas, A. I. R., Lingsma, H. F., & Steyerberg, E. W. (2019) · Journal of Neurotrauma · doi:10.1089/neu.2019.6401 · Cited by 164
Understanding overfitting in random forest for probability estimation: a visualization and simulation study
Barreñada, L., Dhiman, P., Timmerman, D., Boulesteix, A., & CALSTER, B. V. (2024) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00177-1 · Cited by 157
An artificial intelligence model to predict hepatocellular carcinoma risk in Korean and Caucasian patients with chronic hepatitis B
Kim, H. Y., Lampertico, P., Nam, J. Y., Lee, H., Kim, S. U., Sinn, D. H., Seo, Y. S., Lee, H. A., Park, S. Y., Lim, Y., et al. (2021) · Journal of Hepatology · doi:10.1016/j.jhep.2021.09.025 · Cited by 153
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models
Assel, M., Sjoberg, D., & Vickers, A. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0020-3 · Cited by 133
Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease
van Smeden, M., Heinze, G., CALSTER, B. V., Asselbergs, F. W., Vardas, P. E., Bruining, N., de Jaegere, P. P. T., Moore, J. H., Denaxas, S., Boulesteix, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac238 · Cited by 155
Synthesis optimization and adsorption modeling of biochar for pollutant removal via machine learning
Zhang, W., Chen, R., Li, J., Huang, T., Wu, B., Ma, J., Wen, Q., Tan, J., & Huang, W. (2023) · Biochar · doi:10.1007/s42773-023-00225-x · Cited by 126
External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125
Comparison between logistic regression and machine learning algorithms on survival prediction of traumatic brain injuries
Feng, J., Wang, Y., Peng, J., Sun, M., Zeng, J., & Jiang, H. (2019) · Journal of Critical Care · doi:10.1016/j.jcrc.2019.08.010 · Cited by 116
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Machine learning versus physicians’ prediction of acute kidney injury in critically ill adults: a prospective evaluation of the AKIpredictor
Flechet, M., Falini, S., Bonetti, C., Güiza, F., Schetz, M., den Berghe, G. V., & Meyfroidt, G. (2019) · Critical Care · doi:10.1186/s13054-019-2563-x · Cited by 105
Selecting High-Risk Individuals for Lung Cancer Screening: A Prospective Evaluation of Existing Risk Models and Eligibility Criteria in the German EPIC Cohort
Li, K., Hüsing, A., Sookthai, D., Bergmann, M. M., Boeing, H., Becker, N., & Kaaks, R. (2015) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-14-0424 · Cited by 100
Optimizing Clinical Decision Making with Decision Curve Analysis: Insights for Clinical Investigators
Piovani, D., Sokou, R., Tsantes, A. G., Vitello, A. S., & Bonovas, S. (2023) · Healthcare · doi:10.3390/healthcare11162244 · Cited by 113
Artificial Intelligence Based Algorithms for Prostate Cancer Classification and Detection on Magnetic Resonance Imaging: A Narrative Review
Twilt, J. J., van Leeuwen, K. G., Huisman, H. J., Fütterer, J. J., & de Rooij, M. (2021) · Diagnostics · doi:10.3390/diagnostics11060959 · Cited by 105
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 99
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93
Avoiding Unnecessary Biopsy: MRI-based Risk Models versus a PI-RADS and PSA Density Strategy for Clinically Significant Prostate Cancer
Deniffel, D., Healy, G. M., Dong, X., Ghai, S., Salinas-Miranda, E., Fleshner, N. E., Hamilton, R. J., Kulkarni, G. S., Toi, A., van der Kwast, T., et al. (2021) · Radiology · doi:10.1148/radiol.2021204112 · Cited by 86
External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83
Artificial intelligence in orthopaedics: false hope or not? A narrative review along the line of Gartner’s hype cycle
Oosterhoff, J. H. F., & Doornberg, J. N. (2020) · EFORT Open Reviews · doi:10.1302/2058-5241.5.190092 · Cited by 84
Wide range of applications for machine-learning prediction models in orthopedic surgical outcome: a systematic review
Ogink, P. T., Groot, O. Q., Karhade, A. V., Bongers, M. E. R., Öner, F. C., Verlaan, J., & Schwab, J. H. (2021) · Acta Orthopaedica · doi:10.1080/17453674.2021.1932928 · Cited by 77
External Validation of the DAPT Score in a Nationwide Population
Ueda, P., Jernberg, T., James, S. K., Alfredsson, J., Erlinge, D., Ömerovic, E., Persson, J., Ravn‐Fischer, A., Tornvall, P., Svennblad, B., et al. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.06.023 · Cited by 74
Individualized discrimination of tumor recurrence from radiation necrosis in glioma patients using an integrated radiomics-based model
Wang, K., Qiao, Z., Zhao, X., Li, X., Wang, X., Wu, T., Chen, Z., Fan, D., Chen, Q., & Ai, L. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04604-0 · Cited by 71
Predictive Modeling for Metabolomics Data
Ghosh, T., Zhang, W., Ghosh, D., & Kechris, K. J. (2020) · Methods in molecular biology · doi:10.1007/978-1-0716-0239-3_16 · Cited by 74
Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database
Singh, H., Mhasawade, V., & Chunara, R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000023 · Cited by 67
Development and Validation of an 18-Gene Urine Test for High-Grade Prostate Cancer
Tosoian, J. J., Zhang, Y., Xiao, L., Xie, C., Samora, N. L., Niknafs, Y. S., Chopra, Z., Siddiqui, J., Zheng, H., Herron, G. C., et al. (2024) · JAMA Oncology · doi:10.1001/jamaoncol.2024.0455 · Cited by 68
Machine Learning as a Support for the Diagnosis of Type 2 Diabetes
Agliata, A., Giordano, D., Bardozzo, F., Bottiglieri, S., Facchiano, A. M., & Tagliaferri, R. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24076775 · Cited by 68
Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 62
Novel Machine Learning Approach for the Prediction of Hernia Recurrence, Surgical Complication, and 30-Day Readmission after Abdominal Wall Reconstruction
Hassan, A. M., Lu, S.-C., Asaad, M., Liu, J., Offodile, A. C., Sidey-Gibbons, C. J., & Butler, C. E. (2022) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000000141 · Cited by 59
Neural network and logistic regression diagnostic prediction models for giant cell arteritis: development and validation
Ing, E. B., Miller, N. R., Nguyen, A. M., Su, W., Bursztyn, L. L. C. D., Poole, M., Kansal, V., Toren, A., Albreiki, D., Mouhanna, J. G., et al. (2019) · Clinical ophthalmology · doi:10.2147/opth.s193460 · Cited by 57
“37 On DCA, our prediction model is better than “biopsy all” or “biopsy none” strategies for a useful and large range of decision cut points.”
Does the SORG Algorithm Generalize to a Contemporary Cohort of Patients with Spinal Metastases on External Validation?
Bongers, M., Karhade, A., Villavieja, J., Groot, O., Bilsky, M., Laufer, I., & Schwab, J. (2020) · The spine journal · doi:10.1016/j.spinee.2020.05.003 · Cited by 51
The Harms of Class Imbalance Corrections for Machine Learning Based Prediction Models: A Simulation Study
Carriero, A., Luijken, K., de Hond, A., Moons, K. G. M., CALSTER, B. V., & van Smeden, M. (2025) · Statistics in Medicine · doi:10.1002/sim.10320 · Cited by 56
Prediction of Postoperative Delirium in Geriatric Hip Fracture Patients: A Clinical Prediction Model Using Machine Learning Algorithms
Oosterhoff, J. H. F., Karhade, A. V., Oberai, T., Franco‐Garcia, E., Doornberg, J. N., & Schwab, J. H. (2021) · Geriatric Orthopaedic Surgery & Rehabilitation · doi:10.1177/21514593211062277 · Cited by 57
Feasibility of Machine Learning and Logistic Regression Algorithms to Predict Outcome in Orthopaedic Trauma Surgery
Oosterhoff, J. H. F., Gravesteijn, B. Y., Karhade, A. V., Jaarsma, R. L., Kerkhoffs, G. M. M. J., Ring, D., Schwab, J. H., Steyerberg, E. W., & Doornberg, J. N. (2021) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00341 · Cited by 52
Direct Risk Assessment From Myocardial Perfusion Imaging Using Explainable Deep Learning
Singh, A., Miller, R. J. H., Otaki, Y., Kavanagh, P., Hauser, M. T., Tzolos, E., Kwieciński, J., Kriekinge, S. D. V., Wei, C.-C., Sharir, T., et al. (2022) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2022.07.017 · Cited by 51
Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality.
Davis, S. E., Lasko, T. A., Chen, G., & Matheny, M. E. (2017) · PubMed Central · Cited by 50
Impact of Malnutrition on Long-Term Mortality in Elderly Patients with Acute Myocardial Infarction
Komici, K., Vitale, D. F., Mancini, A., Bencivenga, L., Conte, M., Provenzano, S., Grieco, F. V., Visaggi, L., Ronga, I., Cittadini, A., et al. (2019) · Nutrients · doi:10.3390/nu11020224 · Cited by 50
Serum Prostate-Specific Antigen Testing for Early Detection of Prostate Cancer: Managing the Gap between Clinical and Laboratory Practice
Ferraro, S., Bussetti, M., & Panteghini, M. (2021) · Clinical Chemistry · doi:10.1093/clinchem/hvab002 · Cited by 47
Decision analysis and reinforcement learning in surgical decision-making
Loftus, T. J., Filiberto, A. C., Li, Y., Balch, J. A., Cook, A. C., Tighe, P. J., Efron, P. A., Upchurch, G. R., Rashidi, P., Li, X., et al. (2020) · Surgery · doi:10.1016/j.surg.2020.04.049 · Cited by 49
A comprehensive review on efficient artificial intelligence models for classification of abnormal cardiac rhythms using electrocardiograms
Gupta, U., Paluru, N., Nankani, D., Kulkarni, K., & Awasthi, N. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26787 · Cited by 53
Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings
Vaid, A., Sawant, A. S., Suárez‐Fariñas, M., Lee, J., Kaul, S. K., Kovatch, P., Freeman, R., Jiang, J. J., Jayaraman, P., Fayad, Z. A., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m23-0949 · Cited by 50
Random‐effects meta‐analysis of the clinical utility of tests and prediction models
Wynants, L., Riley, R. D., Timmerman, D., & CALSTER, B. V. (2018) · Statistics in Medicine · doi:10.1002/sim.7653 · Cited by 44
Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information: An XGBoost Algorithm Analysis
Yun, H., Choi, J., & Park, J. H. (2021) · JMIR Medical Informatics · doi:10.2196/30770 · Cited by 45
Development and Assessment of Machine Learning Models for Individualized Risk Assessment of Mastectomy Skin Flap Necrosis
Hassan, A. M., Biaggi, A. P., Asaad, M., Andejani, D. F., Liu, J., Offodile2nd, A. C., Selber, J. C., & Butler, C. E. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005386 · Cited by 38
Automatic correction of performance drift under acquisition shift in medical image classification
Roschewitz, M., Khara, G. S., Yearsley, J., Sharma, N., James, J. J., Ambrózay, É., Heroux, A., Kecskemethy, P. D., Rijken, T. M., & Glocker, B. (2023) · Nature Communications · doi:10.1038/s41467-023-42396-y · Cited by 49
From bytes to bedside: a systematic review on the use and readiness of artificial intelligence in the neonatal and pediatric intensive care unit
Schouten, J., Kalden, M. A. C. M., van Twist, E., Reiss, I. K. M., Gommers, D. A. M. P. J., van Genderen, M. E., & Taal, H. R. (2024) · Intensive Care Medicine · doi:10.1007/s00134-024-07629-8 · Cited by 46
An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes
Estiri, H., Strasser, Z. H., Rashidian, S., Klann, J. G., Wagholikar, K. B., McCoy, T. H., & Murphy, S. N. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac070 · Cited by 43
Mitigating bias in deep learning for diagnosis of coronary artery disease from myocardial perfusion SPECT images
Miller, R. J. H., Singh, A., Otaki, Y., Tamarappoo, B. K., Kavanagh, P., Parekh, T., Hu, L.-H., Gransar, H., Sharir, T., Einstein, A. J., et al. (2022) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-022-05972-w · Cited by 40
Semi-supervised learning to improve generalizability of risk prediction models
Chi, S., Li, X., Tian, Y., Li, J., Kong, X., Ding, K., Weng, C., & Li, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103117 · Cited by 40
A novel risk-predicted nomogram for sepsis associated-acute kidney injury among critically ill patients
Yang, S., Su, T., Huang, L., Feng, L.-H., & Liao, T. (2021) · BMC Nephrology · doi:10.1186/s12882-021-02379-x · Cited by 37
Developing and validating clinical prediction models in hepatology – An overview for clinicians
Strandberg, R., Jepsen, P., & Hagström, H. (2024) · Journal of Hepatology · doi:10.1016/j.jhep.2024.03.030 · Cited by 38
A Machine Learning-Based Prediction Model for Cardiovascular Risk in Women With Preeclampsia
Wang, G., Zhang, Y., Li, S., Zhang, J., Jiang, D., Li, X., Li, Y., & Du, J. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.736491 · Cited by 40
Development and internal validation of a clinical prediction model using machine learning algorithms for 90 day and 2 year mortality in femoral neck fracture patients aged 65 years or above
Oosterhoff, J. H. F., Savelberg, A. B. M. C., Karhade, A. V., Gravesteijn, B. Y., Doornberg, J. N., Schwab, J. H., & Heng, M. M. (2022) · European Journal of Trauma and Emergency Surgery · doi:10.1007/s00068-022-01981-4 · Cited by 36
Efficacy and Applications of Artificial Intelligence and Machine Learning Analyses in Total Joint Arthroplasty
Polce, E. M., Kunze, K. N., Dooley, M. S., Piuzzi, N. S., Boettner, F., & Sculco, P. K. (2022) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00717 · Cited by 34
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33
Machine learning for predicting the treatment effect of orthokeratology in children
Fang, J., Zheng, Y., Mou, H., Shi, M., Yu, W., & Du, C. (2023) · Frontiers in Pediatrics · doi:10.3389/fped.2022.1057863 · Cited by 33
Ethnicity-dependent performance of the Global Registry of Acute Coronary Events risk score for prediction of non-ST-segment elevation myocardial infarction in-hospital mortality: nationwide cohort study
Moledina, S. M., Kontopantelis, E., Wijeysundera, H. C., Banerjee, S., Spall, H. G. C. V., Gale, C. P., Shah, B. N., Mohamed, M., Weston, C. F. M., Shoaib, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac052 · Cited by 33
A LASSO-based survival prediction model for patients with synchronous colorectal carcinomas based on SEER
Xu, Y., Wang, X., Huang, Y., Ye, D., & Chi, P. (2022) · Translational Cancer Research · doi:10.21037/tcr-20-1860 · Cited by 35
Reconsidering lactate as a sepsis risk biomarker
Moran, J. L., & Santamaria, J. D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185320 · Cited by 32
Are prediction models for vaginal birth after cesarean accurate?
Harris, B. S., Heine, R. P., Park, J., Faurot, K. R., Hopkins, M. K., Rivara, A., Kemeny, H. R., Grotegut, C. A., & Jelovsek, J. E. (2019) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.01.232 · Cited by 32
Using decision curve analysis to benchmark performance of a magnetic resonance imaging–based deep learning model for prostate cancer risk assessment
Deniffel, D., Abraham, N., Namdar, K., Dong, X., Salinas, E., Milot, L. M., Khalvati, F., & Haider, M. A. (2020) · European Radiology · doi:10.1007/s00330-020-07030-1 · Cited by 33
Validation of VIRSTA and Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT) Scores to Determine the Priority of Echocardiography in Patients With Staphylococcus aureus Bacteremia
Peinado-Acevedo, J. S., Hurtado-Guerra, J. J., Hincapié, C., Abad, J. M., Uribe-Delgado, J. R., Giraldo-Ramírez, S., Lengerke-Diaz, P. A., & Jaimes, F. A. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1844 · Cited by 31
Integrative nomogram of intratumoral, peritumoral, and lymph node radiomic features for prediction of lymph node metastasis in cT1N0M0 lung adenocarcinomas
Das, S. K., Fang, K.-W., Xu, L., Li, B., Zhang, X., & Yang, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-90367-4 · Cited by 31
Predicting radiocephalic arteriovenous fistula success with machine learning
Heindel, P. V., Dey, T., Feliz, J. D., Hentschel, D. M., Bhatt, D. L., Al-Omran, M. A., Belkin, M., Ozaki, C. K., & Hussain, M. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00710-w · Cited by 31
Risk calculators for the detection of prostate cancer: a systematic review
Denijs, F., van Harten, M. J., Meenderink, J. J. L., Leenen, R. C. A., Remmers, S., Venderbos, L. D. F., van den Bergh, R. C. N., Beyer, K., & Roobol, M. J. (2024) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-024-00852-w · Cited by 35
Clinical risk reclassification at 10 years
Cook, N. R., Demler, O., & Paynter, N. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7340 · Cited by 29
Risk-based prediction model for selecting eligible population for lung cancer screening among ever smokers in Korea
Park, B., Kim, Y., Lee, J., Lee, N., & Jang, S. H. (2021) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-566 · Cited by 28
Development, validation and clinical utility of a risk prediction model for adverse pregnancy outcomes in women with gestational diabetes: The PeRSonal GDM model
Cooray, S. D., Boyle, J. A., Soldatos, G., Allotey, J., Wang, H., Fernandez-Felix, B. M., Zamora, J., Thangaratinam, S., & Teede, H. J. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101637 · Cited by 29
Cervical spine osteoradionecrosis or bone metastasis after radiotherapy for nasopharyngeal carcinoma? The MRI-based radiomics for characterization
Zhong, X. Z., Li, L., Jiang, H., Yin, J., Lu, B., Han, W. G., Li, J., & Zhang, J. (2020) · BMC Medical Imaging · doi:10.1186/s12880-020-00502-2 · Cited by 29
Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis
Koola, J., Ho, S. B., Cao, A., Chen, G., Perkins, A. M., Davis, S. E., & Matheny, M. E. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05826-w · Cited by 31
Developing well-calibrated illness severity scores for decision support in the critically ill
Cosgriff, C. V., Celi, L. A., Ko, S. Q., Sundaresan, T. G., de la Hoz, M. Á. A., Kaufman, A. R., Stone, D. J., Badawi, O., & Deliberato, R. O. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0153-6 · Cited by 27
Development and external validation of predictive algorithms for six-week mortality in spinal metastasis using 4,304 patients from five institutions
Karhade, A. V., Fenn, B. P., Groot, O. Q., Shah, A. A., Yen, H., Bilsky, M. H., Hu, M., Laufer, I., Park, D. Y., Sciubba, D. M., et al. (2022) · The Spine Journal · doi:10.1016/j.spinee.2022.07.089 · Cited by 26
How Does the Skeletal Oncology Research Group Algorithm’s Prediction of 5-year Survival in Patients with Chondrosarcoma Perform on International Validation?
Bongers, M. E. R., Karhade, A. V., Setola, E., Gambarotti, M., Groot, O. Q., Erdoğan, K. E., Picci, P., Donati, D. M., Schwab, J. H., & Palmerini, E. (2020) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000001305 · Cited by 23
Recalibration Methods for Improved Clinical Utility of Risk Scores
Mishra, A., McClelland, R. L., Inoue, L. Y. T., & Kerr, K. F. (2021) · Medical Decision Making · doi:10.1177/0272989x211044697 · Cited by 23
Pressure injuries in elderly with acute myocardial infarction
Komici, K., Vitale, D. F., Leosco, D., Mancini, A., Corbi, G., Bencivenga, L., Mezzani, A., Trimarco, B., Morisco, C., Ferrara, N., et al. (2017) · Clinical Interventions in Aging · doi:10.2147/cia.s135676 · Cited by 23
Incorporating longitudinal history of risk factors into atherosclerotic cardiovascular disease risk prediction using deep learning
Yu, J., Yang, X., Deng, Y., Krefman, A. E., Pool, L. R., Zhao, L., Mi, X., Ning, H., Wilkins, J. T., Lloyd‐Jones, D. M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-51685-5 · Cited by 22
Validation of PREdiction of DELIRium in ICu patients (PRE‐DELIRIC) among patients in intensive care units: A retrospective cohort study
Liang, S., Chau, J. P. C., Lo, S. H. S., Bai, L., Yao, L., & Choi, K. C. (2020) · Nursing in Critical Care · doi:10.1111/nicc.12550 · Cited by 22

Authors: Van Calster, B., & Vickers, A. J.
Affiliations: KU Leuven, Department of Development and Regeneration, Leuven, Belgium (BVC)
Memorial Sloan Kettering Cancer Center (US)
ORCID: Vickers, Andrew J.
Venue: Medical Decision Making
Publisher: SAGE Publications
Published: 2014-08-25
Volume: 35
Issue: 2
Pages: 162-169
ISSN: 0272-989X (ISSN-L)
1552-681X
DOI: 10.1177/0272989x14547233
OpenAlex: W2054193584
Cited by: 371
References: 29
Citations by year: 2026: 44 · 2025: 39 · 2024: 48 · 2023: 40 · 2022: 57 · 2021: 40 · 2020: 31 · 2019: 29 · 2018: 10 · 2017: 19 · 2016: 7 · 2015: 6 · 2014: 1
FWCI: 11.61
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Medical Decision Making (publishedVersion)
PubMed (publishedVersion)
Lirias (submittedVersion, other-oa)
RePEc: Research Papers in Economics (submittedVersion)
Funders: Fonds Wetenschappelijk Onderzoek
Vlaamse regering
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords: risk prediction model, model calibration, net benefit, clinical utility, calibration error, overfitting, underfitting, risk overestimation, risk underestimation, risk thresholds, model discrimination, testicular cancer diagnosis
MeSH terms: Computer Simulation, Decision Support Techniques, Humans, Male, Models, Theoretical, Norway, Predictive Value of Tests, ROC Curve, Testicular Neoplasms, Bias, Linear Models, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks