Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Open accessAbstract
Decision-analytic measures to assess clinical utility of prediction models and diagnostic tests incorporate the relative clinical consequences of true and false positives without the need for external information such as monetary costs. Net Benefit is a commonly used metric that weights the relative consequences in terms of the risk threshold at which a patient would opt for treatment. Theoretical results demonstrate that clinical utility is affected by a model';s calibration, the extent to which estimated risks correspond to observed event rates. We analyzed the effects of different types of miscalibration on Net Benefit and investigated whether and under what circumstances miscalibration can make a model clinically harmful. Clinical harm is defined as a lower Net Benefit compared with classifying all patients as positive or negative by default. We used simulated data to investigate the effect of overestimation, underestimation, overfitting (estimated risks too extreme), and underfitting (estimated risks too close to baseline risk) on Net Benefit for different choices of the risk threshold. In accordance with theory, we observed that miscalibration always reduced Net Benefit. Harm was sometimes observed when models underestimated risk at a threshold below the event rate (as in underestimation and overfitting) or overestimated risk at a threshold above event rate (as in overestimation and overfitting). Underfitting never resulted in a harmful model. The impact of miscalibration decreased with increasing discrimination. Net Benefit was less sensitive to miscalibration for risk thresholds close to the event rate than for other thresholds. We illustrate these findings with examples from the literature and with a case study on testicular cancer diagnosis. Our findings strengthen the importance of obtaining calibrated risk models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Expected cost searches for the decision threshold that minimises cost given the misclassification costs.
“In contrast to net benefit, expected cost searches for the decision threshold that minimises cost given the misclassification costs.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
Miscalibration of the model might be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fixes the threshold such that miscalibration reduces the net benefit value.
“Miscalibration of the model might thus be reflected in the decision threshold at which expected cost is minimised, whereas net benefit fi xes the threshold such that miscalibration reduces the net benefit value.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
Miscalibration can compromise predictive AI application by leading to systematic overtreatment or undertreatment.
“Miscalibration can compromise predictive AI application by leading to systematic overtreatment or undertreatment.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [89]
Previous work has shown that miscalibration decreases clinical utility compared to moderate calibration.
“Previous work has shown that miscalibration decreases clinical utility compared to moderate calibration [18].”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
Weak calibration may still result in harmful decision-making.
“it is an important observation that calibration in a weak sense may still result in harmful decisionmaking [18].”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
Moderate calibration prevents the model from becoming harmful, but strong calibration may further enhance clinical utility.
“This suggests that moderate calibration prevents the model from becoming harmful [18] but that strong calibration may further enhance the model's clinical utility.”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D’Agostino, R. B., & Steyerberg, E. W. (2010) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,478 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Criteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M., Greenland, P., Arnett, D., Ballantyne, C., Criqui, M., Elkind, M., Go, A., Harrell, F. E. J., Hong, Y.-L., Howard, B., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,153 · see citing works
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507 · see citing works
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362 · see citing works
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · see citing works
Interpreting diagnostic accuracy studies for patient care
Mallett, S., Halligan, S., Thompson, M., Collins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e3999 · Cited by 249 · see citing works
Collins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e4181 · Cited by 245 · see citing works
Vergouwe, Y., Moons, K. G. M., & Steyerberg, E. W. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq223 · Cited by 243 · see citing works
Assessing the incremental value of diagnostic and prognostic markers: a review and illustration
Steyerberg, E. W., Pencina, M. J., Lingsma, H. F., Kattan, M. W., Vickers, A. J., & Van Calster, B. (2012) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2011.02562.x · Cited by 219 · see citing works
Quantifying the Added Value of a Diagnostic Test or Marker
Moons, K. G. M., de Groot, J. A. H., Linnet, K., Reitsma, J. B., & Bossuyt, P. M. M. (2012) · Clinical Chemistry · doi:10.1373/clinchem.2012.182550 · Cited by 152 · see citing works
Using Relative Utility Curves to Evaluate Risk Prediction
Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00592.x · Cited by 142 · see citing works
Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision Making
Localio, A. R., & Goodman, S. N. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-4-201208210-00014 · Cited by 139 · see citing works
Everything you always wanted to know about evaluating prediction models (but were too afraid to ask)
Vickers, A. J., & Cronin, A. M. (2010) · Urology · doi:10.1016/j.urology.2010.06.019 · Cited by 112 · see citing works
Comparing risk prediction models
Collins, G. S., & Moons, K. G. M. (2012) · BMJ · doi:10.1136/bmj.e3186 · Cited by 110 · see citing works
Prospective Multi-Institutional Study Evaluating the Performance of Prostate Cancer Risk Calculators
Nam, R. K., Kattan, M. W., Chin, J. L., Trachtenberg, J., Singal, R. K., Rendon, R. A., Klotz, L. H., Sugar, L., Sherman, C., Izawa, J. I., et al. (2011) · Journal of Clinical Oncology · doi:10.1200/jco.2010.32.6371 · Cited by 94 · see citing works
Vergouwe, Y., Steyerberg, E. W., Foster, R. S., Sleijfer, D. T., Fosså, S. D., Gerl, A., de Wit, R., Roberts, J. T., & Habbema, J. D. F. (2006) · European Urology · doi:10.1016/j.eururo.2006.06.047 · Cited by 85 · see citing works
Greenland, S. (2007) · Statistics in Medicine · doi:10.1002/sim.2995 · Cited by 80 · see citing works
Evaluation of Markers and Risk Prediction Models
CALSTER, B. V., Vickers, A. J., Pencina, M. J., Baker, S. G., Timmerman, D., & Steyerberg, E. W. (2013) · Medical Decision Making · doi:10.1177/0272989x12470757 · Cited by 76 · see citing works
Health Technology Assessment in the Cost-Disutility Plane
Eckermann, S. D., Briggs, A. H., & Willan, A. R. (2008) · Medical Decision Making · doi:10.1177/0272989x07312474 · Cited by 58 · see citing works
Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update
Baker, S. G., Calster, B. V., & Steyerberg, E. W. (2012) · The International Journal of Biostatistics · doi:10.1515/1557-4679.1395 · Cited by 36 · see citing works
Calibration of clinical prediction rules does not just assess bias
Vach, W. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.06.003 · Cited by 26 · see citing works
PEPE, M. S., Fang, J., Feng, Z., Gerds, T. A., & Hilden, J. (2013) · Collection of Biostatistics Research Archive · Cited by 13 · see citing works
CALSTER, B. V., Steyerberg, E. W., D’Agostino, R. B., & Pencina, M. J. (2013) · Medical Decision Making · doi:10.1177/0272989x13513654 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Huang, Y., Liang, C., He, L., Tian, J., Liang, C., Chen, X., Ma, Z., & Liu, Z. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.9128 · Cited by 1,745 · see citing works
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
CALSTER, B. V., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y. J., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,289 · see citing works
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
A simple, step-by-step guide to interpreting decision curve analysis
Vickers, A. J., van Calster, B., & Steyerberg, E. W. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0064-7 · Cited by 1,174 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697 · see citing works
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · see citing works
Interpreting area under the receiver operating characteristic curve
de Hond, A. A. H., Steyerberg, E. W., & van Calster, B. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00188-1 · Cited by 503 · see citing works
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 375 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
“Errors in individual predicted probabilities can lead to overconfidence, inappropriate treatment, or poor allocation of resources.(1,8,39,40) Although the importance of calibration is emphasized in current recommendations, many studies focus solely on discrimination, and the few existing evaluations of calibration dri…”
BackgroundAustin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · see citing works
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286 · see citing works
Cowley, L., Farewell, D. M., Maguire, S., & Kemp, A. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 265 · see citing works
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246 · see citing works
Big Data and Predictive Analytics
Shah, N. D., Steyerberg, E. W., & Kent, D. M. (2018) · JAMA · doi:10.1001/jama.2018.5602 · Cited by 244 · see citing works
Wang, W., Kiik, M., Peek, N. B., Ćurčin, V., Marshall, I. J., Rudd, A. G., Wang, Y., Douiri, A., Wolfe, C. D. A., & Bray, B. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234722 · Cited by 205 · see citing works
Quantifying the added value of new biomarkers: how and how not
Cook, N. R. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0037-2 · Cited by 198 · see citing works
Ranzani, O., Prina, E., Menéndez, R., Ceccato, A., Cillóniz, C., Méndez, R., Gabarrús, A., Barbeta, E., Bassi, G. L., Ferrer, M., et al. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201611-2262oc · Cited by 180 · see citing works
Poor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 166 · see citing works
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · see citing works
McLernon, D. J., Giardiello, D., CALSTER, B. V., Wynants, L., van Geloven, N., van Smeden, M., Therneau, T. M., Steyerberg, E. W., Initiative, T. G. 6. A. 8. O. T. S., McLernon, D. J., et al. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0844 · Cited by 167 · see citing works
Dijkland, S. A., Foks, K. A., Polinder, S., Dippel, D. W. J., Maas, A. I. R., Lingsma, H. F., & Steyerberg, E. W. (2019) · Journal of Neurotrauma · doi:10.1089/neu.2019.6401 · Cited by 164 · see citing works
Barreñada, L., Dhiman, P., Timmerman, D., Boulesteix, A., & CALSTER, B. V. (2024) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00177-1 · Cited by 157 · see citing works
Kim, H. Y., Lampertico, P., Nam, J. Y., Lee, H., Kim, S. U., Sinn, D. H., Seo, Y. S., Lee, H. A., Park, S. Y., Lim, Y., et al. (2021) · Journal of Hepatology · doi:10.1016/j.jhep.2021.09.025 · Cited by 153 · see citing works
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models
Assel, M., Sjoberg, D., & Vickers, A. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0020-3 · Cited by 133 · see citing works
Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease
van Smeden, M., Heinze, G., CALSTER, B. V., Asselbergs, F. W., Vardas, P. E., Bruining, N., de Jaegere, P. P. T., Moore, J. H., Denaxas, S., Boulesteix, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac238 · Cited by 147 · see citing works
Synthesis optimization and adsorption modeling of biochar for pollutant removal via machine learning
Zhang, W., Chen, R., Li, J., Huang, T., Wu, B., Ma, J., Wen, Q., Tan, J., & Huang, W. (2023) · Biochar · doi:10.1007/s42773-023-00225-x · Cited by 125 · see citing works
Snell, K. I. E., Archer, L., Ensor, J., Bonnett, L. J., Debray, T. P. A., Phillips, B., Collins, P. G. S., & Riley, R. D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.011 · Cited by 125 · see citing works
Feng, J., Wang, Y., Peng, J., Sun, M., Zeng, J., & Jiang, H. (2019) · Journal of Critical Care · doi:10.1016/j.jcrc.2019.08.010 · Cited by 113 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Flechet, M., Falini, S., Bonetti, C., Güiza, F., Schetz, M., den Berghe, G. V., & Meyfroidt, G. (2019) · Critical Care · doi:10.1186/s13054-019-2563-x · Cited by 103 · see citing works
Li, K., Hüsing, A., Sookthai, D., Bergmann, M. M., Boeing, H., Becker, N., & Kaaks, R. (2015) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-14-0424 · Cited by 99 · see citing works
Piovani, D., Sokou, R., Tsantes, A. G., Vitello, A. S., & Bonovas, S. (2023) · Healthcare · doi:10.3390/healthcare11162244 · Cited by 99 · see citing works
Twilt, J. J., van Leeuwen, K. G., Huisman, H. J., Fütterer, J. J., & de Rooij, M. (2021) · Diagnostics · doi:10.3390/diagnostics11060959 · Cited by 97 · see citing works
Key steps and common pitfalls in developing and validating risk models
Wynants, L., Collins, P. G. S., & CALSTER, B. V. (2016) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14170 · Cited by 95 · see citing works
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85 · see citing works
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93 · see citing works
Deniffel, D., Healy, G. M., Dong, X., Ghai, S., Salinas-Miranda, E., Fleshner, N. E., Hamilton, R. J., Kulkarni, G. S., Toi, A., van der Kwast, T., et al. (2021) · Radiology · doi:10.1148/radiol.2021204112 · Cited by 85 · see citing works
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H., Gulati, G., Brazil, R., Van Calster, B., van Klaveren, D., Venema, E., Steyerberg, E., et al. (2021) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.007858 · Cited by 83 · see citing works
Oosterhoff, J. H. F., & Doornberg, J. N. (2020) · EFORT Open Reviews · doi:10.1302/2058-5241.5.190092 · Cited by 81 · see citing works
Ogink, P. T., Groot, O. Q., Karhade, A. V., Bongers, M. E. R., Öner, F. C., Verlaan, J., & Schwab, J. H. (2021) · Acta Orthopaedica · doi:10.1080/17453674.2021.1932928 · Cited by 77 · see citing works
External Validation of the DAPT Score in a Nationwide Population
Ueda, P., Jernberg, T., James, S. K., Alfredsson, J., Erlinge, D., Ömerovic, E., Persson, J., Ravn‐Fischer, A., Tornvall, P., Svennblad, B., et al. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.06.023 · Cited by 74 · see citing works
Wang, K., Qiao, Z., Zhao, X., Li, X., Wang, X., Wu, T., Chen, Z., Fan, D., Chen, Q., & Ai, L. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04604-0 · Cited by 70 · see citing works
Predictive Modeling for Metabolomics Data
Ghosh, T., Zhang, W., Ghosh, D., & Kechris, K. J. (2020) · Methods in molecular biology · doi:10.1007/978-1-0716-0239-3_16 · Cited by 70 · see citing works
Singh, H., Mhasawade, V., & Chunara, R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000023 · Cited by 67 · see citing works
Development and Validation of an 18-Gene Urine Test for High-Grade Prostate Cancer
Tosoian, J. J., Zhang, Y., Xiao, L., Xie, C., Samora, N. L., Niknafs, Y. S., Chopra, Z., Siddiqui, J., Zheng, H., Herron, G. C., et al. (2024) · JAMA Oncology · doi:10.1001/jamaoncol.2024.0455 · Cited by 66 · see citing works
Machine Learning as a Support for the Diagnosis of Type 2 Diabetes
Agliata, A., Giordano, D., Bardozzo, F., Bottiglieri, S., Facchiano, A. M., & Tagliaferri, R. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24076775 · Cited by 63 · see citing works
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 60 · see citing works
Hassan, A. M., Lu, S.-C., Asaad, M., Liu, J., Offodile, A. C., Sidey-Gibbons, C. J., & Butler, C. E. (2022) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000000141 · Cited by 57 · see citing works
Ing, E. B., Miller, N. R., Nguyen, A. M., Su, W., Bursztyn, L. L. C. D., Poole, M., Kansal, V., Toren, A., Albreiki, D., Mouhanna, J. G., et al. (2019) · Clinical ophthalmology · doi:10.2147/opth.s193460 · Cited by 57 · see citing works
“37 On DCA, our prediction model is better than “biopsy all” or “biopsy none” strategies for a useful and large range of decision cut points.”
Bongers, M., Karhade, A., Villavieja, J., Groot, O., Bilsky, M., Laufer, I., & Schwab, J. (2020) · The spine journal · doi:10.1016/j.spinee.2020.05.003 · Cited by 51 · see citing works
Carriero, A., Luijken, K., de Hond, A., Moons, K. G. M., CALSTER, B. V., & van Smeden, M. (2025) · Statistics in Medicine · doi:10.1002/sim.10320 · Cited by 56 · see citing works
Oosterhoff, J. H. F., Karhade, A. V., Oberai, T., Franco‐Garcia, E., Doornberg, J. N., & Schwab, J. H. (2021) · Geriatric Orthopaedic Surgery & Rehabilitation · doi:10.1177/21514593211062277 · Cited by 55 · see citing works
Oosterhoff, J. H. F., Gravesteijn, B. Y., Karhade, A. V., Jaarsma, R. L., Kerkhoffs, G. M. M. J., Ring, D., Schwab, J. H., Steyerberg, E. W., & Doornberg, J. N. (2021) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00341 · Cited by 52 · see citing works
Direct Risk Assessment From Myocardial Perfusion Imaging Using Explainable Deep Learning
Singh, A., Miller, R. J. H., Otaki, Y., Kavanagh, P., Hauser, M. T., Tzolos, E., Kwieciński, J., Kriekinge, S. D. V., Wei, C.-C., Sharir, T., et al. (2022) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2022.07.017 · Cited by 51 · see citing works
Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality.
Davis, S. E., Lasko, T. A., Chen, G., & Matheny, M. E. (2017) · PubMed Central · Cited by 50 · see citing works
Impact of Malnutrition on Long-Term Mortality in Elderly Patients with Acute Myocardial Infarction
Komici, K., Vitale, D. F., Mancini, A., Bencivenga, L., Conte, M., Provenzano, S., Grieco, F. V., Visaggi, L., Ronga, I., Cittadini, A., et al. (2019) · Nutrients · doi:10.3390/nu11020224 · Cited by 50 · see citing works
Ferraro, S., Bussetti, M., & Panteghini, M. (2021) · Clinical Chemistry · doi:10.1093/clinchem/hvab002 · Cited by 47 · see citing works
Decision analysis and reinforcement learning in surgical decision-making
Loftus, T. J., Filiberto, A. C., Li, Y., Balch, J. A., Cook, A. C., Tighe, P. J., Efron, P. A., Upchurch, G. R., Rashidi, P., Li, X., et al. (2020) · Surgery · doi:10.1016/j.surg.2020.04.049 · Cited by 48 · see citing works
Gupta, U., Paluru, N., Nankani, D., Kulkarni, K., & Awasthi, N. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26787 · Cited by 51 · see citing works
Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings
Vaid, A., Sawant, A. S., Suárez‐Fariñas, M., Lee, J., Kaul, S. K., Kovatch, P., Freeman, R., Jiang, J. J., Jayaraman, P., Fayad, Z. A., et al. (2023) · Annals of Internal Medicine · doi:10.7326/m23-0949 · Cited by 46 · see citing works
Random‐effects meta‐analysis of the clinical utility of tests and prediction models
Wynants, L., Riley, R. D., Timmerman, D., & CALSTER, B. V. (2018) · Statistics in Medicine · doi:10.1002/sim.7653 · Cited by 44 · see citing works
Yun, H., Choi, J., & Park, J. H. (2021) · JMIR Medical Informatics · doi:10.2196/30770 · Cited by 45 · see citing works
Hassan, A. M., Biaggi, A. P., Asaad, M., Andejani, D. F., Liu, J., Offodile2nd, A. C., Selber, J. C., & Butler, C. E. (2022) · Annals of Surgery · doi:10.1097/sla.0000000000005386 · Cited by 40 · see citing works
Automatic correction of performance drift under acquisition shift in medical image classification
Roschewitz, M., Khara, G., Yearsley, J. E., Sharma, N., James, J. J., Ambrózay, É., Heroux, A. M., Kecskemethy, P., Rijken, T., & Glocker, B. (2023) · Nature Communications · doi:10.1038/s41467-023-42396-y · Cited by 56 · see citing works
Schouten, J., Kalden, M. A. C. M., van Twist, E., Reiss, I. K. M., Gommers, D. A. M. P. J., van Genderen, M. E., & Taal, H. R. (2024) · Intensive Care Medicine · doi:10.1007/s00134-024-07629-8 · Cited by 46 · see citing works
Estiri, H., Strasser, Z. H., Rashidian, S., Klann, J. G., Wagholikar, K. B., McCoy, T. H., & Murphy, S. N. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac070 · Cited by 40 · see citing works
Miller, R. J. H., Singh, A., Otaki, Y., Tamarappoo, B. K., Kavanagh, P., Parekh, T., Hu, L.-H., Gransar, H., Sharir, T., Einstein, A. J., et al. (2022) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-022-05972-w · Cited by 38 · see citing works
Semi-supervised learning to improve generalizability of risk prediction models
Chi, S., Li, X., Tian, Y., Li, J., Kong, X., Ding, K., Weng, C., & Li, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103117 · Cited by 38 · see citing works
Yang, S., Su, T., Huang, L., Feng, L.-H., & Liao, T. (2021) · BMC Nephrology · doi:10.1186/s12882-021-02379-x · Cited by 38 · see citing works
Developing and validating clinical prediction models in hepatology – An overview for clinicians
Strandberg, R., Jepsen, P., & Hagström, H. (2024) · Journal of Hepatology · doi:10.1016/j.jhep.2024.03.030 · Cited by 37 · see citing works
A Machine Learning-Based Prediction Model for Cardiovascular Risk in Women With Preeclampsia
Wang, G., Zhang, Y., Li, S., Zhang, J., Jiang, D., Li, X., Li, Y., & Du, J. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.736491 · Cited by 38 · see citing works
Oosterhoff, J. H. F., Savelberg, A. B. M. C., Karhade, A. V., Gravesteijn, B. Y., Doornberg, J. N., Schwab, J. H., & Heng, M. M. (2022) · European Journal of Trauma and Emergency Surgery · doi:10.1007/s00068-022-01981-4 · Cited by 35 · see citing works
Polce, E. M., Kunze, K. N., Dooley, M. S., Piuzzi, N. S., Boettner, F., & Sculco, P. K. (2022) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00717 · Cited by 34 · see citing works
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33 · see citing works
Machine learning for predicting the treatment effect of orthokeratology in children
Fang, J., Zheng, Y., Mou, H., Shi, M., Yu, W., & Du, C. (2023) · Frontiers in Pediatrics · doi:10.3389/fped.2022.1057863 · Cited by 33 · see citing works
Moledina, S. M., Kontopantelis, E., Wijeysundera, H. C., Banerjee, S., Spall, H. G. C. V., Gale, C. P., Shah, B. N., Mohamed, M., Weston, C. F. M., Shoaib, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac052 · Cited by 33 · see citing works
Xu, Y., Wang, X., Huang, Y., Ye, D., & Chi, P. (2022) · Translational Cancer Research · doi:10.21037/tcr-20-1860 · Cited by 33 · see citing works
Reconsidering lactate as a sepsis risk biomarker
Moran, J. L., & Santamaria, J. D. (2017) · PLoS ONE · doi:10.1371/journal.pone.0185320 · Cited by 31 · see citing works
Are prediction models for vaginal birth after cesarean accurate?
Harris, B. S., Heine, R. P., Park, J., Faurot, K. R., Hopkins, M. K., Rivara, A., Kemeny, H. R., Grotegut, C. A., & Jelovsek, J. E. (2019) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.01.232 · Cited by 32 · see citing works
Deniffel, D., Abraham, N., Namdar, K., Dong, X., Salinas, E., Milot, L. M., Khalvati, F., & Haider, M. A. (2020) · European Radiology · doi:10.1007/s00330-020-07030-1 · Cited by 33 · see citing works
Peinado-Acevedo, J. S., Hurtado-Guerra, J. J., Hincapié, C., Abad, J. M., Uribe-Delgado, J. R., Giraldo-Ramírez, S., Lengerke-Diaz, P. A., & Jaimes, F. A. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1844 · Cited by 31 · see citing works
Das, S. K., Fang, K.-W., Xu, L., Li, B., Zhang, X., & Yang, H. (2021) · Scientific Reports · doi:10.1038/s41598-021-90367-4 · Cited by 31 · see citing works
Predicting radiocephalic arteriovenous fistula success with machine learning
Heindel, P. V., Dey, T., Feliz, J. D., Hentschel, D. M., Bhatt, D. L., Al-Omran, M. A., Belkin, M., Ozaki, C. K., & Hussain, M. A. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00710-w · Cited by 31 · see citing works
Risk calculators for the detection of prostate cancer: a systematic review
Denijs, F., van Harten, M. J., Meenderink, J. J. L., Leenen, R. C. A., Remmers, S., Venderbos, L. D. F., van den Bergh, R. C. N., Beyer, K., & Roobol, M. J. (2024) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-024-00852-w · Cited by 29 · see citing works
Clinical risk reclassification at 10 years
Cook, N. R., Demler, O., & Paynter, N. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7340 · Cited by 29 · see citing works
Park, B., Kim, Y., Lee, J., Lee, N., & Jang, S. H. (2021) · Translational Lung Cancer Research · doi:10.21037/tlcr-21-566 · Cited by 28 · see citing works
Cooray, S. D., Boyle, J. A., Soldatos, G., Allotey, J., Wang, H., Fernandez-Felix, B. M., Zamora, J., Thangaratinam, S., & Teede, H. J. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101637 · Cited by 29 · see citing works
Zhong, X. Z., Li, L., Jiang, H., Yin, J., Lu, B., Han, W. G., Li, J., & Zhang, J. (2020) · BMC Medical Imaging · doi:10.1186/s12880-020-00502-2 · Cited by 29 · see citing works
Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis
Koola, J., Ho, S. B., Cao, A., Chen, G., Perkins, A. M., Davis, S. E., & Matheny, M. E. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05826-w · Cited by 31 · see citing works
Developing well-calibrated illness severity scores for decision support in the critically ill
Cosgriff, C. V., Celi, L. A., Ko, S. Q., Sundaresan, T. G., de la Hoz, M. Á. A., Kaufman, A. R., Stone, D. J., Badawi, O., & Deliberato, R. O. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0153-6 · Cited by 27 · see citing works
Karhade, A. V., Fenn, B. P., Groot, O. Q., Shah, A. A., Yen, H., Bilsky, M. H., Hu, M., Laufer, I., Park, D. Y., Sciubba, D. M., et al. (2022) · The Spine Journal · doi:10.1016/j.spinee.2022.07.089 · Cited by 26 · see citing works
Bongers, M. E. R., Karhade, A. V., Setola, E., Gambarotti, M., Groot, O. Q., Erdoğan, K. E., Picci, P., Donati, D. M., Schwab, J. H., & Palmerini, E. (2020) · Clinical Orthopaedics and Related Research · doi:10.1097/corr.0000000000001305 · Cited by 23 · see citing works
Recalibration Methods for Improved Clinical Utility of Risk Scores
Mishra, A., McClelland, R. L., Inoue, L. Y. T., & Kerr, K. F. (2021) · Medical Decision Making · doi:10.1177/0272989x211044697 · Cited by 23 · see citing works
Pressure injuries in elderly with acute myocardial infarction
Komici, K., Vitale, D. F., Leosco, D., Mancini, A., Corbi, G., Bencivenga, L., Mezzani, A., Trimarco, B., Morisco, C., Ferrara, N., et al. (2017) · Clinical Interventions in Aging · doi:10.2147/cia.s135676 · Cited by 23 · see citing works
Yu, J., Yang, X., Deng, Y., Krefman, A. E., Pool, L. R., Zhao, L., Mi, X., Ning, H., Wilkins, J. T., Lloyd‐Jones, D. M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-51685-5 · Cited by 22 · see citing works
Liang, S., Chau, J. P. C., Lo, S. H. S., Bai, L., Yao, L., & Choi, K. C. (2020) · Nursing in Critical Care · doi:10.1111/nicc.12550 · Cited by 22 · see citing works
- Authors
- Van Calster, B., & Vickers, A. J.
- Affiliations
- KU Leuven, Department of Development and Regeneration, Leuven, Belgium (BVC)
- Memorial Sloan Kettering Cancer Center (US)
- ORCID
- Vickers, Andrew J.
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2014-08-25
- Volume
- 35
- Issue
- 2
- Pages
- 162-169
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- OpenAlex
- W2054193584
- Cited by
- 371
- References
- 29
- Citations by year
- 2026: 44 · 2025: 39 · 2024: 48 · 2023: 40 · 2022: 57 · 2021: 40 · 2020: 31 · 2019: 29 · 2018: 10 · 2017: 19 · 2016: 7 · 2015: 6 · 2014: 1
- FWCI
- 11.61
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Medical Decision Making (publishedVersion)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Fonds Wetenschappelijk Onderzoek
- Vlaamse regering
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model calibration, net benefit, clinical utility, calibration error, overfitting, underfitting, risk overestimation, risk underestimation, risk thresholds, model discrimination, testicular cancer diagnosis
- MeSH terms
- Computer Simulation, Decision Support Techniques, Humans, Male, Models, Theoretical, Norway, Predictive Value of Tests, ROC Curve, Testicular Neoplasms, Bias, Linear Models, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
