A tutorial on calibration measurements and calibration models for clinical prediction models
Open accessAbstract
Our primary objective is to provide the clinical informatics community with an introductory tutorial on calibration measurements and calibration models for predictive models using existing R packages and custom implemented code in R on real and simulated data. Clinical predictive model performance is commonly published based on discrimination measures, but use of models for individualized predictions requires adequate model calibration. This tutorial is intended for clinical researchers who want to evaluate predictive models in terms of their applicability to a particular population. It is also for informaticians and for software engineers who want to understand the role that calibration plays in the evaluation of a clinical predictive model, and to provide them with a solid starting point to consider incorporating calibration evaluation and calibration models in their work. Covered topics include (1) an introduction to the importance of calibration in the clinical setting, (2) an illustration of the distinct roles that discrimination and calibration play in the assessment of clinical predictive models, (3) a tutorial and demonstration of selected calibration measurements, (4) a tutorial and demonstration of selected calibration models, and (5) a brief discussion of limitations of these methods and practical suggestions on how to use them in practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Learning Bayesian Networks: The Combination of Knowledge and Statistical Data
Heckerman, D. E., Geiger, D., & Chickering, D. M. (1995) · Machine Learning · doi:10.1007/bf00994016 · Cited by 4,082 · see citing works
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Learning Bayesian Networks: The Combination of Knowledge and Statistical Data
Heckerman, D. E., Geiger, D., & Chickering, D. M. (1995) · Machine Learning · doi:10.1023/a:1022623210503 · Cited by 3,251 · see citing works
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928 · see citing works
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,902 · see citing works
Regression modelling strategies for improved prognostic prediction
Harrell, F. E., Lee, K. L., Califf, R. M., Pryor, D. B., & Rosati, R. A. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030207 · Cited by 1,804 · see citing works
Goodness of fit tests for the multiple logistic regression model
Hosmer, D., & Lemesbow, S. (1980) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928008827941 · Cited by 1,780 · see citing works
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Monetary Costs of Dementia in the United States
Hurd, M. D., Martorell, P., Delavande, A., Mullen, K. J., & Langa, K. M. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1204629 · Cited by 1,484 · see citing works
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Kramer, A. A., & Zimmerman, J. E. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000275267.64078.b0 · Cited by 886 · see citing works
Sahm, F., Schrimpf, D., Stichel, D., Jones, D., Hielscher, T., Schefzyk, S., Okonechnikov, K., Koelsche, C., Reuß, D., Capper, D., et al. (2017) · The Lancet Oncology · doi:10.1016/s1470-2045(17)30155-9 · Cited by 863 · see citing works
Cardiovascular Event Prediction by Machine Learning: The Multi-Ethnic Study of Atherosclerosis
Ambale-Venkatesh, B., Yang, X., Wu, C. O., Liu, K., Hundley, W. G., McClelland, R., Gomes, A. S., Folsom, A. R., Shea, S., Guallar, E., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.311312 · Cited by 639 · see citing works
Riley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Predicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 571 · see citing works
PresenceAbsence : An R Package for Presence Absence Analysis
Freeman, E. A., & Moisen, G. G. (2008) · Journal of Statistical Software · doi:10.18637/jss.v023.i11 · Cited by 522 · see citing works
Bendapudi, P., Hurwitz, S., Fry, A., Marques, M., Waldo, S., Li, A., Sun, L., Upadhyay, V., Hamdan, A., Brunner, A., et al. (2017) · The Lancet Haematology · doi:10.1016/s2352-3026(17)30026-1 · Cited by 470 · see citing works
Standardizing the power of the Hosmer–Lemeshow goodness of fit test in large data sets
Paul, P., Pennell, M. L., & Lemeshow, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.5525 · Cited by 434 · see citing works
Use of Brier score to assess binary predictions.
Rufibach, K. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.009 · Cited by 426 · see citing works
Probabilistic prediction in patient management and clinical trials
Spiegelhalter, D. J. (1986) · Statistics in Medicine · doi:10.1002/sim.4780050506 · Cited by 362 · see citing works
Increasing the Reliability of Reliability Diagrams
Bröcker, J., & Smith, L. A. (2007) · Weather and Forecasting · doi:10.1175/waf993.1 · Cited by 316 · see citing works
Austin, P. C., & Steyerberg, E. W. (2019) · Statistics in Medicine · doi:10.1002/sim.8281 · Cited by 311 · see citing works
Tests of calibration and goodness‐of‐fit in the survival setting
Demler, O., Paynter, N. P., & Cook, N. R. (2015) · Statistics in Medicine · doi:10.1002/sim.6428 · Cited by 287 · see citing works
Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12916 · Cited by 264 · see citing works
Isotone Optimization in R : Pool-Adjacent-Violators Algorithm (PAVA) and Active Set Methods
de Leeuw, J., Hornik, K., & Mair, P. (2009) · Journal of Statistical Software · doi:10.18637/jss.v032.i05 · Cited by 246 · see citing works
Calibrating predictive model estimates to support personalized medicine
Jiang, X., Osl, M., Kim, J., & Lucila Ohno-machado, A. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000291 · Cited by 209 · see citing works
Statistical Evaluation of Diagnostic Performance: Topics in Roc Analysis
Rockette, H. E. (2011) · Medical Entomology and Zoology · Cited by 135 · see citing works
Deep Learning for the Internet of Things
Yao, S., Zhao, Y., Zhang, A., Hu, S., Shao, H., Zhang, C., Su, L., & Abdelzaher, T. (2018) · Computer · doi:10.1109/mc.2018.2381131 · Cited by 132 · see citing works
A discussion of calibration techniques for evaluating binary and categorical predictive models.
Fenlon, C., O'Grady, L., Doherty, M., & Dunnion, J. (2018) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2017.11.018 · Cited by 121 · see citing works
A New Method for Estimation of Resource Selection Probability Function
Lele, S. R. (2009) · Journal of Wildlife Management · doi:10.2193/2007-535 · Cited by 119 · see citing works
Kull, M., Filho, T. M. S., & Flach, P. A. (2017) · Bristol Research (University of Bristol) · Cited by 113 · see citing works
Clinical Prediction Models for Cardiovascular Disease
Wessler, B. S., Lana Lai, Y. H., Kramer, W., Cangelosi, M. J., Raman, G., Lutz, J. S., & Kent, D. M. (2015) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.115.001693 · Cited by 106 · see citing works
Licher, S., Yilmaz, P., Leening, M. J. G., Wolters, F. J., Vernooij, M. W., Stephan, B. C. M., Ikram, M. K., & Ikram, M. A. (2018) · European Journal of Epidemiology · doi:10.1007/s10654-018-0403-y · Cited by 103 · see citing works
Walsh, C. G., Sharman, K., & Hripcsak, G. M. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.10.008 · Cited by 102 · see citing works
Steyerberg, E. W., Nieboer, D., Debray, T. P. A., & van Houwelingen, H. C. (2019) · Statistics in Medicine · doi:10.1002/sim.8296 · Cited by 73 · see citing works
Predictors of Poor School Readiness in Children Without Developmental Delay at Age 2
Nelson, B. B., Dudovitz, R., Coker, T., Barnert, E., Biely, C., Li, N., Szilagyi, P., Larson, K., Halfon, N., Zimmerman, F., et al. (2016) · Pediatrics · doi:10.1542/peds.2015-4477 · Cited by 58 · see citing works
A modified Hosmer–Lemeshow test for large data sets
Yu, W., Xu, W., & Zhu, L. (2017) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2017.1285922 · Cited by 55 · see citing works
Manktelow, B., Draper, E. S., & Field, D. (2009) · Archives of Disease in Childhood: Fetal and Neonatal Edition · doi:10.1136/adc.2008.148015 · Cited by 49 · see citing works
Mortality prediction following non-traumatic amputation of the lower extremity
Norvell, D. C., Thompson, M. L., Boyko, E. J., Landry, G. J., Littman, A. J., Henderson, W. G., Turner, A. P., Maynard, C. C., Moore, K. P., & Czerniecki, J. M. (2019) · British journal of surgery · doi:10.1002/bjs.11124 · Cited by 48 · see citing works
Spontaneous breathing test in the prediction of extubation failure in the pediatric population
Nascimento, M. S., Rebello, C. M., Vale, L. A. P. A., et al. (2017) · Einstein (São Paulo) · doi:10.1590/s1679-45082017ao3913 · Cited by 40 · see citing works
Calibrating Classification Probabilities with Shape-Restricted Polynomial Regression
Wang, Y.-Q., Li, L.-S., & Dang, C. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2895794 · Cited by 32 · see citing works
Smooth isotonic regression: a new method to calibrate predictive models.
Jiang, X., Osl, M., Kim, J., et al. (2011) · PubMed · Cited by 29 · see citing works
A simple test procedure in standardizing the power of Hosmer–Lemeshow test in large data sets
Lai, X., & Liu, L. (2018) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2018.1467912 · Cited by 22 · see citing works
External validation of the breast reconstruction risk assessment calculator
Khavanin, N., Qiu, C. S., Mlodinow, A. S., Vu, M. M., Dorfman, R. G., Fine, N. A., & Kim, J. Y. S. (2017) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2017.04.010 · Cited by 21 · see citing works
Statistical Evaluation of Diagnostic Performance
Zou, K. H., Liu, A., Bandos, A. I., Lucila Ohno-machado, A., & Rockette, H. E. (2011) · Chapman & Hall/CRC eBooks · doi:10.1201/b11031 · Cited by 17 · see citing works
Jiang, X., Menon, A. K., Wang, S., Kim, J., & Lucila Ohno-machado, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0048823 · Cited by 14 · see citing works
Estimating Cardiovascular Risk.
Firnhaber, J. M. (2017) · PubMed · Cited by 2 · see citing works
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“18 The z statistic can be calculated with equation 3.”
Method
Flagged first, then by citations
Research works printed but not linked: 8
- [15]Niculescu-Mizil A. Caruana Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning - ICML ‘05 New York, NY: ACM Press; 2005: 625– 32.
- [25]Guo C, Pleiss G, Sun Y, Weinberger K. On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning 2017; Sydney, Australia.
- [27]Cox DR. Two further applications of a model for binary regression. Biometrika 1958; 45 (3–4): 592–65.
- [37]Zadrozny B, Elkan C. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In: Proceedings of the Eighteenth International Conference on Machine Learning; 2001: 609– 16.
- [40]Healthcare Cost and Utilization Project. HCUP Nationwide Inpatient Sample. Rockville, MD: Agency for Healthcare Research and Quality; 2012. 21413206
- [52]Lee K, Lee K, Lee H, Shin J. Training confidence-calibrated classifiers for detecting out-of-distribution samples. Int: proceedings of the Sixth International Conference on Learning Representations; April 30, 2018 to May 3, 2018; Vancouver, Canada.
- [53]Maddox W, Garipov T, Izmailov P, Vetrov D, Wilson AG. A simple baseline for Bayesian uncertainty in deep learning. arXiv 2019 Dec 31 [E-pub ahead of print].
- [57]Zadrozny B. Elkan Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ‘02 New York, NY: ACM Press; 2002: 694– 9.
Cited by
Sorted by citations
Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186 · see citing works
Calibrating machine learning approaches for probability estimation: A comprehensive comparison
Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., & Ziegler, A. (2023) · Statistics in Medicine · doi:10.1002/sim.9921 · Cited by 71 · see citing works
Kamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., et al. (2022) · BMJ · doi:10.1136/bmj-2021-068576 · Cited by 60 · see citing works
Afshar, M., Sharma, B., Dligach, D., Oguss, M., Brown, R., Chhabra, N., Thompson, H. M., Markossian, T., Joyce, C., Churpek, M. M., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00041-3 · Cited by 55 · see citing works
Predicting next-day discharge via electronic health record access logs
Zhang, X., Yan, C., Malin, B. A., Patel, M. B., & Chen, Y. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab211 · Cited by 38 · see citing works
Charu, V., Liang, J. W., Mannalithara, A., Kwong, A., Tian, L., & Kim, W. R. (2024) · Hepatology · doi:10.1097/hep.0000000000000908 · Cited by 35 · see citing works
Cabanillas Silva, P., Sun, H., Rezk, M., Roccaro-Waldmeyer, D. M., Fliegenschmidt, J., Hulde, N., von Dossow, V., Meesseman, L., Depraetere, K., Stieg, J., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51409 · Cited by 20 · see citing works
Aguirre, U., & Urrechaga, E. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2022-0713 · Cited by 17 · see citing works
Iantorno, S. E., Bucher, B. T., Horns, J. J., & McCrum, M. L. (2023) · Journal of Trauma and Acute Care Surgery · doi:10.1097/ta.0000000000003856 · Cited by 11 · see citing works
Zhu, Y., Sui, D., Wang, Z., Hu, X., Gu, L., Qi, Y., Wu, T., Wang, L., Wei, Y., Tang, W., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791458 · Cited by 3 · see citing works
- Authors
- Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L.
- Affiliations
- UC San Diego Health System (US)
- University of California San Diego (US)
- Department of Biomedical Informatics, UC San Diego Health, University of California, San Diego, La Jolla, California, USA
- Corresponding author
- Huang, Yingxiang
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-02-27
- Volume
- 27
- Issue
- 4
- Pages
- 621-633
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7075534
- OpenAlex
- W3007453563
- Cited by
- 486
- References
- 55
- Citations by year
- 2026: 117 · 2025: 104 · 2024: 75 · 2023: 92 · 2022: 55 · 2021: 34 · 2020: 9
- FWCI
- 7.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R01GM118609, R01HL136835)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- model calibration, calibration models, clinical prediction model, calibration measurements, discrimination (statistics), prediction model evaluation, personalized prediction, R package
- MeSH terms
- Calibration, Humans, ROC Curve, Models, Statistical, Logistic Models, Risk Assessment, Area Under Curve, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
