Measuring Calibration in Deep Learning
Open accessAbstract
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, has numerous flaws which we outline, and there is no clear empirical understanding of how its choices affect conclusions in practice, and what recommendations there are to counteract its flaws. In this paper, we perform a comprehensive empirical study of choices in calibration measures including measuring all probabilities rather than just the maximum prediction, thresholding probability values, class conditionality, number of bins, bins that are adaptive to the datapoint density, and the norm used to compare accuracies to confidences. To analyze the sensitivity of calibration measures, we study the impact of optimizing directly for each variant with recalibration techniques. Across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, we find that conclusions on the rank ordering of recalibration methods is drastically impacted by the choice of calibration measure. We find that conditioning on the class leads to more effective calibration evaluations, and that using the L2 norm rather than the L1 norm improves both optimization for calibration metrics and the rank correlation measuring metric consistency. Adaptive binning schemes lead to more stablity of metric rank ordering when the number of bins vary, and is also recommended. We open source a library for the use of our calibration measures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ECE has a severe imbalance of samples across bins because neural networks tend to always predict with high confidence, causing only a few bins to contribute most to ECE; ECE is also sensitive to bin boundaries.
“However, there might be a severe imbalance of samples across the bins because neural networks tend to always predict with high confidence (Nixon et al., 2019). This causes only a few bins (concentrated towards the high confidence region) to contribute the most to ECE. In addition, ECE is also sensitive to the bin boundaries i.e., the number of bins.”
From QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring (2024) · cites this paper as Nixon et al., 2019
A cited study reports a greater reduction in closed-domain hallucinations than in open-domain ones.
“Consistent with this, [27] reports a greater reduction in closed-domain hallucinations over open-domain ones.”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [27]
The Abstract (p. 1) describes a study of calibration measures, and §5.3 (pp. 6–7) compares their sensitivity to binning choices. Searches for closed-domain and open-domain hallucinations found no passage reporting a reduction in either, so our copy does not address the attributed comparison. (LLM, conf=0.78)
Experimental — accuracy not yet measuredPost-training alignment was used to reduce hallucination but was also found to reduce calibration.
“Post-training alignment was applied to reduce hallucination (among other factors) but was also found to reduce calibration [27].”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [27]
The Introduction (p. 2) describes a study of calibration measures, and §2.3 (p. 3) discusses post-processing classifier outputs for recalibration. Searches for post-training alignment, hallucination, and reduced calibration found no passage addressing the claimed trade-off. (LLM, conf=0.76)
Experimental — accuracy not yet measuredPost-training reduces hallucination rates at the cost of increasing calibration error, as suggested by the model and shown in GPT-4 calibration curves.
“As suggested by our model, post-training reduces hallucination rates at the cost of increasing calibration error.Note that calibration here is on a multiple-choice test rather than generative calibration which we study.”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [27]
Section 5.4 (pp. 7–8) examines how calibration metrics rank recalibration methods; Section 2.3 (p. 3) describes post-processing classifiers to calibrate their predictions. Our copy does not address hallucination rates, GPT-4 calibration curves, or the claimed trade-off from post-training (searched: post-training, hallucination, GPT-4, multiple-choice). (LLM, conf=0.77)
Experimental — accuracy not yet measuredIn the first stage of pretraining, common today, the generative language model is optimized for predictive text performance.
“Notably in the first stage of pretraining, common today, the generative LM is optimized for predictive text performance [9,27].”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [27]
The Introduction (p. 1) discusses calibrated prediction probabilities, and the other retrieved passages concern classifier calibration. Searches for generative language models, pretraining, and next-token prediction found no passage addressing whether a generative LM is optimized for predictive text performance in its first stage of pretraining. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 1× · see citing works
“One mathematical formulation of the reliability of confidence is calibration (Murphy and Epstein, 1967; Dawid, 1982).”
BackgroundThe Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · see citing works
Tsoukalas, A., Albertson, T., & Tagkopoulos, I. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3445 · Cited by 96 · cited 1× · see citing works
“Decision costs along both ethical and monetary axes are non-uniform, and thus clinical decision policies necessitate a comprehensive view of the likelihoods of potential outcomes (e.g., Tsoukalas et al., 2015).”
BackgroundVerification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 2×
“Unlike the Brier score (Brier, 1950) and Hosmer-Lemeshow (Hosmer & Lemesbow, 1980), we’d like the metric to be scalar-valued in order to easily benchmark methods, and to only measure calibration.”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 18
- Cynthia S Crowson, Elizabeth J Atkinson, and Terry M Therneau. Assessing calibration of prognostic risk scores. Statistical methods in medical research , 25(4):1692-1706, 2016.
- Michael W. Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M. Dai. Analyzing the role of model uncertainty for electronic health records. In Proc. of the ACM Conference on Health, Inference, and Learning (ACM CHIL) , pp. 204-213, Toronto Ontario Canada, April 2020. ACM. ISBN 978-1-4503-7046-2. doi: 10.1145/3368555.3384457. URL http://arxiv.org/abs/1906.03842 . · DOI
- Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007.
- Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 , 2018. · arXiv
- David W Hosmer and Stanley Lemesbow. Goodness of fit tests for the multiple logistic regression model. Communications in statistics-Theory and Methods , 9(10):1043-1069, 1980.
- Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association , 19(2):263-274, 2011.
- Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv:1807.00263 , 2018. · arXiv
- Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning , pp. 2810-2819, 2018.
- Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. arXiv preprint arXiv:1711.09325 , 2017. · arXiv
- Allan H Murphy and Edward S Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology , 6(5):748-755, 1967.
- Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.
- Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:1508.05154 , 2015. · arXiv
- Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning , pp. 625-632. ACM, 2005.
- John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.
- Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Robert Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. arXiv preprint arXiv:1807.01771 , 2018. · arXiv
- Seonguk Seo, Paul Hongsuck Seo, and Bohyung Han. Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 9030-9038, 2019.
- Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. arXiv preprint arXiv:1905.11001 , 2019. · arXiv
- Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schön. Evaluating model calibration in classification. arXiv preprint arXiv:1902.06977 , 2019. · arXiv
Cited by
Sorted by citations
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3155327 · Cited by 879 · see citing works
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Self-citationMix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning
Zhang, J., Kailkhura, B., & Han, Y.-J. (2020) · Unpublished · doi:10.13140/rg.2.2.11390.13125 · Cited by 86 · see citing works
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
“For 𝐼 ⊆ [ 0 , 1 ] , define: where The miscalibration of 𝑔 withrespectto 𝑝 is Mis 𝑏 ( 𝑔,𝑝 ) : = (cid:13)(cid:13)(cid:13) 𝑝 V 𝑏 ( 𝑔 ) − 𝑔 .”
A Close Look into the Calibration of Pre-trained Language Models
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.75 · Cited by 22 · see citing works
Calibration of Model Uncertainty for Dropout Variational Inference
Laves, M.-H., Ihler, S., Kortmann, K.-P., & Ortmaier, T. (2020) · arXiv · doi:10.48550/arxiv.2006.11584 · Cited by 11 · see citing works
QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring
Ghanathe, N. P., & Wilton, S. J. E. (2024) · arXiv · doi:10.48550/arxiv.2404.12599
- Authors
- Nixon, J., Dusenberry, M., Jerfel, G., Nguyen, T., Liu, J., Zhang, L., & Tran, D.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.01685
- OpenAlex
- W2933254221
- Cited by
- 209
- References
- 8
- Citations by year
- 2026: 3 · 2025: 22 · 2024: 40 · 2023: 60 · 2022: 28 · 2021: 37 · 2020: 17 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- classifier calibration, expected calibration error, overconfidence, underconfidence, recalibration, adaptive binning, calibration metrics, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
