Calibration of Model Uncertainty for Dropout Variational Inference
Open accessAbstract
The model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. In this paper, different logit scaling methods are extended to dropout variational inference to recalibrate model uncertainty. Expected uncertainty calibration error (UCE) is presented as a metric to measure miscalibration. The effectiveness of recalibration is evaluated on CIFAR-10/100 and SVHN for recent CNN architectures. Experimental results show that logit scaling considerably reduce miscalibration by means of UCE. Well-calibrated uncertainty enables reliable rejection of uncertain predictions and robust detection of out-of-distribution data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Having a proper uncertainty method can assist in having better medical image analysis performance.
“As a result, they concluded that having a proper uncertainty method can assist having a better medical image analysis performance.”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [505]
The proposed UMCT method outperformed the other applied methods on the same datasets.
“According to the obtained results, the proposed UMCT method outperformed the other applied methods to the same datasets.”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [505]
Section 4.1 (p. 6) describes UCE and classwise UCE metrics, and Table 1 (p. 7) compares calibration methods; neither reports results for a UMCT method. Searches for “UMCT” and for the claimed comparison found no passage addressing whether UMCT outperformed other methods on the same datasets. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
(2014) · Cited by 33,419 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834 · see citing works
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · see citing works
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · see citing works
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · see citing works
The Concrete Distribution: A Continuous Relaxation of Discrete Random VariablesPreprint
Maddison, C. J., Mnih, A., & Teh, Y. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00712 · Cited by 1,771 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,280 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial AttacksPreprint
Lee, K., Lee, K., Lee, H., & Shin, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03888 · Cited by 1,166 · see citing works
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991 · see citing works
Variational Dropout and the Local Reparameterization TrickPreprint
Kingma, D. P., Salimans, T., & Welling, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02557 · Cited by 708 · see citing works
Regularizing Neural Networks by Penalizing Confident Output Distributions
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655 · see citing works
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 636 · see citing works
Accurate Uncertainties for Deep Learning Using Calibrated RegressionPreprint
Kuleshov, V., Fenner, N., & Ermon, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00263 · Cited by 455 · see citing works
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., & Wilson, A. G. (2019) · arXiv · doi:10.48550/arxiv.1902.02476 · Cited by 429 · see citing works
Deep Bayesian Active Learning with Image Data
Ghahramani, Z., Gal, Y., & Islam, R. (2017) · PMLR · doi:10.17863/cam.11070 · Cited by 392 · see citing works
(2013) · Cited by 326 · see citing works
Multiplicative Normalizing Flows for Variational Bayesian Neural NetworksPreprint
Louizos, C., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01961 · Cited by 254 · see citing works
Measuring Calibration in Deep Learning
Nixon, J., Dusenberry, M., Jerfel, G., Nguyen, T., Liu, J., Zhang, L., & Tran, D. (2019) · arXiv · doi:10.48550/arxiv.1904.01685 · Cited by 209 · see citing works
Bayesian Uncertainty Estimation for Batch Normalized Deep NetworksPreprint
Teye, M., Azizpour, H., & Smith, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06455 · Cited by 189 · see citing works
Evaluating model calibration in classification
Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., & Schön, T. B. (2019) · arXiv · doi:10.48550/arxiv.1902.06977 · Cited by 117 · see citing works
Concrete DropoutPreprint
Gal, Y., Hron, J., & Kendall, A. (2017) · arXiv (Cornell University) · Cited by 115 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Bishop, C. M. Pattern Recognition and Machine Learning . Springer, 2006. ISBN 978-0-387-31073-2.
- Gal, Y. Uncertainty in Deep Learning . PhD thesis, Department of Engineering, University of Cambridge, 2016.
- Kull, M., Nieto, M. P., Kängsepp, M., Silva Filho, T., Song, H., and Flach, P. Beyond temperature scaling: Obtaining wellcalibrated multi-class probabilities with dirichlet calibration. In NeurIPS , pp. 12295-12305, 2019.
- Xiao, T. Z., Gomez, A. N., and Gal, Y. Wat heb je gezegd? detecting out-of-distribution translations with variational transformers. In Bayesian Deep Learning Workshop (NeurIPS) , 2019.
Cited by
Sorted by citations
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
“[282] (UCE: uncertainty calibration error), Yang et al.”
BackgroundConfidence-based interactable neural-symbolic visual question answering
Bao, Y., Xing, T., & Chen, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126991 · Cited by 21 · see citing works
Uncertainty-Aware Self-Knowledge Distillation
Yang, Y., Wang, C., Gong, L., Wu, M., Chen, Z., Gao, Y., Wang, T., & Zhou, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3516145 · Cited by 8 · see citing works
Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression
Yu, X., Franchi, G., Gu, J., & Aldea, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28508 · Cited by 8 · see citing works
Autoencoder Attractors for Uncertainty Estimation
Cruz, S., Taetz, B., Stifter, T., & Stricker, D. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956240 · Cited by 8 · see citing works
Bench, C. A., Desai, V., Moulaeifard, M., Strodthoff, N., Aston, P. J., & Thompson, A. (2025) · Machine Learning Health · doi:10.1088/3049-477x/ae0b74 · Cited by 3 · see citing works
An uncertainty aided framework for learning based liver T 1 ρ mapping and analysis
Huang, C., Wong, V. W., Chan, Q., Chu, C. W., & Chen, W. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad027e · Cited by 3 · see citing works
Non-relevant segment recognition via hard example mining under sparsely distributed events
Park, B., Chi, H., Lee, J. H., Park, B., Lee, J., Shin, S., Hyung, W. J., & Choi, M.-K. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108906 · Cited by 1 · see citing works
Weight Pruning and Uncertainty in Radio Galaxy ClassificationPreprint
Mohan, D., & Scaife, A. M. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11654 · Cited by 1 · see citing works
Improving the Behaviour of Vision Transformers with Token-Consistent Stochastic LayersPreprint
Popović, N., Paudel, D. P., Probst, T., & Gool, L. V. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4137177 · Cited by 1 · see citing works
Xu, P., Coen-Cagli, R., & Schwartz, O. (2024) · Neural Computation · doi:10.1162/neco_a_01652
- Authors
- Laves, M.-H., Ihler, S., Kortmann, K.-P., & Ortmaier, T.
- Affiliations
- Leibniz University Hannover (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.11584
- OpenAlex
- W3036633009
- Cited by
- 11
- References
- 33
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 2 · 2022: 2 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Keywords
- uncertainty calibration, Monte Carlo dropout, variational inference, out-of-distribution detection, convolutional neural networks, CIFAR-10, CIFAR-100, SVHN
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
