CoChat
Sign Up
Cited by 2Adversarial Robustness in Machine LearningGaussian Processes and Bayesian InferenceAdvanced Neural Network Applications

Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference

Open access
Huang, Jiayi, Park, Sangwoo, Simeone, Osvaldo·2025·IEEE Transactions on Signal Processing
DOI 10.1109/tsp.2025.3629292
Open this paper in CoChat — free accountOpen PDF

Abstract

The application of artificial intelligence (AI) models in fields such as engineering is limited by the known difficulty of quantifying the reliability of an AI’s decision. A well-calibrated AI model must correctly report its accuracy on indistribution (ID) inputs, while also enabling the detection of out-of-distribution (OOD) inputs. Conventional solutions are effective at addressing only one of these two conflicting requirements. This paper proposes a novel solution that endows Bayesian learning with enhanced ID calibration and OOD detection capabilities by integrating calibration regularization for improved ID performance, confidence minimization for OOD detection, and selective calibration to ensure a synergistic use of calibration regularization and confidence minimization. Selective calibration rejects inputs for which the calibration performance is expected to be insufficient, supporting effective OOD detection, while also ensuring ID calibration. Prior art had only considered these ideas in isolation and for frequentist learning. Numerical results illustrate the trade-offs between ID accuracy, ID calibration, and OOD calibration, showing that the proposed novel Bayesian approach achieves the best ID and OOD performance compared to existing state-of-the-art approaches, at the cost of rejecting a fraction of the inputs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tsp.2025.3629292

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional point predictions, even when accompanied by confidence scores from probabilistic models, often fail to quantify uncertainty reliably due to miscalibration.

    “Traditional point predictions, even when accompanied by confidence scores from probabilistic models, often fail to quantify uncertainty reliably due to miscalibration [3], [4].”

    From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. [2][2] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and - J. Snoek, 'Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift,' Advances in neural information processing systems , vol. 32, 2019. -
  2. [3][3] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, 'Dermatologist-level classification of skin cancer with deep neural networks,' nature , vol. 542, no. 7639, pp. 115-118, 2017. -
  3. [4][4] A. Kumar, S. Sarawagi, and U. Jain, 'Trainable calibration measures for neural networks from kernel mean embeddings,' in International Conference on Machine Learning . PMLR, 2018, pp. 2805-2814. -
  4. [5][5] O. Bohdal, Y. Yang, and T. Hospedales, 'Metacalibration: Learning of model calibration using differentiable expected calibration error,' arXiv preprint arXiv:2106.09613 , 2021. - · arXiv
  5. [6][6] H. S. Yoon, J. T. J. Tee, E. Yoon, S. Yoon, G. Kim, Y. Li, and C. D. Yoo, 'ESD: Expected squared difference as a tuning-free trainable calibration measure,' arXiv preprint arXiv:2303.02472 , 2023. - · arXiv
  6. [7][7] J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania, 'Calibrating deep neural networks using focal loss,' Advances in Neural Information Processing Systems , vol. 33, pp. 15 288-15 299, 2020. -
  7. [8][8] A. Karandikar, N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs, 'Soft calibration objectives for neural networks,' Advances in Neural Information Processing Systems , vol. 34, pp. 29 76829 779, 2021. -
  8. [9][9] R. Krishnan and O. Tickoo, 'Improving model calibration with accuracy versus uncertainty optimization,' Advances in Neural Information Processing Systems , vol. 33, pp. 18 237-18 248, 2020. -
  9. [10][10] J. Huang, S. Park, and O. Simeone, 'Calibration-aware Bayesian learning,' in 2023 IEEE 33rd International Workshop on Machine Learning for Signal Processing (MLSP) . IEEE, 2023, pp. 1-6. -
  10. [11][11] C. Choi, F. Tajwar, Y. Lee, H. Yao, A. Kumar, and C. Finn, 'Conservative prediction via data-driven confidence minimization,' arXiv preprint arXiv:2306.04974 , 2023. - · arXiv
  11. [12][12] A. Fisch, T. Jaakkola, and R. Barzilay, 'Calibrated selective classification,' arXiv preprint arXiv:2208.12084 , 2022. - · arXiv
  12. [13][13] G. Detommaso, M. Bertran, R. Fogliato, and A. Roth, 'Multicalibration for confidence scoring in LLMs,' arXiv preprint arXiv:2404.04689 , 2024. - · arXiv
  13. [14][14] B. Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam, 'Conformal prediction with large language models for multi-choice question answering,' arXiv preprint arXiv:2305.18404 , 2023. - · arXiv
  14. [15][15] V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. S. Jaakkola, and R. Barzilay, 'Conformal language modeling,' arXiv preprint arXiv:2306.10193 , 2023. - · arXiv
  15. [16][16] M. Zecchin, S. Park, and O. Simeone, 'Forking uncertainties: Reliable prediction and model predictive control with sequence models via conformal risk control,' IEEE Journal on Selected Areas in Information Theory , 2024. -
  16. [17][17] L. Lindemann, M. Cleaveland, G. Shim, and G. J. Pappas, 'Safe planning in dynamic environments using conformal prediction,' IEEE Robotics and Automation Letters , 2023. -
  17. [18][18] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley et al. , 'Robots that ask for help: Uncertainty alignment for large language model planners,' arXiv preprint arXiv:2307.01928 , 2023. - · arXiv
  18. [19][19] O. Simeone, Machine learning for engineers . Cambridge university press, 2022. -
  19. [20][20] A. Masegosa, 'Learning under model misspecification: Applications to variational and ensemble methods,' Advances in Neural Information Processing Systems , vol. 33, pp. 5479-5491, 2020. -
  20. [21][21] J. Knoblauch, J. Jewson, and T. Damoulas, 'Generalized variational inference: Three arguments for deriving new posteriors,' arXiv preprint arXiv:1904.02063 , 2019. - · arXiv
  21. [22][22] F. Wenzel et al. , 'How good is the Bayes posterior in deep neural networks really?' arXiv preprint arXiv:2002.02405 , 2020. - · arXiv
  22. [23][23] Y. Wald, A. Feder, D. Greenfeld, and U. Shalit, 'On calibration and out-of-domain generalization,' Advances in neural information processing systems , vol. 34, pp. 2215-2227, 2021. -
  23. [24][24] C. Henning, F. D'Angelo, and B. F. Grewe, 'Are Bayesian neural networks intrinsically good at out-of-distribution detection?' arXiv preprint arXiv:2107.12248 , 2021. - · arXiv
  24. [25][25] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, 'On calibration of modern neural networks,' in International conference on machine learning . PMLR, 2017, pp. 1321-1330. -
  25. [26][26] L. Tao, Y. Zhu, H. Guo, M. Dong, and C. Xu, 'A benchmark study on calibration,' arXiv preprint arXiv:2308.11838 , 2023. - · arXiv
  26. [27][27] D. Tran, J. Liu, M. W. Dusenberry, D. Phan, M. Collier, J. Ren, K. Han, Z. Wang, Z. Mariet, H. Hu et al. , 'Plex: Towards reliability using pretrained large model extensions,' arXiv preprint arXiv:2207.07411 , 2022. - · arXiv
  27. [28][28] D. Hendrycks, M. Mazeika, and T. Dietterich, 'Deep anomaly detection with outlier exposure,' arXiv preprint arXiv:1812.04606 , 2018. - · arXiv
  28. [29][29] B. Lakshminarayanan, A. Pritzel, and C. Blundell, 'Simple and scalable predictive uncertainty estimation using deep ensembles,' Advances in neural information processing systems , vol. 30, 2017. -
  29. [30][30] Y. Geifman, 'Selectivenet: A deep neural network with an integrated reject option,' in International conference on machine learning . PMLR, 2019, pp. 2151-2159. -
  30. [31][31] L. Huang, C. Zhang, and H. Zhang, 'Self-adaptive training: Beyond empirical risk minimization,' Advances in neural information processing systems , vol. 33, pp. 19 365-19 376, 2020. -
  31. [32][32] A. Pugnana, L. Perini, J. Davis, and S. Ruggieri, 'Deep neural network benchmarks for selective classification,' arXiv preprint arXiv:2401.12708 , 2024. - · arXiv
  32. [34][34] S. Liang, Y. Li, and R. Srikant, 'Principled detection of out-of-distribution examples in neural networks,' CoRR, abs/1706.02690 , vol. 1, 2017. - · arXiv
  33. [35][35] C. Wang, 'Calibration in deep learning: A survey of the state-of-the-art,' arXiv preprint arXiv:2308.01222 , 2023. - · arXiv
  34. [37][37] M. P. Naeini, G. Cooper, and M. Hauskrecht, 'Obtaining well calibrated probabilities using Bayesian binning,' in Proceedings of the AAAI conference on artificial intelligence , vol. 29, no. 1, 2015. -
  35. [38][38] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, 'Weight uncertainty in neural network,' in International conference on machine learning . PMLR, 2015, pp. 1613-1622. -
  36. [39][39] Y. Gal and Z. Ghahramani, 'Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,' in international conference on machine learning . PMLR, 2016, pp. 1050-1059. -
  37. [40][40] S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih, 'Monte carlo gradient estimation in machine learning,' The Journal of Machine Learning Research , vol. 21, no. 1, pp. 5183-5244, 2020. -
  38. [41][41] W. R. Morningstar, A. Alemi, and J. V. Dillon, 'PAC m -Bayes: Narrowing the empirical risk gap in the misspecified Bayesian regime,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2022, pp. 8270-8298. -
  39. [42][42] M. Zecchin, S. Park, O. Simeone, M. Kountouris, and D. Gesbert, 'Robust PAC m : Training ensemble models under misspecification and outliers,' IEEE Transactions on Neural Networks and Learning Systems , 2023. -
  40. [43][43] C. Geng, S.-j. Huang, and S. Chen, 'Recent advances in open set recognition: A survey,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 10, pp. 3614-3631, 2020. -
  41. [44][44] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, 'Likelihood ratios for out-of-distribution detection,' Advances in neural information processing systems , vol. 32, 2019. -
  42. [45][45] D. Hendrycks and K. Gimpel, 'A baseline for detecting misclassified and out-of-distribution examples in neural networks,' arXiv preprint arXiv:1610.02136 , 2016. - · arXiv
  43. [47][47] D. Hendrycks, S. Basart, M. Mazeika, A. Zou, J. Kwon, M. Mostajabi, J. Steinhardt, and D. Song, 'Scaling outof-distribution detection for real-world settings,' arXiv preprint arXiv:1911.11132 , 2019. - · arXiv
  44. [48][48] W. Liu, X. Wang, J. Owens, and Y. Li, 'Energy-based out-of-distribution detection,' Advances in neural information processing systems , vol. 33, pp. 21 464-21 475, 2020. -
  45. [49][49] S. P. Boyd and L. Vandenberghe, Convex optimization . Cambridge university press, 2004. -
  46. [50][50] D. W. Scott, Multivariate density estimation: Theory, practice, and visualization . John Wiley & Sons, 2015. -
  47. [51][51] S. Zagoruyko and N. Komodakis, 'Wide residual networks,' arXiv preprint arXiv:1605.07146 , 2016. - · arXiv
  48. [52][52] T. Raviv, S. Park, O. Simeone, and N. Shlezinger, 'Modular model-based Bayesian learning for uncertaintyaware and reliable deep MIMO receivers,' in 2023 IEEE International Conference on Communications Workshops (ICC Workshops) . IEEE, 2023, pp. 1032-1037. -
  49. [53][53] R. Chitale, A. Vaidya, A. Kane, and A. Ghotkar, 'Task arithmetic with LoRA for continual learning,' arXiv preprint arXiv:2311.02428 , 2023. - · arXiv
  50. [54][54] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning . MIT press Cambridge, 2016, vol. 1, no. 2. -
  51. [56][56] M. Falkiewicz, N. Takeishi, I. Shekhzadeh, A. Wehenkel, A. Delaunoy, G. Louppe, and A. Kalousis, 'Calibrating neural simulation-based inference with differentiable coverage probability,' Advances in Neural Information Processing Systems , vol. 36, 2024. -
  52. [57][57] L. Li, E. Piccoli, A. Cossu, D. Bacciu, and V. Lomonaco, 'Calibration of continual learning models,' arXiv preprint arXiv:2404.07817 , 2024. - · arXiv
  53. [58][58] F. T. Liu, K. M. Ting, and Z.-H. Zhou, 'Isolation forest,' in 2008 IEEE 8th International Conference on Data Mining . IEEE, 2008, pp. 413-422. -
  54. [59][59] B. Sch¨ olkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, 'Estimating the support of a highdimensional distribution,' Neural computation , vol. 13, no. 7, pp. 1443-1471, 2001. -
  55. [60][60] D. O. Loftsgaarden and C. P. Quesenberry, 'A nonparametric estimate of a multivariate density function,' The Annals of Mathematical Statistics , vol. 36, no. 3, pp. 1049-1051, 1965. -
  56. [61][61] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'Pytorch: An imperative style, high-performance deep learning library,' Advances in neural information processing systems , vol. 32, 2019. -
  57. [62][62] D. Hendrycks, M. Mazeika, and T. Dietterich, 'Deep anomaly detection with outlier exposure,' Proceedings of the International Conference on Learning Representations , 2019.

Cited by

Sorted by citations

Authors
Huang, J., Park, S., & Simeone, O.
Affiliations
King's College London (GB)
ORCID
Venue
IEEE Transactions on Signal Processing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2025
Volume
73
Pages
4492-4505
ISSN
  • 1053-587X (ISSN-L)
  • 1941-0476
DOI
10.1109/tsp.2025.3629292
OpenAlex
W4416078055
Cited by
2
References
3
Citations by year
2025: 1 · 2024: 1
FWCI
0.40
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • King’s College London and China Scholarship Council for their Joint Full-Scholarship (202206150005)
  • European Union’s Horizon Europe project CENTRIC under Grant (101096379)
  • Engineering and Physical Sciences Research Council (EP/W024101/1, EP/X011852/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
Bayesian neural networks, out-of-distribution detection, uncertainty quantification, AI reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review