Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation
Open accessAbstract
Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we present the following contributions. First, we show that the classical approaches fail to approximate the classification probability. Second, we propose a scalable and intuitive framework for uncertainty quantification in medical image segmentation that yields measurements that approximate the classification probability. Third, we suggest the usage of k-fold cross-validation to overcome the need for held out calibration data. Lastly, we motivate the adoption of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MCC's consensus mechanism further reinforces this by requiring convergent evidence, similar to ensemble-based uncertainty calibration in medical image segmentation.
“MCC’s consensus mechanism further reinforces this by requiring convergent evidence, similar to ensemble-based uncertainty calibration in medical image segmentation.28”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [28]
Cites
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015
Navab, N., Joachim Hornegger, A., Wells, W. M., & Frangi, A. F. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9 · Cited by 2,288 · see citing works
U-Net: deep learning for cell counting, detection, and morphometry
Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., et al. (2018) · Nature Methods · doi:10.1038/s41592-018-0261-2 · Cited by 1,912 · see citing works
Heller, N. E., Isensee, F., Maier-Hein, K. H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101821 · Cited by 586 · see citing works
Schaffter, T., Buist, D. S. M., Lee, C. I., Nikulin, Y., Ribli, D., Guan, Y., Lotter, W., Jie, Z., Du, H., Wang, S., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.0265 · Cited by 409 · see citing works
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019
Shen, D. G., Liu, T., Peters, T. M., Staib, L. H., Essert, C., Zhou, S., Yap, P., & Khan, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9 · Cited by 395 · see citing works
High-Performance Large-Scale Image Recognition Without NormalizationPreprint
Brock, A., De, S., Smith, S. L., & Simonyan, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06171 · Cited by 356 · see citing works
Mathematical Considerations in the Estimation of Spectra
Parzen, E. (1961) · Technometrics · doi:10.1080/00401706.1961.10489939 · Cited by 294 · see citing works
A survey on U-shaped networks in medical image segmentations
Liu, L., Cheng, J., Quan, Q., Wu, F., Wang, Y., & Wang, J. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.05.070 · Cited by 287 · see citing works
TotalSegmentator: robust segmentation of 104 anatomical structures in CT images
Wasserthal, J., Breit, H., Meyer, M. T., et al. (2022) · PubMed · doi:10.48550/arxiv.2208.05868 · Cited by 276 · see citing works
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision
Gustafsson, F., Danelljan, M., & Schön, T. B. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00167 · Cited by 272 · see citing works
Abdar, M., Samami, M., Mahmoodabad, S. D., Doan, T., Mazoure, B., Hashemifesharaki, R., Liu, L., Khosravi, A., Acharya, U. R., Makarenkov, V., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104418 · Cited by 241 · see citing works
A Survey on Active Deep Learning: From Model Driven to Data Driven
Liu, P., Wang, L., Ranjan, R. K., He, G., & Zhao, L. (2022) · ACM Computing Surveys · doi:10.1145/3510414 · Cited by 199 · see citing works
Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy
Yeung, M., Sala, E., Schönlieb, C.-B., & Rundo, L. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104815 · Cited by 186 · see citing works
Self-citationSelf-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · see citing works
Robust and Efficient Medical Imaging with Self-SupervisionPreprint
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., MacWilliams, P., Mahdavi, S. S., Wulczyn, E., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2205.09723 · Cited by 82 · see citing works
A Three-Stage Self-Training Framework for Semi-Supervised Semantic Segmentation
Ke, R., Avilés-Rivero, A. I., Pandey, S., Reddy, S., & Schönlieb, C. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3144036 · Cited by 78 · see citing works
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning
Arazo, E., Ortego, D., Albert, P., O'Connor, N. E., & McGuinness, K. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207304 · Cited by 77 · see citing works
González, C., Gotkowski, K., Fuchs, M., Bucher, A. M., Dadras, A. A., Fischbach, R., Kaltenborn, I., & Mukhopadhyay, A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102596 · Cited by 59 · see citing works
S2wintounet: Star-Shaped Window Transformer Onion U-Net for Medical Image Segmentation
Zhang, J., Liu, Y., Wu, Q., Wang, Y., Liu, Y., Xu, X., & Song, B. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105954 · Cited by 49 · see citing works
Uncalibrated Models Can Improve Human-AI CollaborationPreprint
Vodrahalli, K., Gerstenberg, T., & Zou, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.05983 · Cited by 15 · see citing works
Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?Preprint
Ma, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00232 · Cited by 12 · see citing works
Know Your Limits: Monotonicity & Softmax Make Neural Classifiers Overconfident on OOD Data.Preprint
Ulmer, D., & Cinà, G. (2020) · arXiv (Cornell University) · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 1 · see citing works
“28 This bidirectional optimization mechanism, which corrects initially incorrect answers while maintaining correct ones, enables superior performance of MCC over simple voting schemes for contentious multiple-choice questions.”
Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation
Kirscher, T., Bujotzek, M., Kirchhoff, Y., Rokuss, M., Isensee, F., Kahl, K.-C., Kovacs, B., & Maier-Hein, K. (2026) · arXiv · doi:10.48550/arxiv.2605.18329
- Authors
- Buddenkotte, T., Escudero Sanchez, L., Crispin-Ortuzar, M., Woitek, R., McCague, C., Brenton, J. D., Öktem, O., Sala, E., & Rundo, L.
- Affiliations
- University of Cambridge (GB)
- Danube Private University (AT)
- KTH Royal Institute of Technology (SE)
- Università Cattolica del Sacro Cuore (IT)
- University of Salerno (IT)
- Corresponding author
- Buddenkotte, Thomas
- ORCID
- Venue
- Computers in Biology and Medicine
- Publisher
- Elsevier BV
- Published
- 2023-09
- Volume
- 163
- Pages
- 107096
- ISSN
- 0010-4825 (ISSN-L)
- OpenAlex
- W4378977505
- Cited by
- 42
- References
- 25
- Citations by year
- 2026: 13 · 2025: 11 · 2024: 12 · 2023: 6
- FWCI
- 3.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Computers in Biology and Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- NIHR
- Mark Foundation For Cancer Research
- Engineering and Physical Sciences Research Council (EP/P020259/1, EP/R511675/1)
- National Institute for Health and Care Research
- Wellcome Trust (RG98755)
- UCLH Biomedical Research Centre (BRC-1215-20014)
- CRUK (C42780/A27066)
- Cancer Research UK
- Swedish Foundation for Strategic Research (AM13-0049)
- Austrian Science Fund (J4025-B26)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- uncertainty quantification, deep learning, medical image segmentation, ensemble calibration, dropout, k-fold cross-validation, active learning (machine learning), pseudo-labeling, human-machine collaboration
- MeSH terms
- Deep Learning, Calibration, Humans, Image Processing, Computer-Assisted, Probability, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
