The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation
Open accessAbstract
Fully convolutional neural networks (FCNs), and in particular U-Nets, have achieved state-of-the-art results in semantic segmentation for numerous medical imaging applications. Moreover, batch normalization and Dice loss have been used successfully to stabilize and accelerate training. However, these networks are poorly calibrated i.e. they tend to produce overconfident predictions for both correct and erroneous classifications, making them unreliable and hard to interpret. In this paper, we study predictive uncertainty estimation in FCNs for medical image segmentation. We make the following contributions: 1) We systematically compare cross-entropy loss with Dice loss in terms of segmentation quality and uncertainty estimation of FCNs; 2) We propose model ensembling for confidence calibration of the FCNs trained with batch normalization and Dice loss; 3) We assess the ability of calibrated FCNs to predict segmentation quality of structures and detect out-of-distribution test examples. We conduct extensive experiments across three medical image segmentation applications of the brain, the heart, and the prostate to evaluate our contributions. The results of this study offer considerable insight into the predictive uncertainty estimation and out-of-distribution detection in medical image segmentation and provide practical recipes for confidence calibration. Moreover, we consistently demonstrate that model ensembling improves confidence calibration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Natekar al. [694] | et | 2020 | DenseUnet, ResUnet, SimUnet | Segmentation | Brain tumor | 1 | TTD (test time dropout) for VI”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [694]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Increasingly, disagreement among these CV models is reused as a proxy for epistemic uncertainty in downstream applications such as quality control and human-in-the-loop review [11,18,15].”
From Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation (2026) · cites this paper as [18]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_49 · Cited by 7,945
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Freymann, J. B., Farahani, K., & Davatzikos, C. (2017) · Scientific Data · doi:10.1038/sdata.2017.117 · Cited by 3,049
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917
Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., & Jorge Cardoso, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_28 · Cited by 2,829
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1610.02136 · Cited by 1,968
Enhancing The Reliability of Out-of-distribution Image Detection in Neural NetworksPreprint
Liang, S., Li, Y., & Srikant, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02690 · Cited by 1,463
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991
Evaluation of prostate segmentation algorithms for mri: The promise12 challenge
Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.12.002 · Cited by 857
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution SamplesPreprint
Lee, K., Lee, H., Lee, K., & Shin, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09325 · Cited by 650
Learning Confidence for Out-of-Distribution Detection in Neural NetworksPreprint
DeVries, T., & Taylor, G. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04865 · Cited by 567
Leveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510
Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation
Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M. D., Pawlowski, N., Rajchl, M., Lee, M., Kainz, B., Rueckert, D., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_38 · Cited by 506
Computer-Aided Detection of Prostate Cancer in MRI
Litjens, G. J. S., Debats, O. A., Barentsz, J. O., Karssemeijer, N., & Huisman, H. J. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2303821 · Cited by 453
A probabilistic u-net for segmentation of ambiguous images
Kohl, S. A. A., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K. H., Eslami, S. M. A., Rezende, D. J., & Ronneberger, O. (2018) · arXiv · doi:10.48550/arxiv.1806.05034 · Cited by 439
Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation
Ghafoorian, M., Mehrtash, A., Kapur, T., Karssemeijer, N., Marchiori, E., Pesteie, M., Guttmann, C. R. G., de Leeuw, F., Tempany, C. M., van Ginneken, B., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_59 · Cited by 337
Self-citationKendall, A., Badrinarayanan, V., & Cipolla, R. (2015) · arXiv · doi:10.48550/arxiv.1511.02680 · Cited by 292
Kuijf, H. J., Casamitjana, A., Collins, D. L., Dadar, M., Georgiou, A., Ghafoorian, M., Jin, D., Khademi, A., Knight, J., Li, H., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2905770 · Cited by 291
Self-citationLearning From Noisy Labels by Regularized Estimation of Annotator Confusion
Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D. C., & Silberman, N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01150 · Cited by 273
Bayesian Uncertainty Estimation for Batch Normalized Deep NetworksPreprint
Teye, M., Azizpour, H., & Smith, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06455 · Cited by 199
Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation
Jungo, A., & Reyes, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_6 · Cited by 156
Fidon, L., Li, W., García-Peraza-Herrera, L. C., Ekanayake, J., Kitchen, N. D., Sebastien Ourselin, A., & Vercauteren, T. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_6 · Cited by 155
Automatic Segmentation and Disease Classification Using Cardiac Cine MR Images
Wolterink, J. M., Leiner, T., Viergever, M. A., & Išgum, I. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75541-0_11 · Cited by 144
Karimi, D., Zeng, Q., Mathur, P., Avinash, A., Mahdavi, S. S., Spadinger, I., Abolmaesumi, P., & Salcudean, S. E. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.07.005 · Cited by 127
Self-citationSupervised uncertainty quantification for segmentation with multiple annotations
Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., & Welling, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_16 · Cited by 105
Seo, S., Seo, P. H., & Han, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00924 · Cited by 73
Out-of-Distribution Detection using Multiple Semantic Label RepresentationsPreprint
Shalev, G., Adi, Y., & Keshet, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06664 · Cited by 70
Mehrtash, A., Ghafoorian, M., Pernelle, G., Ziaei, A., Heslinga, F. G., Tuncali, K., Fedorov, A., Kikinis, R., Tempany, C. M., Wells, W. M., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2876796 · Cited by 69
Self-citationKwon, Y., Won, J., Kim, B. J., & Paik, M. C. (2018) · Cited by 67
Indrayan, A. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b12662 · Cited by 63
Sedai, S., Antony, B. J., Mahapatra, D., & Garnavi, R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00949-6_26 · Cited by 56
Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI
Sander, J., de Vos, B. D., Wolterink, J. M., & Išgum, I. (2019) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2511699 · Cited by 53
Uncertainty-Aware Attention for Reliable Interpretation and Prediction
Huh, J. (2019) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 41
PHiSeg: Capturing Uncertainty in Medical Image Segmentation
Baumgartner, C. F., Tezcan, K. C., Chaitanya, K., Hötker, A. M., Muehlematter, U. J., Schawkat, K., Becker, A. S., Donati, O., & Konukoglu, E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_14 · Cited by 34
Rottmann, M., & Schubert, M. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00176 · Cited by 32
Automatic high resolution segmentation of the prostate from multi-planar MRI
Meyer, A., Mehrtash, A., Rak, M., Schindele, D., Schostak, M., Tempany, C. M., Kapur, T., Abolmaesumi, P., Fedorov, A., & Hansen, C. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363549 · Cited by 29
Self-citationOptimization with Soft Dice Can Lead to a Volumetric Bias
Bertels, J., Robben, D., Vandermeulen, D., & Suetens, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46640-4_9 · Cited by 27
Niethammer, M., Pohl, K. M., Janoos, F., & Wells, W. M. (2017) · SIAM Journal on Imaging Sciences · doi:10.1137/16m1058601 · Cited by 6
Self-citationVERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“Negative log-likelihood (NLL), and Brier score [49], are both strictly proper scoring rules that have been previously used in several studies for evaluating predictive uncertainty [5], [18], [27].”
Background
Flagged first, then by citations
Research works printed but not linked: 10
- 2018, Tgs Salt Identification Challenge
- 2018, Grand Challenge on MR Brain Segmentation at MICCAI 2018
- 10.5555/3045118.3045167 · DOI
- 10.21236/ADA459827 · DOI
- 10.5555/3045390.3045502 · DOI
- Liao, 2019, On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2D echocardiography quality assessment, arXiv:1911.00674 · arXiv
- 10.1109/CVPR.2018.00464 · DOI
- 10.1201/9781420002690 · DOI
- 10.5555/3295222.3295309 · DOI
- 10.1007/978-3-030-32251-9_73 · DOI
Cited by
Sorted by citations
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications
Siddique, N. A., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021) · IEEE Access · doi:10.1109/access.2021.3086020 · Cited by 2,185
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,670
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M. A., Thai, T., Moore, K. N., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,180
TransMorph: Transformer for unsupervised medical image registration
Chen, J., Frey, E. C., He, Y., Segars, W. P., Li, Y., & Du, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102615 · Cited by 883
Metadata
- Authors
- Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., & Kapur, T.
- Affiliations
- Brigham and Women's Hospital (US)
- University of British Columbia (CA)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-12
- Volume
- 39
- Issue
- 12
- Pages
- 3868-3878
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- PMC
- PMC7704933
- OpenAlex
- W2991523350
- Cited by
- 428
- References
- 55
- Citations by year
- 2026: 91 · 2025: 92 · 2024: 71 · 2023: 71 · 2022: 56 · 2021: 41 · 2020: 6
- FWCI
- 16.91
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. National Institutes of Health (P41EB015898)
- Natural Sciences and Engineering Research Council (NSERC) of Canada
- Canadian Institutes of Health Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- confidence calibration, predictive uncertainty estimation, medical image segmentation, fully convolutional network, U-Net, Dice loss, cross-entropy loss, model ensembling, out-of-distribution detection, batch normalization, brain segmentation, cardiac image segmentation, prostate segmentation
- MeSH terms
- Brain, Calibration, Humans, Image Processing, Computer-Assisted, Male, Neural Networks, Computer, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
