A review of uncertainty estimation and its application in medical imaging
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
The use of AI systems in healthcare for the early screening of diseases is of great clinical importance. Deep learning has shown great promise in medical imaging, but the reliability and trustworthiness of AI systems limit their deployment in real clinical scenes, where patient safety is at stake. Uncertainty estimation plays a pivotal role in producing a confidence evaluation along with the prediction of the deep model. This is particularly important in medical imaging, where the uncertainty in the model’s predictions can be used to identify areas of concern or to provide additional information to the clinician. In this paper, we review the various types of uncertainty in deep learning, including aleatoric uncertainty and epistemic uncertainty. We further discuss how they can be estimated in medical imaging. More importantly, we review recent advances in deep learning models that incorporate uncertainty estimation in medical imaging. Finally, we discuss the challenges and future directions in uncertainty estimation in deep learning for medical imaging. We hope this review will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of uncertainty estimation models in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
The WU-Minn Human Connectome Project: An overview
Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., & Ugurbil, K. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.041 · Cited by 5,651
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Tschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Aleatory or epistemic? Does it matter?
Kiureghian, A. D., & Ditlevsen, O. (2009) · Structural Safety · doi:10.1016/j.strusafe.2008.06.020 · Cited by 2,454
Artificial intelligence in cancer imaging: Clinical challenges and applications
Bi, W. L., Hosny, A., Schabath, M. B., Giger, M. L., Birkbak, N. J., Mehrtash, A., Allison, T., Arnaout, O., Abbosh, C., Dunn, I. F., et al. (2019) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21552 · Cited by 1,846
Abràmoff, M. D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J. C., & Niemeijer, M. (2016) · Investigative Opthalmology & Visual Science · doi:10.1167/iovs.16-19964 · Cited by 1,101
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.03368 · Cited by 984
Two public chest X-ray datasets for computer-aided screening of pulmonary diseases.
(2014) · PubMed Central · doi:10.3978/j.issn.2223-4292.2014.11.20 · Cited by 918
Evaluation of prostate segmentation algorithms for mri: The promise12 challenge
Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.12.002 · Cited by 857
DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation
Roth, H. R., Lu, L., Farag, A., Shin, H.-C., Liu, J., Turkbey, E. B., & Summers, R. M. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24553-9_68 · Cited by 784
LaMontagne, P. J., Benzinger, T. L., Morris, J. C., Keefe, S., Hornbeck, R., Xiong, C., Grant, E., Hassenstab, J., Moulder, K., Vlassenko, A. G., et al. (2019) · medRxiv · doi:10.1101/2019.12.13.19014902 · Cited by 779
Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., & Vercauteren, T. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.103 · Cited by 659
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Leveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510
Nair, T., Precup, D., Arnold, D. L., & Arbel, T. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101557 · Cited by 451
A probabilistic u-net for segmentation of ambiguous images
Kohl, S. A. A., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K. H., Eslami, S. M. A., Rezende, D. J., & Ronneberger, O. (2018) · arXiv · doi:10.48550/arxiv.1806.05034 · Cited by 439
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., & Wilson, A. G. (2019) · arXiv · doi:10.48550/arxiv.1902.02476 · Cited by 429
Confidence calibration and predictive uncertainty estimation for deep medical image segmentation
Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., & Kapur, T. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3006437 · Cited by 428
Kwon, Y., Won, J.-H., Kim, B. J., & Paik, M. C. (2020) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2019.106816 · Cited by 424
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392
EyePACS: An Adaptable Telemedicine System for Diabetic Retinopathy Screening
Cuadros, J., & Bresnick, G. (2009) · Journal of Diabetes Science and Technology · doi:10.1177/193229680900300315 · Cited by 390
Wickstrøm, K., Kampffmeyer, M., & Jenssen, R. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101619 · Cited by 185
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020) · arXiv · doi:10.48550/arxiv.2002.06470 · Cited by 178
Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal OCT
Seebock, P., Orlando, J. I., Schlegl, T., Waldstein, S. M., Bogunovic, H., Klimscha, S., Langs, G., & Schmidt-Erfurth, U. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2919951 · Cited by 164
Dolezal, J. M., Srisuwananukorn, A., Karpeyev, D., Ramesh, S., Kochanny, S., Cody, B., Mansfield, A. S., Rakshit, S., Bansal, R., Bois, M. C., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34025-x · Cited by 147
DR|GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images
Araújo, T., Aresta, G., Mendonça, L., Penas, S., Maia, C., Carneiro, Â., Mendonça, A. M., & Campilho, A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101715 · Cited by 133
Reducing Uncertainty in Undersampled MRI Reconstruction With Active Acquisition
Zhang, Z., Romero, A., Muckley, M. J., Vincent, P., Yang, L., & Drozdzal, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00215 · Cited by 106
Supervised uncertainty quantification for segmentation with multiple annotations
Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., & Welling, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_16 · Cited by 105
Predictive uncertainty estimation for out-of-distribution detection in digital pathology
Linmans, J., Elfwing, S., van der Laak, J., & Litjens, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102655 · Cited by 75
Integrating uncertainty in deep neural networks for MRI based stroke analysis
Herzog, L., Murina, E., Dürr, O., Wegener, S., & Sick, B. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101790 · Cited by 66
Figini, M., Riva, M., Graham, M., Castelli, G. M., Fernandes, B., Grimaldi, M., Baselli, G., Pessina, F., Bello, L., Zhang, H., et al. (2018) · Radiology · doi:10.1148/radiol.2018180054 · Cited by 49
PHiSeg: Capturing Uncertainty in Medical Image Segmentation
Baumgartner, C. F., Tezcan, K. C., Chaitanya, K., Hötker, A. M., Muehlematter, U. J., Schawkat, K., Becker, A. S., Donati, O., & Konukoglu, E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_14 · Cited by 34
Ozdemir, O., Woodward, B., & Berlin, A. A. (2017) · arXiv · doi:10.48550/arxiv.1712.00497 · Cited by 30
Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
Tanno, R., Worrall, D., Kaden, E., Ghosh, A., Grussu, F., Bizzi, A., Sotiropoulos, S. N., Criminisi, A., & Alexander, D. C. (2019) · arXiv · doi:10.48550/arxiv.1907.13418 · Cited by 30
As Easy as 1, 2...4? Uncertainty in Counting Tasks for Medical Imaging
Eaton-Rosen, Z., Varsavsky, T., Ourselin, S., & Cardoso, M. J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_39 · Cited by 14
Gantenbein, M., Erdil, E., & Konukoglu, E. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60365-6_2 · Cited by 10
Cited by
Kanber, B. M., Gou, S., Liu, B., Noaman, N. F., Mukred, M., & Al Smadi, A. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134426
Metadata
- Authors
- Zou, K., Chen, Z., Yuan, X., Shen, X., Wang, M., & Fu, H.
- Venue
- Meta-Radiology
- Publisher
- Elsevier BV
- Published
- 2023-06
- Volume
- 1
- Issue
- 1
- Pages
- 100003
- ISSN
- 2950-1628 (ISSN-L)
- OpenAlex
- W4383377291
- Cited by
- 158
- References
- 47
- FWCI
- 5.47
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.