Uncertainty-aware deep learning in healthcare: A scoping review
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To bridge this gap, explainable AI (XAI) has emerged, providing tools to make model predictions more interpretable and, ideally, more trustworthy to reduce perceived uncertainty [48].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Computer vision models have matched or outperformed physicians for several common and essential clinical tasks, albeit in select circumstances [3,4].”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · cited 1×
“Computer vision models have matched or outperformed physicians for several common and essential clinical tasks, albeit in select circumstances [3,4].”
BackgroundMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · cited 1×
“Data extraction was performed according to a modification of CHARMS criteria, which included methods for measuring uncertainty in deep learning predictions [10].”
MethodAleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
Hüllermeier, E., & Waegeman, W. (2021) · Machine Learning · doi:10.1007/s10994-021-05946-3 · Cited by 1,834 · cited 2×
“Epistemic uncertainty can be seen as a lack of information about the best model and can be reduced by adding more training data [13].”
BackgroundLiu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421
Lederer, D. J., Bell, S. C., Branson, R. D., Chalmers, J. D., Marshall, R., Maslove, D. M., Ost, D. E., Punjabi, N. M., Schatz, M., Smyth, A. R., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201808-564ps · Cited by 715
Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., Ashrafian, H., Beam, A. L., Collins, G. S., Darzi, A., Deeks, J. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30219-3 · Cited by 486
Nair, T., Precup, D., Arnold, D. L., & Arbel, T. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101557 · Cited by 451
Research works printed but not linked: 7
- K Stubbs, 2007, Autonomy and common ground in human-robot interaction: A field study (vol 22, pg 42, 2007)., Ieee Intell Syst, 22, 3
- Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. In: Finale D-V, Jim F, Ken J, David K, Rajesh R, Byron W, et al., editors. Proceedings of the 4th Machine Learning for Healthcare Conference; Proceedings of Machine Learning Research: PMLR; 2019. p. 359–80.
- Gal Y, Ghahramani Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In: Maria Florina B, Kilian QW, editors. Proceedings of The 33rd International Conference on Machine Learning; Proceedings of Machine Learning Research: PMLR; 2016. p. 1050–9.
- M Valiuddin, 2021, Improving Aleatoric Uncertainty Quantification in Multi-annotated Medical Image Segmentation with Normalizing Flows. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis, 75
- MS Graham, 2020, Hierarchical brain parcellation with uncertainty. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis, 23
- SI Ktena, 2017, Distance metric learning using graph convolutional networks: Application to functional brain networks. International Conference on Medical Image Computing and Computer-Assisted Intervention
- A Sedghi, 2019, Probabilistic image registration via deep multi-class classification: characterizing uncertainty. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging and Clinical Image-Based Procedures, 12
Cited by
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Dojat, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102830 · Cited by 269
Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
A Survey on Uncertainty Quantification Methods for Deep Learning
He, W., Jiang, Z., Xiao, T., Xu, Z., & Li, Y. (2025) · ACM Computing Surveys · doi:10.1145/3786319 · Cited by 333
Artificial Intelligence–enabled Decision Support in Surgery: State-of-the-art and Future Directions
Loftus, T. J., Altieri, M. S., Balch, J. A., Abbott, K. L., Choi, J., Marwaha, J. S., Hashimoto, D. A., Brat, G. A., Raftopoulos, Y., Evans, H. L., et al. (2023) · Annals of Surgery · doi:10.1097/sla.0000000000005853 · Cited by 105
“52,53 There is no universally effective approach to determining adequate sample sizes for arti fi cial intelligence modeling.”
BackgroundSelf-citation
Metadata
- Authors
- Loftus, T. J., Shickel, B., Ruppert, M. M., Balch, J. A., Ozrazgat-Baslanti, T., Tighe, P. J., Efron, P. A., Hogan, W. R., Rashidi, P., Upchurch, G. R., & Bihorac, A.
- Affiliations
- University of Florida Health (US)
- University of Florida (US)
- Corresponding author
- Loftus, Tyler J.
- ORCID
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2022-08-10
- Volume
- 1
- Issue
- 8
- Pages
- e0000085
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9802673
- OpenAlex
- W4293152273
- Cited by
- 73
- References
- 35
- Citations by year
- 2026: 27 · 2025: 14 · 2024: 19 · 2023: 9 · 2022: 4
- FWCI
- 3.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of General Medical Sciences of the National Institutes of Health (K23 GM140268)
- National Institute of Diabetes and Digestive and Kidney Diseases (K01 DK120784)
- National Institute of General Medical Sciences (R01GM110240, R01GM114290, R01GM-110240, R01 GM110240)
- University of Florida Research, and Gator Trust (AWD09459)
- National Institute on Aging (R01AG121647, P30AG028740, R01AG05533)
- National Science CAREER Award (1750192)
- National Institute of Biomedical Imaging and Bioengineering (1R21EB027344)
- W. Martin Smith Interdisciplinary Patient Quality and Safety Award (IPQSA),
- Sepsis and Critical Illness Research Center Award
- University of Florida Research (AWD09458)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- uncertainty estimation, deep learning in healthcare, entrustment, Monte Carlo dropout, conformal prediction, epistemic uncertainty, diagnostic imaging, convolutional neural networks, trust in AI, reporting guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Uncertainty-aware deep learning in healthcare: A scoping review. https://app.cochat.ai/papers/10.1371/journal.pdig.0000085
- The paper
- Loftus, T. J., Shickel, B., Ruppert, M. M., Balch, J. A., Ozrazgat-Baslanti, T., Tighe, P. J., Efron, P. A., Hogan, W. R., Rashidi, P., Upchurch, G. R., & Bihorac, A. (2022). Uncertainty-aware deep learning in healthcare: A scoping review. PLOS Digital Health. https://doi.org/10.1371/journal.pdig.0000085
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

