CoChat
Sign Up
Cited by 239Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareAdversarial Robustness in Machine Learning

Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013–2023)

Open access
Seoni, Silvia, Jahmunah, Vicnesh, Salvi, Massimo, Barua, Prabal Datta, Molinari, Filippo, Acharya, U. Rajendra·2023·Computers in Biology and Medicine
DOI 10.1016/j.compbiomed.2023.107441
Open this paper in CoChat — free account

Abstract

Uncertainty estimation in healthcare involves quantifying and understanding the inherent uncertainty or variability associated with medical predictions, diagnoses, and treatment outcomes. In this era of Artificial Intelligence (AI) models, uncertainty estimation becomes vital to ensure safe decision-making in the medical field. Therefore, this review focuses on the application of uncertainty techniques to machine and deep learning models in healthcare. A systematic literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our analysis revealed that Bayesian methods were the predominant technique for uncertainty quantification in machine learning models, with Fuzzy systems being the second most used approach. Regarding deep learning models, Bayesian methods emerged as the most prevalent approach, finding application in nearly all aspects of medical imaging. Most of the studies reported in this paper focused on medical images, highlighting the prevalent application of uncertainty quantification techniques using deep learning models compared to machine learning models. Interestingly, we observed a scarcity of studies applying uncertainty quantification to physiological signals. Thus, future research on uncertainty quantification should prioritize investigating the application of these techniques to physiological signals. Overall, our review highlights the significance of integrating uncertainty techniques in healthcare applications of machine learning and deep learning models. This can provide valuable insights and practical solutions to manage uncertainty in real-world medical data, ultimately improving the accuracy and reliability of medical diagnoses and treatment recommendations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.compbiomed.2023.107441

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Healthcare providers who use AI may be integrating inaccurate data into the decision-making process.

    “Therefore, healthcare providers who use AI, may be integrating inaccurate data into the decision-making process (Seoni, 2023).”

    From Procedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks (2026) · cites this paper as Seoni, 2023

    The Discussion (p. 14) warns that AI model decisions may be unreliable and that uncertainty estimates are needed for safe medical decision-making. The Conclusion (p. 19) also discusses reducing misclassifications. These passages support a risk of inaccurate AI outputs affecting decisions, but do not specifically say that providers integrate inaccurate data into their decision-making. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Predictive AI models are not clear and may be based on highly variable data, including inaccurate diagnoses, errors with data entry, and inconsistencies in how healthcare measurements are derived and reported.

    “Finally, the predictive AI models are not clear and may be based on highly variable data, including inaccurate diagnoses, errors with data entry, and inconsistencies with how healthcare measurements are derived and reported based on human variability (e.g., blood pressure, heart rate, etc.; Seoni, 2023).”

    From Procedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks (2026) · cites this paper as Seoni, 2023

    The Introduction (p. 3) says predictive AI uncertainty can arise from flawed models and from incomplete, noisy, or conflicting data. The review also describes inter-observer variability in assessing cardiovascular images (Section 3.2.1, p. 9). These passages support the general concern, but they do not substantiate the cited sentence’s specific examples of inaccurate diagnoses, data-entry errors, or human variability in blood-pressure and heart-rate measurements. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Lack of transparency in how AI identifies and synthesizes information introduces potential biases into clinical decision-making and distrust.

    “This introduces potential biases into the clinical decision-making process and distrust with lack of transparency in this process (Nilsen et al., 2024;Seoni, 2023).”

    From Procedure to Evaluate Artificial Intelligence (AI) Output for Educational and Clinical Tasks (2026) · cites this paper as Seoni, 2023

    The Discussion (p. 14) says explainability alone cannot guarantee reliable model decisions and recommends uncertainty estimates for safe medical decision-making. The Conclusion (p. 19) also addresses AI reliability and safety, but neither passage says that lack of transparency introduces bias into clinical decisions or causes distrust; searches for those terms found no supporting passage. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 193

  1. [1][1] I.H. Sarker, Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions, SN Comput. Sci. 2 (6) (2021) 420, https:// doi.org/10.1007/s42979-021-00815-1. - · DOI
  2. [2][2] L. Wells, T. Bednarz, Explainable AI and reinforcement learning -a systematic review of current approaches and trends, Front. Artif. Intell. 4 (2021), https:// doi.org/10.3389/frai.2021.550030. - · DOI
  3. [3][3] [D. Seu ß , Bridging the Gap between Explainable AI and Uncertainty Quantification to Enhance Trustability, 2021, pp. 1 -10.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref3) -
  4. [4][4] J. Gawlikowski, et al., A Survey of Uncertainty in Deep Neural Networks, 2021, pp. 1 -41. -
  5. [5][5] [D. Amodei, C. Olah, J. Steinhardt, P.F. Christiano, J. Schulman, D. Man ´ e, Concrete problems in AI safety, ArXiv abs/1606.0 (2016).](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref5) -
  6. [6][6] M. Ru ß wurm, M. Ali, X.X. Zhu, Y. Gal, M. K ¨ orner, Model and data uncertainty for satellite time series forecasting with deep recurrent models, in: IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium, 2020, pp. 7025 -7028, https://doi.org/10.1109/IGARSS39084.2020.9323890. - · DOI
  7. [7][7] [Y. Gal, R. Islam, Z. Ghahramani, Deep Bayesian active learning with image data, in: 34th International Conference on Machine Learning, ICML 2017 vol. 3, 2017, pp. 1923 -1932.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref7) -
  8. [9][9] A. Der Kiureghian, O. Ditlevsen, Aleatory or epistemic? Does it matter? Struct. Saf. 31 (2) (2009) 105 -112, https://doi.org/10.1016/j.strusafe.2008.06.020. - · DOI · Paper page
  9. [11][11] [A. Malinin, M.J.F. Gales, Uncertainty Estimation in Deep Learning with Application to Spoken Language Assessment, May, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref11) -
  10. [12][12] [A. Ashukha, A. Lyzhov, D. Molchanov, D. Vetrov, Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning, 2020, pp. 1 -30.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref12) S. Seoni et al. -
  11. [15][15] [S. Liang, Y. Li, R. Srikant, Enhancing the reliability of out-of-distribution image detection in neural networks, in: 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, 2018, pp. 1 -15.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref15) -
  12. [16][16] H. Broekhuizen, C.G.M. Groothuis-Oudshoorn, J.A. van Til, J.M. Hummel, M. J. IJzerman, A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions, Pharmacoeconomics 33 (5) (May 28, 2015) 445 -455, https://doi.org/10.1007/ s40273-014-0251-x. Springer International Publishing. -
  13. [17][17] [B. Lambert, F. Forbes, A. Tucholka, S. Doyle, H. Dehaene, M. Dojat, Trustworthy Clinical AI Solutions: a Unified Review of Uncertainty Quantification in Deep Learning Models for Medical Image Analysis, Oct. 2022.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref17) -
  14. [18][18] T.J. Loftus, et al., Uncertainty-aware deep learning in healthcare: a scoping review, PLOS Digit. Health 1 (8) (Aug. 2022), e0000085, https://doi.org/ 10.1371/journal.pdig.0000085. - · DOI · Paper page
  15. [21][21] E. Ocampo, M. Maceiras, S. Herrera, C. Maurente, D. Rodríguez, M.A. Sicilia, Comparing Bayesian inference and case-based reasoning as support techniques in the diagnosis of Acute Bacterial Meningitis, Expert Syst. Appl. 38 (8) (2011) 10343 -10354, https://doi.org/10.1016/j.eswa.2011.02.055. - · DOI
  16. [22][22] [Mooney, Z. Christopher, Monte Carlo Simulation, Sage, 1997.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref22) -
  17. [23][23] D. Karaboga, E. Kaya, Adaptive network based fuzzy inference system (ANFIS) training approaches: a comprehensive survey, Artif. Intell. Rev. 52 (4) (2019) 2263 -2293, https://doi.org/10.1007/s10462-017-9610-2. - · DOI
  18. [24][24] T. Den œ ux, 40 years of Dempster-Shafer theory, Int. J. Approx. Reason. 79 (Dec. 2016) 1 -6, https://doi.org/10.1016/J.IJAR.2016.07.010. - · DOI
  19. [26][26] [T. Augustin, F.P. Coolen, G. De Cooman, M.C. Troffaes, Introduction to Imprecise Probabilities, John Wiley & Sons, 2014.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref26) -
  20. [27][27] H.-C. Lin, et al., Bayesian inference of nonylphenol exposure for assessing human dietary risk, Sci. Total Environ. 713 (2020), 136710, https://doi.org/10.1016/j. scitotenv.2020.136710. - · DOI
  21. [28][28] Q. Zhou, T. Yu, X. Zhang, J. Li, Bayesian inference and uncertainty quantification for medical image reconstruction with Poisson data, SIAM J. Imag. Sci. 13 (1) (Jan. 2020) 29 -52, https://doi.org/10.1137/19M1248352. - · DOI
  22. [29][29] E. Akkoyun, S.T. Kwon, A.C. Acar, W. Lee, S. Baek, Predicting abdominal aortic aneurysm growth using patient-oriented growth models with two-step Bayesian inference, Comput. Biol. Med. 117 (C) (Feb. 2020), https://doi.org/10.1016/j. compbiomed.2020.103620. - · DOI
  23. [30][30] B.P. Magnusson, H. Schmidli, N. Rouyrre, D.O. Scharfstein, Bayesian inference for a principal stratum estimand to assess the treatment effect in a subgroup characterized by post-randomization events, Stat. Med. 38 (23) (2019) 4761 -4771, https://doi.org/10.1002/sim.8333. - · DOI
  24. [31][31] J. Lipkova, et al., Personalized radiotherapy design for glioblastoma: integrating mathematical tumor models, multimodal scans, and Bayesian inference, IEEE Trans. Med. Imag. 38 (8) (2019) 1875 -1884, https://doi.org/10.1109/ TMI.2019.2902044. -
  25. [32][32] [S. Flügge, S. Zimmer, U. Petersohn, Knowledge Representation and Diagnostic Inference Using Bayesian Networks in the Medical Discourse, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref32) -
  26. [33][33] K.-J. Wang, J.-L. Chen, K.-M. Wang, Medical expenditure estimation by Bayesian network for lung cancer patients at different severity stages, Comput. Biol. Med. 106 (Mar. 2019) 97 -105, https://doi.org/10.1016/j.compbiomed.2019.01.015. - · DOI
  27. [34][34] M.V. Salgado, et al., Projected impact of a reduction in sugar-sweetened beverage consumption on diabetes and cardiovascular disease in Argentina: a modeling study, PLoS Med. 17 (7) (Jul. 2020), e1003224, https://doi.org/10.1371/ JOURNAL.PMED.1003224. -
  28. [35][35] M.Y. Tsai, et al., A new open-source GPU-based microscopic Monte Carlo simulation tool for the calculations of DNA damages caused by ionizing radiation - Part I: core algorithm and validation, Med. Phys. 47 (4) (Apr. 2020) 1958 -1970, https://doi.org/10.1002/MP.14037. - · DOI
  29. [36][36] Y. Lee, Preliminary evaluation of dual-head Compton camera with Si/CZT material for breast cancer detection: Monte Carlo simulation study, Optik 202 (Feb. 2020), https://doi.org/10.1016/J.IJLEO.2019.163519. - · DOI
  30. [37][37] T.Y. Shih, Y.L. Liu, H.H. Chen, J. Wu, Dose evaluation of a blood irradiator using Monte Carlo simulation and MAGAT gel dosimeter, Nucl. Instrum. Methods Phys. Res. 954 (Feb. 2020), https://doi.org/10.1016/J.NIMA.2018.09.084. - · DOI
  31. [38][38] A. Gasparini, et al., Mixed-effects models for health care longitudinal data with an informative visiting process: a Monte Carlo simulation study, Stat. Neerl. 74 (1) (Feb. 2020) 5 -23, https://doi.org/10.1111/STAN.12188. - · DOI
  32. [39][39] C. Lee, K.P. Kim, D.J. Long, W.E. Bolch, Organ doses for reference pediatric and adolescent patients undergoing computed tomography estimated by Monte Carlo simulation, Med. Phys. 39 (4) (2012) 2129 -2146, https://doi.org/10.1118/ 1.3693052. -
  33. [40][40] G. Castellazzi, et al., A machine learning approach for the differential diagnosis of alzheimer and vascular dementia fed by MRI selected features, Front. Neuroinf. 14 (Jun. 2020), https://doi.org/10.3389/fninf.2020.00025. - · DOI · Paper page
  34. [41][41] H. Das, B. Naik, H.S. Behera, Medical disease analysis using neuro-fuzzy with feature extraction model for classification, Inform. Med. Unlocked 18 (Jan. 2020), https://doi.org/10.1016/j.imu.2019.100288. - · DOI
  35. [42][42] K. Vidhya, R. Shanmugalakshmi, Modified adaptive neuro-fuzzy inference system (M-ANFIS) based multi-disease analysis of healthcare Big Data, J. Supercomput. 76 (11) (Nov. 2020) 8657 -8678, https://doi.org/10.1007/s11227-019-03132-w. - · DOI
  36. [43][43] R. Kaur, K. Kaur, A. Khamparia, D. Anand, An improved and adaptive approach in ANFIS to predict knee diseases, Int. J. Healthc. Inf. Syst. Inf. 15 (2) (Apr. 2020) 22 -37, https://doi.org/10.4018/IJHISI.2020040102. - · DOI
  37. [44][44] K. Liu, et al., Big medical data decision-making intelligent system exploiting fuzzy inference logic for prostate cancer in developing countries, IEEE Access 7 (2019) 2348 -2363, https://doi.org/10.1109/ACCESS.2018.2886198. - · DOI
  38. [45][45] I.B. de Medeiros, M.A.S. Machado, W.J. Damasceno, A.M. Caldeira, R. C. dos Santos, J.B. da Silva Filho, A fuzzy inference system to support medical diagnosis in real time, Procedia Comput. Sci. 122 (2017) 167 -173, https://doi. org/10.1016/j.procs.2017.11.356. - · DOI
  39. [46][46] T. Nguyen, A. Khosravi, D. Creighton, S. Nahavandi, Medical data classification using interval type-2 fuzzy logic system and wavelets, Appl. Soft Comput. J. 30 (2015) 812 -822, https://doi.org/10.1016/j.asoc.2015.02.016. - · DOI
  40. [47][47] M.L.C. Buono, N. Pandiangan, S.H.D. Loppies, The implementation of an expert system in diagnosing skin diseases using the dempster-shafer method, J. Phys. Conf. Ser. 1569 (Jul. 2020) 2, https://doi.org/10.1088/1742-6596/1569/2/ 022028. - · DOI
  41. [48][48] [E.A. Prameswari, A. Triayudi, I.D. Sholihati, Web-based E-Diagnostic for Digestive System Disorders in Mumans Using the Demster Shafer Method, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref48) -
  42. [49][49] S. Razi, M.R. Karami Mollaei, J. Ghasemi, A novel method for classification of BCI multi-class motor imagery task based on Dempster -Shafer theory, Inf. Sci. 484 (May 2019) 14 -26, https://doi.org/10.1016/J.INS.2019.01.053. - · DOI
  43. [50][50] J.Y. Shi, X.Q. Shang, K. Gao, S.W. Zhang, S.M. Yiu, An integrated local classification model of predicting drug-drug interactions via dempster-shafer theory of evidence, Sci. Rep. 8 (Dec. 2018) 1, https://doi.org/10.1038/S41598018-30189-Z. - · DOI
  44. [51][51] B. Kang, G. Chhipi-Shrestha, Y. Deng, J. Mori, K. Hewage, R. Sadiq, Development of a predictive model for Clostridium difficile infection incidence in hospitals using Gaussian mixture model and Dempster -Shafer theory, Stoch. Environ. Res. Risk Assess. 32 (6) (2018) 1743 -1758, https://doi.org/10.1007/s00477-0171459-z. - · DOI
  45. [52][52] J. Wang, Y. Hu, F. Xiao, X. Deng, Y. Deng, A novel method to use fuzzy soft sets in decision making based on ambiguity measure and Dempster-Shafer theory of evidence: an application in medical diagnosis, Artif. Intell. Med. 69 (May 2016) 1 -11, https://doi.org/10.1016/J.ARTMED.2016.04.004. - · DOI
  46. [53][53] F.C. Ghesu, et al., Quantifying and leveraging predictive uncertainty for medical image assessment [Online]. Available: http://arxiv.org/abs/2007.04258, Jul. 2020. - · arXiv
  47. [54][54] B. Biswas, S.K. Ghosh, S. Bhattacharyya, J. Platos, V. Snasel, A. Chakrabarti, Chest X-ray enhancement to interpret pneumonia malformation based on fuzzy soft set and Dempster-Shafer theory of evidence, Appl. Soft Comput. J. 86 (Jan. 2020), https://doi.org/10.1016/J.ASOC.2019.105889. - · DOI
  48. [55][55] S. Porebski, P. Porwik, E. Straszecka, T. Orczyk, Liver fibrosis diagnosis support using the Dempster-Shafer theory extended for fuzzy focal elements, Eng. Appl. Artif. Intell. 76 (Nov. 2018) 67 -79, https://doi.org/10.1016/j. engappai.2018.09.004. - · DOI
  49. [56][56] F. Xiao, A hybrid fuzzy soft sets decision making method in medical diagnosis, IEEE Access 6 (2018) 25300 -25312, https://doi.org/10.1109/ ACCESS.2018.2820099. -
  50. [57][57] J. Ghasemi, R. Ghaderi, M.R.K. Mollaei, S.A. Hojjatoleslami, A novel fuzzy Dempster-Shafer inference system for brain MRI segmentation, undefined 223 (Feb. 2013) 205 -220, https://doi.org/10.1016/J.INS.2012.08.026. - · DOI
  51. [58][58] Z. Li, G. Wen, N. Xie, An approach to fuzzy soft sets in decision making based on grey relational analysis and Dempster-Shafer theory of evidence: an application in medical diagnosis, Artif. Intell. Med. 64 (3) (Jul. 2015) 161 -171, https://doi. org/10.1016/J.ARTMED.2015.05.002. - · DOI
  52. [59][59] K. A. P, D.P. Acharjya, A hybrid scheme for heart disease diagnosis using rough set and cuckoo search technique, J. Med. Syst. 44 (1) (Jan. 2020), https://doi. org/10.1007/s10916-019-1497-9. - · DOI
  53. [60][60] D. Santra, S.K. Basu, J.K. Mandal, S. Goswami, Rough set based lattice structure for knowledge representation in medical expert systems: low back pain management case study, Expert Syst. Appl. 145 (May 2020), https://doi.org/ 10.1016/J.ESWA.2019.113084. - · DOI
  54. [61][61] R.K. Bania, A. Halder, R-Ensembler: A greedy rough set based ensemble attribute selection algorithm with kNN imputation for classification of medical data, ' , Comput. Methods Progr. Biomed. 184 (Feb. 2020) https://doi.org/10.1016/J. CMPB.2019.105122. - · DOI
  55. [62][62] [C. Blake, UCI Repository of Machine Learning Databases, 1998 ics. uci. edu/~ mlearn/MLRepository. html.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref62) -
  56. [63][63] Q.Y. Jiang, X.J. Yang, X.S. Sun, An aided diagnosis model of sub-health based on rough set and fuzzy mathematics: a case of TCM, J. Intell. Fuzzy Syst. 32 (6) (2017) 4135 -4143, https://doi.org/10.3233/JIFS-15958. - · DOI
  57. [64][64] P. Giustinelli, C.F. Manski, F. Molinari, Precise or imprecise probabilities? Evidence from survey response related to late-onset dementia, J. Eur. Econ. Assoc. 20 (1) (Feb. 2022) 187 -221, https://doi.org/10.1093/jeea/jvab023. - · DOI
  58. [65][65] M.T. McKenna, J.A. Weis, A. Brock, V. Quaranta, T.E. Yankeelov, Precision medicine with imprecise therapy: computational modeling for chemotherapy in breast cancer, Transl. Oncol. 11 (3) (Jun. 2018) 732 -742, https://doi.org/ 10.1016/J.TRANON.2018.03.009. S. Seoni et al. - · DOI
  59. [66][66] [A.M. Mahmoud, Suitability of various intelligent tree based classifiers for diagnosing noisy medical data, Egypt. Comput. Sci. J. 40 (2) (2016).](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref66) -
  60. [67][67] [A. Kendall, Y. Gal, What uncertainties do we need in Bayesian deep learning for computer vision?, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, in NIPS ' 17 Curran Associates Inc., Red Hook, NY, USA, 2017, pp. 5580 -5590.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref67) -
  61. [68][68] [A. Malinin, M. Gales, Predictive uncertainty estimation via prior networks, Adv. Neural Inf. Process. Syst. 2018 (2018) 7047 -7058. Decem, no. NeurIPS.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref68) -
  62. [70][70] Neal, M. Radford, Bayesian learning for neural networks, Springer Sci. Bus. Media 118 (2012). -
  63. [71][71] J. Maro ˜ nas, R. Paredes, and D. Ramos, ' Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks ' , doi: 10.1016/j. neucom.2020.04.103.. - · DOI
  64. [76][76] [K. Lee, H. Lee, K. Lee, J. Shin, Training confidence-calibrated classifiers for detecting out-of-distribution samples, 6th Int. Conf. Learn. Representations, ICLR 2018 Conf. Track Proc. (2018) 1 -16.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref76) -
  65. [77][77] [S. Thulasidasan, G. Chennupati, J. Bilmes, T. Bhattacharya, S. Michalak, On mixup training: improved calibration and predictive uncertainty for deep neural networks, Adv. Neural Inf. Process. Syst. 32 (2019) 1 -15. NeurIPS.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref77) -
  66. [78][78] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2818 -2826, https://doi.org/10.1109/ CVPR.2016.308. -
  67. [79][79] [B. Zadrozny, C. Elkan, Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers, Icml (March 2013) 1 -8, 2001.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref79) -
  68. [80][80] [J. Wenger, H. Kjellstr ¨ om, R. Triebel, Non-Parametric Calibration for Classification, 108, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref80) -
  69. [82][82] [B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles, Adv. Neural Inf. Process. Syst. 2017 (2017) 6403 -6414. Decem, no. Nips.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref82) -
  70. [83][83] [P. Izmailov, W.J. Maddox, P. Kirichenko, T. Garipov, D. Vetrov, A.G. Wilson, Subspace inference for Bayesian deep learning, in: 35th Conference on Uncertainty in Artificial Intelligence, UAI 2019, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref83) -
  71. [84][84] P. Oberdiek, M. Rottmann, H. Gottschalk, Classification uncertainty of deep neural networks based on gradient information, Lect. Notes Comput. Sci. 11081 LNAI (2018) 113 -125, https://doi.org/10.1007/978-3-319-99978-4\_9. - · DOI
  72. [85][85] J. Lee, G. Alregib, Gradients as a measure of uncertainty in neural networks, in: Proceedings - International Conference on Image Processing, ICIP, Vol. 2020Octob, 2020, pp. 2416 -2420, https://doi.org/10.1109/ ICIP40778.2020.9190679. -
  73. [86][86] [M. Raghu, et al., Direct uncertainty prediction for medical second opinions, in: 36th International Conference on Machine Learning, ICML 2019, 2019, pp. 9202 -9211, 2019-June.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref86) -
  74. [87][87] T. Ramalho, M. Miranda, Density estimation in representation space to predict model uncertainty, Commun. Comput. Inf. Sci. 1272 (2020) 84 -96, https://doi. org/10.1007/978-3-030-62144-5\_7. - · DOI
  75. [88][88] [M. Sensoy, L. Kaplan, M. Kandemir, Evidential deep learning to quantify classification uncertainty, Adv. Neural Inf. Process. Syst. (2018) 3179 -3189, 2018-Decem, no. Nips.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref88) -
  76. [89][89] S.I. Ktena, et al., Distance metric learning using graph convolutional networks: application to functional brain networks, Lect. Notes Comput. Sci. 10433 LNCS (2017) 469 -477, https://doi.org/10.1007/978-3-319-66182-7\_54. - · DOI
  77. [90][90] R. McKinley, et al., Simultaneous lesion and brain segmentation in multiple sclerosis using deep neural networks, Sci. Rep. 11 (1) (2021) 1087, https://doi. org/10.1038/s41598-020-79925-4. - · DOI
  78. [91][91] [T. DeVries, G.W. Taylor, Leveraging Uncertainty Estimates for Predicting Segmentation Quality, 2018.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref91) -
  79. [93][93] F.C. Ghesu, et al., Quantifying and leveraging classification uncertainty for chest radiograph assessment, Lect. Notes Comput. Sci. 11769 LNCS (2019) 676 -684, https://doi.org/10.1007/978-3-030-32226-7\_75. - · DOI
  80. [94][94] M.S. Graham, et al., Hierarchical brain parcellation with uncertainty, Lect. Notes Comput. Sci. 12443 LNCS (2020) 23 -31, https://doi.org/10.1007/978-3-03060365-6\_3. - · DOI
  81. [96][96] [X. Li, Y. Zhou, N.C. Dvornek, Y. Gu, P. Ventola, J.S. Duncan, in: A.L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M.A. Zuluaga, S.K. Zhou, D. Racoceanu, L. Joskowicz (Eds.), Efficient Shapley Explanation for Features Importance Estimation under Uncertainty BT - Medical Image Computing and Computer Assisted Intervention -MICCAI 2020, Springer International Publishing, Cham, 2020, pp. 792 -801.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref96) -
  82. [97][97] C. Ye, Y. Li, X. Zeng, An improved deep network for tissue microstructure estimation with uncertainty quantification, Med. Image Anal. 61 (2020), https:// doi.org/10.1016/j.media.2020.101650. - · DOI
  83. [98][98] [M. Tardy, B. Scheffer, D. Mateus, in: D. Shen, T. Liu, T.M. Peters, L.H. Staib, C. Essert, S. Zhou, P.-T. Yap, A. Khan (Eds.), Uncertainty Measurements for the Reliable Classification of Mammograms BT - Medical Image Computing and Computer Assisted Intervention -MICCAI 2019, Springer International Publishing, Cham, 2019, pp. 495 -503.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref98) -
  84. [99][99] [M.H. Jensen, D.R. J ø rgensen, R. Jalaboi, M.E. Hansen, M.A. Olsen, in: D. Shen, T. Liu, T.M. Peters, L.H. Staib, C. Essert, S. Zhou, P.-T. Yap, A. Khan (Eds.), Improving Uncertainty Estimation in Convolutional Neural Networks Using Interrater Agreement BT - Medical Image Computing and Computer Assisted Intervention -MICCAI 2019, Springer International Publishing, Cham, 2019, pp. 540 -548.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref99) -
  85. [102][102] G.E. Hinton, D. van Camp, Keeping Neural Networks Simple by Minimizing the Description Length of the Weights, 1993, pp. 5 -13, https://doi.org/10.1145/ 168304.168306. -
  86. [103][103] [MethodRadford Carlo, NealTechnical, Bayesian Training of Backpropagation Networks by the Hybrid Monte, 1993.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref103) -
  87. [104][104] [J.S. Denker, Y. LeCun, Transforming neural-net output levels to probability distributions, Adv. Neural Inf. Process. Syst. 3 (1991) 853 -859.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref104) -
  88. [105][105] C. Leibig, V. Allken, M.S. Ayhan, P. Berens, S. Wahl, Leveraging uncertainty information from deep neural networks for disease detection, Sci. Rep. 7 (1) (2017) 1 -14, https://doi.org/10.1038/s41598-017-17876-z. - · DOI · Paper page
  89. [107][107] A. Jungo, et al., Towards uncertainty-assisted brain tumor segmentation and survival prediction, Lect. Notes Comput. Sci. 10670 LNCS (2018) 474 -485, https://doi.org/10.1007/978-3-319-75238-9\_40. - · DOI · Paper page
  90. [108][108] A. Jungo, et al., On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation, Lect. Notes Comput. Sci. 11070 LNCS (2018) 682 -690, https://doi.org/10.1007/978-3-030-00928-1\_77. - · DOI
  91. [109][109] J.I. Orlando, et al., U2-net: a Bayesian u-net model with epistemic uncertainty feedback for photoreceptor layer segmentation in pathological oct scans, Proc. Int. Symp. Biomed. Imag. (2019) 1441 -1445, https://doi.org/10.1109/ ISBI.2019.8759581, 2019-April, no. Isbi. -
  92. [110][110] [J. Heo, et al., Uncertainty-aware attention for reliable interpretation and prediction, Adv. Neural Inf. Process. Syst. 2018-Decem (2018) 909 -918.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref110) -
  93. [111][111] [T.A. Adrian Tousignant, Lemaitre Paul, Doina Precup, Douglas L. Arnold, Prediction of Future Multiple Sclerosis Disease Progression Using Deep Learning Analysis of MRI Data, Proceesings of Machine Learning Research, 2019, pp. 483 -492.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref111) -
  94. [112][112] A.G. Roy, S. Conjeti, N. Navab, C. Wachinger, Bayesian QuickNAT: model uncertainty in deep whole-brain segmentation for structure-wise quality control, Neuroimage 195 (2019) 11 -22, https://doi.org/10.1016/j. neuroimage.2019.03.042. - · DOI
  95. [114][114] C.F. Baumgartner, et al., PHiSeg: capturing uncertainty in medical image segmentation, Lect. Notes Comput. Sci. 11765 LNCS (2019) 119 -127, https://doi. org/10.1007/978-3-030-32245-8\_14. - · DOI · Paper page
  96. [115][115] Ł . R ą czkowski, M. Mo ˙ zejko, J. Zambonelli, E. Szczurek, ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning, Sci. Rep. 9 (1) (2019) 1 -12, https://doi.org/10.1038/s41598-01950587-1. - · DOI
  97. [116][116] Z. Eaton-Rosen, T. Varsavsky, S. Ourselin, M.J. Cardoso, As easy as 1, 2..4? Uncertainty in counting tasks for medical imaging, Lect. Notes Comput. Sci. 11767 LNCS (2019) 356 -364, https://doi.org/10.1007/978-3-030-32251-9\_39. - · DOI · Paper page
  98. [117][117] [M.L. di Scandalea, C.S. Perone, M. Boudreau, J. Cohen-Adad, Deep Active Learning for Axon-Myelin Segmentation on Histology Data, 2019, pp. 1 -8.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref117) -
  99. [118][118] [R. Jena, S.P. Awate, A Bayesian neural net to segment images with uncertainty estimates and good calibration, in: A.C.S. Chung, J.C. Gee, P.A. Yushkevich, S. Bao (Eds.), Information Processing in Medical Imaging, Springer International Publishing, Cham, 2019, pp. 3 -15.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref118) -
  100. [119][119] [R.D. Soberanis-Mukul, N. Navab, S. Albarqouni, in: Uncertainty-based Graph Convolutional Networks for Organ Segmentation Refinement, 2019, pp. 1 -15, 1.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref119) -
  101. [120][120] S. Hu, D. Worrall, S. Knegt, B. Veeling, H. Huisman, M. Welling, Supervised uncertainty quantification for segmentation with multiple annotations, Lect. Notes Comput. Sci. 11765 LNCS (2019) 137 -145, https://doi.org/10.1007/9783-030-32245-8\_16. - · DOI
  102. [121][121] M. Combalia, F. Hueto, S. Puig, J. Malvehy, V. Vilaplana, Uncertainty estimation in deep neural networks for dermoscopic image classification, IEEE Comput. Soc. Conf. Comput. Vis. Patter Recogn. Workshops 2020-June (2020) 3211 -3220, https://doi.org/10.1109/CVPRW50498.2020.00380. S. Seoni et al. - · DOI
  103. [122][122] [S. Toledo-Cort ´ es, M.D. La Pava, O. Perd ' omo, F.A. Gonz ' alez, Hybrid Deep Learning Gaussian Process for Diabetic Retinopathy Diagnosis and Uncertainty Quantification, OMIA@MICCAI, 2020.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref122) -
  104. [123][123] [M.-H. Laves, J.F. Fast, L.A. Kahrs, T. Ortmaier, Well-calibrated regression uncertainty in medical imaging with deep learning, Proc. Mach. Learn Res. 121 (2020) 393 -412.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref123) -
  105. [124][124] X. Hu, et al., Coarse-to-Fine adversarial networks and zone-based uncertainty analysis for NK/T-Cell lymphoma segmentation in CT/PET images, IEEE J. Biomed. Health Inform. 24 (9) (2020) 2599 -2608, https://doi.org/10.1109/ JBHI.2020.2972694. -
  106. [127][127] R. Selvan, F. Faye, J. Middleton, A. Pai, Uncertainty quantification in medical image segmentation with normalizing flows, Lect. Notes Comput. Sci. 12436 (2020) 80 -90, https://doi.org/10.1007/978-3-030-59861-7\_9. LNCS. - · DOI
  107. [130][130] Y. Xia et al., ' 3D Semi-supervised Learning with Uncertainty-Aware Multi-View Co-training, ' pp. 3646 -3655.. -
  108. [131][131] [E.K. Marc Gantenbein, Ertunc Erdil, RevPHiSeg : A Memory-Efficient Neural Network for Uncertainty Quantification in Medical Image Segmentation, 2020.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref131) -
  109. [132][132] K. Wickstr ø m, M. Kampffmeyer, R. Jenssen, Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps, Med. Image Anal. 60 (2020), https://doi.org/10.1016/j.media.2019.101619. - · DOI · Paper page
  110. [133][133] G. Carneiro, L. Zorron Cheng Tao Pu, R. Singh, A. Burt, Deep learning uncertainty and confidence calibration for the five-class polyp classification from colonoscopy, Med. Image Anal. 62 (2020), https://doi.org/10.1016/j. media.2020.101653. - · DOI
  111. [135][135] [L. Quan, Y. Li, X. Chen, N. Zhang, in: A.L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M.A. Zuluaga, S.K. Zhou, D. Racoceanu, L. Joskowicz (Eds.), An Effective Data Refinement Approach for Upper Gastrointestinal Anatomy Recognition BT - Medical Image Computing and Computer Assisted Intervention -MICCAI 2020, Springer International Publishing, Cham, 2020, pp. 43 -52.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref135) -
  112. [136][136] Y. Wang, et al., Double-uncertainty weighted method for semi-supervised learning, Lect. Notes Comput. Sci. 12261 LNCS (2020) 542 -551, https://doi.org/ 10.1007/978-3-030-59710-8\_53. - · DOI
  113. [137][137] C. Bian, et al., Uncertainty-aware domain alignment for anatomical structure segmentation, Med. Image Anal. 64 (2020), https://doi.org/10.1016/j. media.2020.101732. - · DOI
  114. [138][138] R. Tanno, et al., Uncertainty modelling in deep learning for safer neuroimage enhancement: demonstration in diffusion MRI, Neuroimage 225 (August 2020), 117366, https://doi.org/10.1016/j.neuroimage.2020.117366, 2021. - · DOI
  115. [139][139] P. Thiagarajan, P. Khairnar, S. Ghosh, Explanation and use of uncertainty quantified by Bayesian neural network classifiers for breast histopathology images, IEEE Trans. Med. Imag. 41 (4) (2022) 815 -825, https://doi.org/10.1109/ TMI.2021.3123300. -
  116. [140][140] B. Ghoshal, A. Tucker, B. Sanghera, W.L. Wong, Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection, Comput. Intell. 37 (2) (May 2021) 701 -734, https://doi.org/ 10.1111/COIN.12411. - · DOI
  117. [141][141] V. Edupuganti, M. Mardani, S. Vasanawala, J. Pauly, Uncertainty quantification in deep MRI reconstruction, IEEE Trans. Med. Imag. 40 (1) (2021) 239 -250, https://doi.org/10.1109/TMI.2020.3025065. - · DOI
  118. [142][142] M.M.A. Valiuddin, C.G.A. Viviers, R.J.G. van Sloun, P.H.N. de With, F. van der Sommen, Improving aleatoric uncertainty quantification in multi-annotated medical image segmentation with normalizing flows, Lect. Notes Comput. Sci. 12959 (2021) 75 -88, https://doi.org/10.1007/978-3-030-87735-4\_8. LNCS. - · DOI
  119. [143][143] X. Teng, S. Pei, Y.R. Lin, StoCast: stochastic disease forecasting with progression uncertainty, IEEE J. Biomed. Health Inform. 25 (3) (2021) 850 -861, https://doi. org/10.1109/JBHI.2020.3006719. - · DOI
  120. [144][144] X. Wang, et al., UD-MIL: uncertainty-driven deep multiple instance learning for OCT image classification, IEEE J. Biomed. Health Inform. 24 (12) (2020) 3431 -3442, https://doi.org/10.1109/JBHI.2020.2983730. - · DOI
  121. [145][145] J. Zhang, U. Norinder, F. Svensson, Deep learning-based conformal prediction of toxicity, J. Chem. Inf. Model. 61 (6) (2021) 2648 -2657, https://doi.org/10.1021/ acs.jcim.1c00208. -
  122. [146][146] J.F. Vranken, et al., Uncertainty estimation for deep learning-based automated analysis of 12-lead electrocardiograms, Eur. Heart J. Digit. Health 2 (3) (2021) 401 -415, https://doi.org/10.1093/ehjdh/ztab045. - · DOI
  123. [147][147] I. Sieradzki, D. Le ´ sniak, S. Podlewska, How sure can we be about ML methodsbased evaluation of compound activity: incorporation of information about prediction uncertainty using deep learning techniques, Molecules 25 (6) (2020), https://doi.org/10.3390/molecules25061452. - · DOI
  124. [149][149] A. Sedghi, T. Kapur, J. Luo, P. Mousavi, W.M. Wells, Probabilistic image registration via deep multi-class classification: characterizing uncertainty, Lect. Notes Comput. Sci. 11840 (2019) 12 -22, https://doi.org/10.1007/978-3-03032689-0\_2. LNCS, no. March 2021. - · DOI
  125. [150][150] A. Norouzi, A. Emami, K. Najarian, N. Karimi, S. samavi, S.M.R. Soroushmehr, Exploiting Uncertainty of Deep Neural Networks for Improving Segmentation Accuracy in MRI Images, ICASSP, IEEE, 2019, pp. 2322 -2326, https://doi.org/ 10.1109/ICASSP.2019.8682530. - · DOI
  126. [151][151] [A. Filos, et al., A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks, NeurIPS, 2019, pp. 1 -12.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref151) -
  127. [152][152] [B. Ghoshal, A. Tucker, Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection, 2020 [Online]. Available.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref152) -
  128. [154][154] P.K. Mensah, et al., Uncertainty estimation using variational mixture of Gaussians capsule network for health image classification, Comput. Intell. Neurosci. 2022 (2022), 4984490, https://doi.org/10.1155/2022/4984490. - · DOI
  129. [155][155] B. Mazoure, A. Mazoure, J. B ´ edard, V. Makarenkov, DUNEScan: a web server for uncertainty estimation in skin cancer detection with deep neural networks, Sci. Rep. 12 (1) (2022) 179, https://doi.org/10.1038/s41598-021-03889-2. - · DOI
  130. [156][156] V. Jahmunah, E.Y.K. Ng, R.-S. Tan, S.L. Oh, U.R. Acharya, Uncertainty quantification in DenseNet model using myocardial infarction ECG signals, Comput. Methods Progr. Biomed. 229 (2023), 107308, https://doi.org/10.1016/ j.cmpb.2022.107308. -
  131. [157][157] C. Stoean, et al., Automated detection of presymptomatic conditions in spinocerebellar ataxia type 2 using Monte Carlo dropout and deep neural network techniques with electrooculogram signals, Sensors (Switzerland) 20 (11) (Jun. 2020), https://doi.org/10.3390/s20113032. - · DOI
  132. [158][158] F. Guo, et al., Improving cardiac MRI convolutional neural network segmentation on small training datasets and dataset shift: a continuous kernel cut approach, Med. Image Anal. 61 (Apr. 2020), https://doi.org/10.1016/j. media.2020.101636. - · DOI
  133. [159][159] J.H.B. da Silva, P.C. Cortez, S.K. Jagatheesaperumal, V.H.C. de Albuquerque, ECG measurement uncertainty based on Monte Carlo approach: an effective analysis for a successful cardiac health monitoring system, Bioengineering 10 (1) (Jan. 2023), https://doi.org/10.3390/bioengineering10010115. - · DOI
  134. [160][160] T. Nasir, M.K. Malik, SACDNet: towards Early Type 2 Diabetes Prediction with Uncertainty for Electronic Health Records, Jan. 2023 [Online]. Available: htt p://arxiv.org/abs/2301.04844. -
  135. [161][161] M. Abdar, et al., UncertaintyFuseNet: robust uncertainty-aware hierarchical feature fusion model with Ensemble Monte Carlo Dropout for COVID-19 detection, Inf. Fusion 90 (Feb. 2023) 364 -381, https://doi.org/10.1016/j. inffus.2022.09.023. - · DOI
  136. [162][162] S. MacDonald, et al., Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology, Sci. Rep. 13 (1) (May 2023) 7395, https://doi.org/10.1038/s41598-023-31126-5. - · DOI · Paper page
  137. [163][163] M.U. Farooq, Z. Ullah, J. Gwak, Residual attention based uncertainty-guided mean teacher model for semi-supervised breast masses segmentation in 2D ultrasonography, Comput. Med. Imag. Graph. 104 (Mar. 2023), 102173, https:// doi.org/10.1016/J.COMPMEDIMAG.2022.102173. - · DOI
  138. [164][164] M. Abdar, et al., Hercules: deep hierarchical attentive multilevel fusion model with uncertainty quantification for medical image classification, IEEE Trans. Ind. Inf. 19 (1) (Jan. 2023) 274 -285, https://doi.org/10.1109/TII.2022.3168887. - · DOI
  139. [165][165] A. Zakeri, et al., DragNet: learning-based deformable registration for realistic cardiac MR sequence generation from a single frame, Med. Image Anal. 83 (Jan. 2023), https://doi.org/10.1016/j.media.2022.102678. - · DOI
  140. [166][166] M. Abdar, et al., Binarized multi-gate mixture of Bayesian experts for cardiac syndrome X diagnosis: a clinician-in-the-loop scenario with a belief-uncertainty fusion paradigm, Inf. Fusion 97 (Sep. 2023), https://doi.org/10.1016/j. inffus.2023.101813. - · DOI
  141. [167][167] R. Tanno, et al., Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement, Jul. 2019 [Online]. Available: http://arxiv.org/abs/1 907.13418. -
  142. [168][168] [Z. Wang, et al., Bayesian Logistic Shape Model Inference : Application to Cochlear Image Segmentation, 2021.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref168) -
  143. [169][169] C. Corrado, et al., Quantifying atrial anatomy uncertainty from clinical data and its impact on electro-physiology simulation predictions, Med. Image Anal. 61 (Apr. 2020), https://doi.org/10.1016/j.media.2019.101626. - · DOI
  144. [170][170] [J. Dhamala, et al., Quantifying the Uncertainty in Model Parameters Using Gaussian Process-Based Markov Chain Monte Carlo in Cardiac Electrophysiology, 2018.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref170) -
  145. [171][171] J. Chen, E.C. Frey, Y. He, W.P. Segars, Y. Li, Y. Du, TransMorph: Transformer for Unsupervised Medical Image Registration, Nov. 2021, https://doi.org/10.1016/j. media.2022.102615. - · DOI
  146. [172][172] A.A. Abdullah, M.M. Hassan, Y.T. Mustafa, Uncertainty quantification for MLPmixer using Bayesian deep learning, Appl. Sci. 13 (7) (2023), https://doi.org/ 10.3390/app13074547. Apr. - · DOI
  147. [174][174] [K. Chitta, J.M. Alvarez, A. Lesnikowski, Large-Scale Visual Active Learning with Deep Probabilistic Ensembles, 2018.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref174) S. Seoni et al. -
  148. [175][175] [L. Smith, Y. Gal, Understanding measures of uncertainty for adversarial example detection, in: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 vol. 2, 2018, pp. 560 -569.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref175) -
  149. [177][177] [P. McClure, et al., Distributed weight consolidation: a brain segmentation case study, Adv. Neural Inf. Process. Syst. 2018-Decem (2018) 4093 -4103, no. NeurIPS 2018.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref177) -
  150. [178][178] [Q. Wu, H. Li, L. Li, Z. Yu, Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref178) -
  151. [179][179] [J. Linmans, J. Van Der Laak, G. Litjens, Efficient out-of-distribution detection in digital pathology using multi-head convolutional neural networks, Proc. Mach. Learn Res. 121 (2020) 465 -478.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref179) -
  152. [180][180] [G. Liang, Y. Zhang, N. Jacobs, Neural network calibration for medical imaging classification using DCA regularization, ICML Workshop Uncertain, in: Robustness Deep Learn. presented in Workshop Uncertainty and Robustness in Deep Learning Workshop (ICML 2020), 2020.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref180) -
  153. [181][181] K. Hoebel, et al., An exploration of uncertainty information for segmentation quality assessment, Proc. SPIE (Mar. 2020), https://doi.org/10.1117/ 12.2548722. -
  154. [183][183] [L. Dahal, A. Kafle, B. Khanal, Echocardiography SegmentationUncertainty Estimation in Deep 2D, May, 2020.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref183) -
  155. [184][184] E. Chiou, F. Giganti, S. Punwani, I. Kokkinos, E. Panagiotaki, Harnessing uncertainty in domain adaptation for mri prostate lesion segmentation, Lect. Notes Comput. Sci. 12261 LNCS (2020) 510 -520, https://doi.org/10.1007/9783-030-59710-8\_50. - · DOI
  156. [185][185] X. Cao, H. Chen, Y. Li, Y. Peng, S. Wang, L. Cheng, Uncertainty aware temporalensembling model for semi-supervised ABUS mass segmentation, IEEE Trans. Med. Imag. 40 (1) (2021) 431 -443, https://doi.org/10.1109/TMI.2020.3029161. - · DOI
  157. [186][186] Y. Qin, Z. Liu, C. Liu, Y. Li, X. Zeng, C. Ye, Super-Resolved q-Space deep learning with uncertainty quantification, Med. Image Anal. 67 (2021), 101885, https:// doi.org/10.1016/j.media.2020.101885. - · DOI
  158. [187][187] [R.K. Singh, R. Gorantla, S.G.R. Allada, P. Narra, SkiNet: a deep learning framework for skin lesion diagnosis with uncertainty estimation and explainability, PLoS One 17 (10) (Oct. 2022), e0276836.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref187) -
  159. [188][188] F. Guo, M. Ng, G. Kuling, G. Wright, Cardiac MRI segmentation with sparse annotations: ensembling deep learning uncertainty and shape priors, Med. Image Anal. 81 (Oct) (2022), https://doi.org/10.1016/j.media.2022.102532. - · DOI
  160. [189][189] T. Buddenkotte, et al., Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation, Comput. Biol. Med. 163 (Sep. 2023), https://doi.org/10.1016/j.compbiomed.2023.107096. - · DOI · Paper page
  161. [190][190] [D. Molchanov, A. Lyzhov, Y. Molchanova, A. Ashukha, D. Vetrov, Greedy policy search : a simple baseline for learnable test-time augmentation 2 (7) (2020).](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref190) -
  162. [192][192] [M.S. Ayhan, P. Berens, Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks, Med. Imag. Deep Learn. (MIDL), no. Midl (2018) 1 -9.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref192) -
  163. [193][193] G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, T. Vercauteren, Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks, Neurocomputing 338 (2019) 34 -45, https://doi.org/10.1016/j.neucom.2019.01.103. - · DOI · Paper page
  164. [194][194] Z. Zhang, A. Romero, M.J. Muckley, P. Vincent, L. Yang, M. Drozdzal, Reducing uncertainty in undersampled MRI reconstruction with active acquisition, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. (2019-June) 2049 -2053, https://doi.org/10.1109/CVPR.2019.00215, 2019. - · DOI · Paper page
  165. [195][195] C. Athanasiadis, E. Hortal, S. Asteriadis, Audio -visual domain adaptation using conditional semi-supervised Generative Adversarial Networks, Neurocomputing 397 (xxxx) (2020) 331 -344, https://doi.org/10.1016/j.neucom.2019.09.106. - · DOI
  166. [196][196] M.S. Ayhan, L. Kühlewein, G. Aliyeva, W. Inhoffen, F. Ziemssen, P. Berens, Expert-validated estimation of diagnostic uncertainty for deep neural networks in diabetic retinopathy detection, Med. Image Anal. 64 (2020), 101724, https://doi. org/10.1016/j.media.2020.101724. - · DOI
  167. [197][197] T. Araújo, et al., DR|GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images, Med. Image Anal. 63 (2020), 101715, https://doi.org/10.1016/j.media.2020.101715. - · DOI · Paper page
  168. [198][198] M. Abdar, et al., Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning, Comput. Biol. Med. 135 (2021), 104418, https://doi.org/10.1016/j.compbiomed.2021.104418. - · DOI
  169. [199][199] G. Scalia, C.A. Grambow, B. Pernici, Y.-P. Li, W.H. Green, Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction, J. Chem. Inf. Model. 60 (6) (Jun. 2020) 2697 -2717, https://doi.org/ 10.1021/acs.jcim.9b00975. - · DOI
  170. [200][200] S. Dong, Q. Yang, Y. Fu, M. Tian, C. Zhuo, RCoNet: deformable mutual information maximization and high-order uncertainty-aware learning for robust COVID-19 detection, IEEE Transact. Neural Networks Learn. Syst. 32 (8) (Aug. 2021) 3401 -3411, https://doi.org/10.1109/TNNLS.2021.3086570. - · DOI
  171. [201][201] I. Cort ´ es-Ciriano, A. Bender, Deep confidence: a computationally efficient framework for calculating reliable prediction errors for deep neural networks, J. Chem. Inf. Model. 59 (3) (2019) 1269 -1281, https://doi.org/10.1021/acs. jcim.8b00542. - · DOI
  172. [202][202] I. Cort ´ es-Ciriano, A. Bender, Reliable prediction errors for deep neural networks using test-time dropout, J. Chem. Inf. Model. 59 (7) (Jul. 2019) 3330 -3339, https://doi.org/10.1021/acs.jcim.9b00297. - · DOI · Paper page
  173. [203][203] F. Hamedani-KarAzmoudehFar, R. Tavakkoli-Moghaddam, A.R. Tajally, S.S. Aria, Breast cancer classification by a new approach to assessing deep neural networkbased uncertainty quantification methods, Biomed. Signal Process Control 79 (Jan. 2023), https://doi.org/10.1016/j.bspc.2022.104057. - · DOI
  174. [204][204] B.L. Graham, et al., Standardization of spirometry 2019 update an official American Thoracic Society and European Respiratory Society technical statement, Am. J. Respir. Crit. Care Med. 200 (8) (Oct. 15, 2019) E70 -E88, https://doi.org/10.1164/rccm.201908-1590ST. American Thoracic Society. - · DOI · Paper page
  175. [205][205] [S. Flügge, S. Zimmer, U. Petersohn, Knowledge Representation and Diagnostic Inference Using Bayesian Networks in the Medical Discourse, 2019.](http://refhub.elsevier.com/S0010-4825(23)00906-X/sref205) -
  176. [206][206] H. Li, H. Luo, Uncertainty quantification in medical image segmentation, in: 2020 IEEE 6th International Conference on Computer and Communications, ICCC), 2020, pp. 1936 -1940, https://doi.org/10.1109/ICCC51575.2020.9345043. - · DOI
  177. [207][207] A.S. Lundervold, A. Lundervold, An overview of deep learning in medical imaging focusing on MRI, Z. Med. Phys. 29 (2) (2019) 102 -127, https://doi.org/10.1016/ j.zemedi.2018.11.002. -
  178. [208][208] A. Kamal, et al., Recent advances and challenges in uncertainty visualization: a survey, J. Vis. 24 (5) (2021) 861 -890, https://doi.org/10.1007/s12650-02100755-1. - · DOI
  179. [209][209] D. Milan ´ es-Hermosilla, et al., Monte Carlo dropout for uncertainty estimation and motor imagery classification, Sensors 21 (21) (2021), https://doi.org/10.3390/ s21217241. -
  180. [211][211] M.W. Nadeem, H.G. Goh, V. Ponnusamy, I. Andonovic, M.A. Khan, M. Hussain, A fusion-based machine learning approach for the prediction of the onset of diabetes, Healthcare (Switzerland) 9 (10) (Oct. 2021), https://doi.org/10.3390/ healthcare9101393. -
  181. [212][212] B. Ihnaini, et al., A Smart Healthcare Recommendation System for Multidisciplinary Diabetes Patients with Data Fusion Based on Deep Ensemble Learning, 2021, https://doi.org/10.1155/2021/4243700. - · DOI
  182. [214][214] M.G. Uddin, S. Nash, A. Rahman, A.I. Olbert, A novel approach for estimating and predicting uncertainty in water quality index model using machine learning approaches, Water Res. 229 (Feb. 2023), https://doi.org/10.1016/j. watres.2022.119422. - · DOI
  183. [215][215] N. Molchanova, et al., Novel Structural-Scale Uncertainty Measures and Error Retention Curves: Application to Multiple Sclerosis, Nov. 2022, https://doi.org/ 10.48550/ARXIV.2211.04825. - · DOI
  184. [216][216] T.N. Kipf, M. Welling, Semi-Supervised Classification with Graph Convolutional Networks, Sep. 2016 [Online]. Available: http://arxiv.org/abs/1609.02907. - · arXiv
  185. [218][218] G.C. Al, A machine learning approach for the differential diagnosis of alzheimer and vascular dementia fed by MRI selected features, Front. Neuroinf. 14 (Jun) (2020), https://doi.org/10.3389/fninf.2020.00025. - · DOI · Paper page
  186. [219][219] S.K. Sood, S. Kaur, K.K. Chahal, An intelligent framework for monitoring dengue fever risk using LDA-ANFIS, J. Ambient Intell. Smart Environ. 12 (1) (2020) 5 -20, https://doi.org/10.3233/AIS-200547. - · DOI
  187. [220][220] M.Y.T. al, A new open-source GPU-based microscopic Monte Carlo simulation tool for the calculations of DNA damages caused by ionizing radiation - Part I: core algorithm and validation, Med. Phys. 47 (4) (Apr. 2020) 1958 -1970, https:// doi.org/10.1002/MP.14037. - · DOI
  188. [221][221] M.V.S. al, Projected impact of a reduction in sugar-sweetened beverage consumption on diabetes and cardiovascular disease in Argentina: a modeling study, PLoS Med. 17 (7) (Jul. 2020), https://doi.org/10.1371/JOURNAL. PMED.1003224. - · DOI
  189. [222][222] A.G. al, Mixed-effects models for health care longitudinal data with an informative visiting process: a Monte Carlo simulation study, Stat. Neerl. 74 (1) (Feb. 2020) 5 -23, https://doi.org/10.1111/STAN.12188. - · DOI
  190. [223][223] X.L. Sun, H. Wang, X.K. Li, G.H. Cao, Y. Kuang, X.C. Zhang, Monte Carlo computer simulation of a camera system for proton beam range verification in cancer treatment, Future Generat. Comput. Syst. 102 (Jan. 2020) 978 -991, https://doi. org/10.1016/J.FUTURE.2019.09.011. - · DOI
  191. [224][224] B.P. Magnusson, H. Schmidli, N. Rouyrre, D.O. Scharfstein, Bayesian inference for a principal stratum estimand to assess the treatment effect in a subgroup characterized by postrandomization event occurrence, Stat. Med. 38 (23) (Oct. 2019) 4761 -4771, https://doi.org/10.1002/sim.8333. - · DOI
  192. [225][225] K.L. al, Big medical data decision-making intelligent system exploiting fuzzy inference logic for prostate cancer in developing countries, IEEE Access 7 (2019) 2348 -2363, https://doi.org/10.1109/ACCESS.2018.2886198. - · DOI
  193. [226][226] S. Razi, M.R.K. Mollaei, J. Ghasemi, A novel method for classification of BCI multi-class motor imagery task based on Dempster-Shafer theory, Inf. Sci. 484 (May 2019) 14 -26, https://doi.org/10.1016/J.INS.2019.01.053. · DOI

Cited by

Sorted by citations

Authors
Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R.
Affiliations
  • Politecnico di Torino (IT)
  • University of Technology Sydney (AU)
  • University of Southern Queensland (AU)
Corresponding author
Molinari, Filippo
ORCID
Venue
Computers in Biology and Medicine
Publisher
Elsevier BV
Published
2023-10
Volume
165
Pages
107441
ISSN
0010-4825 (ISSN-L)
DOI
10.1016/j.compbiomed.2023.107441
OpenAlex
W4386385472
Cited by
239
References
22
Citations by year
2026: 85 · 2025: 88 · 2024: 53 · 2023: 13
FWCI
2.81
Type
journal article
Language
en
Open access
Licence
Locations
  • Computers in Biology and Medicine (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • PubMed (publishedVersion)
  • PORTO@Iris (submittedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
uncertainty quantification, uncertainty estimation, artificial intelligence in healthcare, Bayesian analysis, fuzzy systems, deep learning, machine learning, diagnostic imaging, physiological signals, diagnosis, treatment outcome prediction
MeSH terms
Artificial Intelligence, Bayes Theorem, Delivery of Health Care, Reproducibility of Results, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review