Explainable Deep Learning Models in Medical Image Analysis
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Attributions of VGG-16 with images from Imagenet using the methods implemented in [19].](https://figures.cochat.ai/paper-figures/a357c2dc02ed4a63a419e6c762416bfb/explainable-deep-learning-models-in-medical-image-analysis-figure-3.jpg)
![Figure 4 Example of heat maps from a retinal OCT image [40].](https://figures.cochat.ai/paper-figures/002116f86db043c7aaeedcab11546bae/explainable-deep-learning-models-in-medical-image-analysis-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More details about attention mechanisms in neural networks is discussed in [51].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An earlier study for breast mass classification from mammograms [40] using two different CNNs - AlexNet [41] and GoogleNet [42] - employed saliency maps to visualize the image features.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Both approaches are beneficial for medical imaging datasets which tend to be relatively smaller than computer vision benchmarks like ImageNet [33].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“rey patch and hence occluding them from the system’s view. It can provide both positive and negative evidence by highlighting the responsible features. This technique was applied by Zeiler and Fergus [18] to the CNN for the image classification task. Occlusion is the benchmark for any attribution study as it is a simpl…”
MethodWold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606 · cited 1×
“Principal component analysis (PCA) [12], t-Distributed Stochastic Neighbor Embedding (t-SNE) [13] are some common examples of these methods.”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 2×
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
Background
Research works printed but not linked: 2
- El-Baz, A.S. (2020). Recent Advances in Deep Learning Applications for Retinal Diagnosis using OCT. State of the Art in Neural Networks, Elsevier. in press.
- Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
Cited by
Items 201–300 of 772
Improving the Segmentation Accuracy of Ovarian-Tumor Ultrasound Images Using Image Inpainting
Chen, L., Qiao, C., Wu, M., Cai, L., Yin, C., Yang, M., Sang, X., & Bai, W. (2023) · Bioengineering · doi:10.3390/bioengineering10020184 · Cited by 27
Extendable and explainable deep learning for pan-cancer radiogenomics research
Liu, Q., & Hu, P. (2022) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.102111 · Cited by 27
Boushehri, S. S., Essig, K., Chlis, N., Herter, S., Bacac, M., Theis, F. J., Glasmacher, E., Marr, C., & Schmich, F. (2023) · Nature Communications · doi:10.1038/s41467-023-43429-2 · Cited by 33
“Extensive documentation, as well as how to reproduce the analysis in the form of Jupyter notebooks, is provided online at https://github.com/marrlab/scifAI/ and https:// github.com/marrlab/scifAI-notebooks. scifAI ’ s explainable design follows the de fi nition of Singh et al. 39 and Tjoa et al. 40 , where machine lear…”
MethodCorti, L., Oltmans, R., Jung, J., Balayn, A., Wijsenbeek, M. S., & Yang, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642551 · Cited by 43
“It is indeed common for patient conditions and treatments to require clinical progression before actions can be taken – both by clinicians and researchers striving to support clinical workfows.”
Xu, R., Cao, J.-S., Luo, J., Ni, B., Fang, F., Liu, W., & Wang, P. (2024) · npj Clean Water · doi:10.1038/s41545-024-00367-w · Cited by 30
“The CNN classi fi er cannot say “ I don ’ t know ” in ambiguous situations and instead returns the class with the highest probability 14 .”
An explainable fast deep neural network for emotion recognition
Luzio, F. D., Rosato, A., & Panella, M. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107177 · Cited by 33
“For this reason, the employment of explainability for a better understanding of the inner comprehension of DL models is the key concept for a fair, safe, and understandable use of AI, in addition to a possible enabler for its deployment into everyday activity [12].”
A 2.5D Deep Learning-Based Method for Drowning Diagnosis Using Post-Mortem Computed Tomography
Zeng, Y., Zhang, X., Kawasumi, Y., Usui, A., Ichiji, K., Funayama, M., & Homma, N. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3225416 · Cited by 27
“Thelackofinspectingthebehaviorof modelsaffectstheuseofdeeplearninginalldomains,especially medicalimagediagnosis,whereexplainabilityandreliability arethekeyelementsfortrustbytheend-user[12].”
Vamsidhar, D., Desai, P., Joshi, S., Kolhar, S., Deshpande, N. M., & Gite, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-06455-2 · Cited by 34
Advancements in Healthcare Services using Deep Learning Techniques
Rana, M., & Bhushan, M. (2022) · International Mobile and Embedded Technology Conference (MECON) · doi:10.1109/mecon53876.2022.9752020 · Cited by 26
Interpretable Machine Learning with Brain Image and Survival Data
Eder, M., Moser, E., Holzinger, A., Jean-Quartier, C., & Jeanquartier, F. (2022) · BioMedInformatics · doi:10.3390/biomedinformatics2030031 · Cited by 31
Atcı, Ş. Y., Güneş, A., Zontul, M., & Arslan, Z. (2024) · Tomography · doi:10.3390/tomography10020017 · Cited by 26
“…features, which are subsequently channeled into a bespoke classifier tailored for evaluating the severity levels of DR. Additionally, we use SHapley Additive exPlanations (SHAP) analysis [19,20] approaches to examine the characteristics that contributed to the results of our prediction models.”
MethodShrivastava, A. D., & Kell, D. B. (2021) · Molecules · doi:10.3390/molecules26072065 · Cited by 25
“This is often referred to as ‘explainable AI’ [11-18].”
BackgroundExplainable AI for all - A roadmap for inclusive XAI for people with cognitive disabilities
Tielman, M. L., Suárez-Figueroa, M. C., Jönsson, A., Neerincx, M. A., & Siebert, L. C. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102685 · Cited by 33
Transformers-based architectures for stroke segmentation: a review
Zafari-Ghadim, Y., Rashed, E. A., Mohamed, A. M. S., & Mabrok, M. A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10900-5 · Cited by 30
“…additional characteristics, such as privacy-preserving algorithms (Sheller et al. 2020; Li et al. 2021, 2019), and embrace explainable artificial intelligence (XAI) (Alicioglu and Sun 2022; Mondal et al. 2021; Singh et al. 2020) as integral components of trustworthy AI (Liu et al. 2022a).”
Large Language Models in Medical Image Analysis: A Systematic Survey and Future Directions
Urooj, B., Fayaz, M., Ali, S., Dang, L., & Kim, K. W. (2025) · Bioengineering · doi:10.3390/bioengineering12080818 · Cited by 27
“Furthermore, studies have explored interpretable AI methods in medical image processing, categorizing approaches, and evaluating their potential to make AI more transparent and trustworthy for healthcare professionals [39,40].”
Achilleos, K. G., Leandrou, S., Prentzas, N., Kyriacou, P. A., Kakas, A. C., & Pattichis, C. S. (2020) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe50027.2020.00175 · Cited by 30
“Classification models and especially medical diagnosis systems need to be transparent and explainable [11] to earn the trust of end users.”
BackgroundSon, J., Shin, J. Y., Kong, S. T., Park, J., Kwon, G., Kim, H. D., Park, K. H., Jung, K.-H., & Park, S. J. (2023) · Scientific Reports · doi:10.1038/s41598-023-32518-3 · Cited by 29
Zhu, X., Kim, Y., Ravid, O., He, X., Suarez‐Jimenez, B., Zilcha‐Mano, S., Lazarov, A., Lee, S., Abdallah, C. G., Angstadt, M., et al. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120412 · Cited by 29
“However, deep learning need not be uninterpretable - as witnessed by the rapid expansion of methods for explainable deep learning (Bai et al., 2021; Singh et al., 2020), which uses new forms of visualization and representations of model outcomes.”
BackgroundZhang, Y., Yu, S., Zhu, X., Ning, X., Liu, W., Wang, C., Liu, X., Zhao, D., Zheng, Y., & Bao, J. (2021) · Biomedical Optics Express · doi:10.1364/boe.432654 · Cited by 25
“Guided by visual appeal, existing studies [18–20] that consider interpretability mainly focus on extracting salient spatial regions of the input image.”
MethodA survey of post-hoc xai methods from a visualization perspective: Challenges and opportunities
Bhati, D., Amiruzzaman, M., Zhao, Y., Guercio, A., & Le, T. (2025) · IEEE Access · doi:10.1109/access.2025.3581136 · Cited by 24
“Research has also explored XAI’s role in clinical decision support systems [14], medical image analysis [15], [16], trustworthy AI techniques in biomedical imaging [17], [18], [19], AI interpretability in radiology [20], and AI applications in response to COVID-19 [21], [22].”
Automated Detection and Diagnosis of Diabetic Retinopathy: A Comprehensive Survey
Lakshminarayanan, V., Kheradfallah, H., Sarkar, A., & Balaji, J. J. (2021) · Journal of Imaging · doi:10.3390/jimaging7090165 · Cited by 142
“This black box nature of DCNNs is the major problem that makes them unsuitable for clinical application [86,152,153] and has made the topic of eXplainable AI (XAI) of major importance [153].”
BackgroundSelf-citationAhmed, F. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01893-4 · Cited by 29
Explainability of deep learning models in medical video analysis: a survey
Kolárik, M., Sarnovský, M., Paralič, J., & Babič, F. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1253 · Cited by 31
“A nice overview of works that deal with visual data such as 2D images, 3D images, and videos was provided in Cazuguel (2017) or Singh, Sengupta & Lakshminarayanan (2020) .”
BackgroundHu, C., Jia, S., Zhang, F., Xiao, C., Ruan, M., Thrasher, J., & Li, X. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.08.017 · Cited by 29
Rokhshad, R., Salehi, S. N., Yavari, A., Shobeiri, P., Esmaeili, M., Manila, N., Motamedian, S. R., & Mohammad-Rahimi, H. (2024) · Oral Radiology · doi:10.1007/s11282-023-00715-5 · Cited by 23
Explainable deep learning and biomechanical modeling for TMJ disorder morphological risk factors
Sun, S., Xu, P., Buchweitz, N. F., Hill, C. N., Ahmadi, F., Wilson, M. B., Mei, A., She, X., Sagl, B., Slate, E. H., et al. (2024) · JCI Insight · doi:10.1172/jci.insight.178578 · Cited by 25
“Instead, explainable deep learning (22) algorithms provide insight into model prediction, enabling identification of key TMJ morphological region(s) for determining TMJ disorder risk.”
BackgroundWhat is the Optimal Attribution Method for Explainable Ophthalmic Disease Classification?
Singh, A., Sengupta, S., Balaji, J. J., Mohammed, A. R., Faruq, I., Jayakumar, V., Zelek, J., & Lakshminarayanan, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63419-3_3 · Cited by 27
Self-citationInterpreting models interpreting brain dynamics
Rahman, M. M., Mahmood, U., Lewis, N., Gazula, H., Fedorov, A., Fu, Z., Calhoun, V. D., & Plis, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-15539-2 · Cited by 29
“A recent review 41 reveals that deep learning models are a viable clinical supportive tool in the neuroimaging domain.”
BackgroundLin, X., Dai, L., Yang, Q., Yang, Q., He, H., Ma, L., Liu, J., Cheng, J., Cai, C., Bao, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09417-2 · Cited by 25
Explainable AI for Interpretation of Ovarian Tumor Classification Using Enhanced ResNet50
Guha, S., Kodipalli, A., Fernandes, S. L., & Dasar, S. K. (2024) · Diagnostics · doi:10.3390/diagnostics14141567 · Cited by 24
“Singh, A. et al. [14] discussed the explainable AI methods in the medical image analysis domain.”
Fréchet radiomic distance (FRD): A versatile metric for comparing medical imaging datasets
Konz, N., Osuala, R. O., Verma, P., Chen, Y., Gu, H., Dong, H., Chen, Y., Marshall, A., Garrucho, L., Kushibar, K., et al. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103943 · Cited by 33
“Additionally, medical imaging requires metrics that specifically capture anatomical consistency and realism, as well as clinical interpretability [14, 76, 106, 112].”
Brain Stroke Classification Using CT Scans with Transformer-Based Models and Explainable AI
Qari, S., & Thafar, M. A. (2025) · Diagnostics · doi:10.3390/diagnostics15192486 · Cited by 28
“By localizing discriminative areas, such as infarcted or hemorrhagic regions, Grad-CAM assists clinicians verify whether the model’s focus aligns with radiological findings [42].”
R2U3D: Recurrent Residual 3D U-Net for Lung Segmentation
Kadia, D. D., Alom, M. Z., Burada, R., Nguyen, T., & Asari, V. K. (2021) · IEEE Access · doi:10.1109/access.2021.3089704 · Cited by 22
Shahabi, M. S., & Shalbaf, A. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3207350 · Cited by 26
Nakajo, K., Ninomiya, Y., Kondo, H., Takeshita, N., Uchida, E., Aoyama, N., Inaba, A., Ikematsu, H., Shinozaki, T., Matsuura, K., et al. (2023) · Head & Neck · doi:10.1002/hed.27370 · Cited by 22
“In the field of medical image analysis, it is crucial to prove that the predictions formulated by a model are based on the proper detection of the most relevant components in a given image.(23) Grad-CAM visualizes the relevance of the regions or features “focused” by the model to classify an image, in an attention hea…”
BackgroundDeep Learning Algorithms for Pancreas Segmentation from Radiology Scans
Zhang, Z., Yao, L., Keleş, E., Velichko, Y., & Bağcı, U. (2023) · Advances in Clinical Radiology · doi:10.1016/j.yacr.2023.05.001 · Cited by 26
An Overview of Artificial Intelligence Applications in Liver and Pancreatic Imaging
Cardobi, N., Palù, A. D., Pedrini, F., Beleù, A., Nocini, R., Robertis, R. D., Ruzzenente, A., Salvia, R., Montemezzi, S., & D’Onofrio, M. (2021) · Cancers · doi:10.3390/cancers13092162 · Cited by 24
“XAI analyses report human understandable features that are responsible for the AI model output [75].”
BackgroundArtzi, M., Redmard, E., Tzemach, O., Zeltser, J., Gropper, O., Roth, J., Shofty, B., Kozyrev, D. A., Constantini, S., & Ben‐Sira, L. (2021) · IEEE Access · doi:10.1109/access.2021.3085771 · Cited by 24
Learning to Synthesize Programs as Interpretable and Generalizable PoliciesPreprint
Trivedi, D., Zhang, J. M., Sun, S.-H., & Lim, J. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13643 · Cited by 64
“[103] Amitojdeep Singh, Sourya Sengupta, and Vasudevan Lakshminarayanan.”
BackgroundLarge language model for medical images: a survey of taxonomy, systematic review, and future trends
Wang, P., Lu, W., Lu, C., Zhou, R., Li, M., & Qin, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020090 · Cited by 21
Dong, Y., Batallé, D., & Deprez, M. (2025) · Human Brain Mapping · doi:10.1002/hbm.70190 · Cited by 24
“The complexity of deep learning models renders machine decision-making opaque and significantly reduces the trust of doctors and patients in artificial intelligence (Singh et al. 2020).”
Ramesh, E., Ganesan, A., Lakshmi, K. C., & Natarajan, P. M. (2025) · Frontiers in Oral Health · doi:10.3389/froh.2025.1414524 · Cited by 21
“Finding the best diagnosis at the lowest feasible cost seems to be possible with the combination of web-based interface and explanatory AI techniques (6).”
Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
Li, J., Kuang, K., Li, L., Chen, L., Zhang, S., Shao, J., & Xiao, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475337 · Cited by 28
“When relating to the moral problem or the environmental factors that are uncertain such as crime judgment [10], financial analysis [5], and medical diagnosis [37], it is essential to mine the evidence for the model’s prediction (interpret model knowledge) to convince humans.”
BackgroundA quantitative evaluation of explainable AI methods using the depth of decision tree
AHMED, N. A. M., & Alpkoçak, A. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3924 · Cited by 23
“Singh et al. [4] mentioned two main explainability groups that are used to explain deep neural networks in the medical images: attribution-based methods, and architecture or domain-specific methods.”
BackgroundBenchmarking Polyp Segmentation Methods in Narrow-Band Imaging Colonoscopy Images
Yue, G., Zhuo, G., Li, S., Zhou, T., Du, J., Yan, W., Hou, J., Liu, W., & Wang, T. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3270724 · Cited by 27
“At this point, the recent advances in the field of explainable deep learning [67] have the prospect to at least partially remedy the deficiency of current studies by allowing .”
BackgroundA Review of Medical Diagnostic Video Analysis Using Deep Learning Techniques
Farhad, M., Masud, M. M., Beg, A., Ahmad, A., & Ahmed, L. A. (2023) · Applied Sciences · doi:10.3390/app13116582 · Cited by 25
DeepExplain: Enhancing DeepFake Detection Through Transparent and Explainable AI model
Sunkari, V., & Srinagesh, A. (2024) · Informatica · doi:10.31449/inf.v48i8.5792 · Cited by 26
“In security-sensitive contexts, where decisions may have legal or societal consequences, XAI provides a mechanism for accountability and ethical assurance[22].”
BackgroundLung Sound Classification for Respiratory Disease Identification Using Deep Learning: A Survey
Wanasinghe, T., Bandara, S., Madusanka, S., Meedeniya, D., Bandara, M., & Díez, I. D. L. T. (2024) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v20i10.49585 · Cited by 27
Wyatt, L. S., van Karnenbeek, L. M., Wijkhuizen, M., Geldof, F., & Dashtbozorg, B. (2024) · Applied Sciences · doi:10.3390/app14188108 · Cited by 24
Grad-CAM based Visualization for Interpretable Lung Cancer Categorization using Deep CNN Models
Mothkur, R., Soubhagyalakshmi, P., & Swetha, C. B. (2025) · Journal of Electronics Electromedical Engineering and Medical Informatics · doi:10.35882/jeeemi.v7i3.690 · Cited by 25
“The latest XAI methods used in medical imaging are covered by Singh, Sengupta et al.[5] , with an emphasis on categorizing lung cancer from CT and X-ray images.”
Ganitidis, T., Athanasiou, M., Dalakleidi, K. V., Melanitis, N., Golemati, S., & Nikita, K. S. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630402 · Cited by 20
“Following the aforementioned demands, a series of methods, known as interpretable artificial intelligence, have been developed [11], [12].”
BackgroundDeep learning-based medical image analysis with explainable transfer learning
Hao, J. (2023) · International Conference on Computer Engineering and Distance Learning (CEDL) · doi:10.1109/cedl60560.2023.00029 · Cited by 23
Sengupta, S., & Anastasio, M. A. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3348699 · Cited by 21
“Popular such methods include gradient-based class activation maps (CAMs) [1].”
Self-citationChen, X., Chen, H., Wan, J., Li, J., & Wei, F. (2024) · Journal of bone oncology · doi:10.1016/j.jbo.2024.100626 · Cited by 21
Lokaj, B., de Gevigney, V. D., Djema, D. A., Zaghir, J., Goldman, J.-P., Bjelogrlic, M., Turbé, H., Kinkel, K., Lovis, C., & Schmid, J. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109721 · Cited by 20
Uncertainty aware and explainable diagnosis of retinal disease
Singh, A., Sengupta, S., Rasheed, M. A., Jayakumar, V., & Lakshminarayanan, V. (2021) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.2581362 · Cited by 26
“The adoption of these methods in clinics is slow if not negligible The key factors hindering trust from end-users, regulators, and patients on deep learning methods in medical imaging are the opaqueness of the algorithms and the tendency to make wrong decisions on complex cases.(4,5) There is a need to describe the fa…”
MethodSelf-citationAchievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis SystemsPreprint
Lucieri, A., Bajwa, M. N., Dengel, A. R., & Ahmed, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13169 · Cited by 31
“Therefore, another direction of research to make AI algorithms more transparent and explainable is to use multimodel data like medical images and patients’ records together in the decision-making process and attribute the model decisions to each of them [14].”
BackgroundInfluentialExplainable AI applications in the Medical Domain: a systematic review
Prentzas, N., Kakas, A., & Pattichis, C. S. (2023) · arXiv · doi:10.48550/arxiv.2308.05411 · Cited by 19
“A number of taxonomies can be found in the literature for the classification of explainability techniques, types of explanation, properties of good explanations, and goals of explanation [8], [16], [19], [22], [26], [37], [62].”
BackgroundMethodResultInfluentialVisualizing Multimodal Deep Learning for Lesion Prediction
Gillmann, C., Peter, L., Schmidt, C., Saur, D., & Scheuermann, G. (2021) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2021.3099881 · Cited by 19
Artificial intelligence in cardiac magnetic resonance fingerprinting
Velasco, C., Fletcher, T. J., Botnar, R. M., & Prieto, C. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.1009131 · Cited by 21
“There are efforts to improve interpretability, such as explainable AI (120), which tries to generate solutions that can be more easily understood by the end user.”
BackgroundA Short Review on Explainable Artificial Intelligence in Renewable Energy and Resources
Ersöz, B., Sağıroğlu, Ş., & Bülbül, H. İ. (2022) · International Conference on Renewable Energy Research and Applications (ICRERA) · doi:10.1109/icrera55966.2022.9922870 · Cited by 22
“The main purpose of recently introduced XAI approaches is to help developers uncover decision and take into account a clearly defined set of elements that are important in explaining the outputs of AI or deep learning models [11].”
BackgroundVo, H.-T., Thien, N. N., & Mui, K. C. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01411137 · Cited by 19
One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical ImagesPreprint
Jin, W., Li, X., & Hamarneh, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05047 · Cited by 25
“Regarding medical imaging-related tasks, the most common and clinical end-user-friendly explanation is a saliency map (Jin et al., 2021; Reyes et al., 2020; Singh et al., 2020).”
BackgroundFedVGM: Enhancing Federated Learning Performance on Multi-Dataset Medical Images With XAI
Tahosin, M. S., Sheakh, M. A., Alam, M. J., Hassan, M. M., Bairagi, A. K., Abdulla, S., AlShathri, S. I., & El‐Shafai, W. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3600361 · Cited by 19
“Two widely usedXAItechniquesareSHAP[43]andGrad-CAM[44].”
DeepHP: A New Gastric Mucosa Histopathology Dataset for Helicobacter pylori Infection Diagnosis
Gonçalves, W. G. E., Santos, M. H. P. D., Brito, L. M., Palheta, H. G. A., Lobato, F. M. F., Demachki, S., Ribeiro‐dos‐Santos, Â., & de Araújo, G. S. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232314581 · Cited by 22
“In addition, we previously identified several issues regarding model reproducibility caused by a lack of experimental setup information, such as uninformed network hyper-parameters, computational cost, and the availability of private or public image collections [10,12].”
BackgroundCross X-AI: Explainable Semantic Segmentation of Laparoscopic Images in Relation to Depth Estimation
Bardozzo, F., Priscoli, M. D., Collins, T., Forgione, A., Hostettler, A., & Tagliaferri, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892345 · Cited by 19
“Several techniques to implement this type of analysis have already been published in recent years [6, 7, 8].”
MethodPallawi, S., & Singh, D. K. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00271-y · Cited by 19
“KNN (K-nearest neighbor) [15, 48, 96, 110, 115, 130, 131] KNN is another popular unsupervised learning method that uses Euclidean distance as a distance measure to determine the class of a sample.”
BackgroundMethodOsadebey, M. E., Liu, Q., Fuster‐García, E., & Emblem, K. E. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02320-2 · Cited by 19
“Although current XAI techniques reveal the association between the contributions from pixels in the input data to the model prediction, they suffer from low inter-pretability because the clinical value of the features extracted by an AI system and its predictions are hard to depict [19, 20].”
BackgroundExplainability of Brain Tumor Classification Based on Region
Narayankar, P. M., & Baligar, V. P. (2024) · International Conference on Emerging Technologies in Computer Science for Interdisciplinary Applications (ICETCS) · doi:10.1109/icetcs61022.2024.10544289 · Cited by 21
“Singh, A., Sengupta, S., & Lakshminarayanan, [16].”
Högberg, C., Larsson, S., & Lång, K. (2024) · Digital Health · doi:10.1177/20552076241287958 · Cited by 23
“25 One instrument stressed as able to improve trust in AI in the clinic 26 and medical imaging 27 is increased transparency.”
BackgroundNovel Hybrid UNet++ and LSTM Model for Enhanced Attack Detection and Classification in IoMT Traffic
Mezina, A., Nurmi, J., & Ometov, A. (2025) · IEEE Access · doi:10.1109/access.2025.3553966 · Cited by 18
“Additionally, it can be valuable to investigate the impact of features used as input using statistical methods [40] or approaches for ‘‘unpacking’’ the closed box nature of DL [43].”
Future artificial intelligence tools and perspectives in medicine
Chaddad, A., Katib, Y., & Hassan, L. (2021) · Current Opinion in Urology · doi:10.1097/mou.0000000000000884 · Cited by 20
“Given CNN’s hidden layers as an example of explainability, the neurons with their weight are considered as a black-box [40,70,71].”
BackgroundVLFATRollout: Fully transformer-based classifier for retinal OCT volumes
Oghbaie, M., Araújo, T., Schmidt-Erfurth, U. M., & Bogunović, H. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102452 · Cited by 18
Deep Learning Techniques for Ear Diseases Based on Segmentation of the Normal Tympanic Membrane
Park, Y. S., Jeon, J. H., Kong, T. H., Chung, T.-Y., & Seo, Y. J. (2022) · Clinical and Experimental Otorhinolaryngology · doi:10.21053/ceo.2022.00675 · Cited by 19
“to complement medical professionals should have a certain amount of explainability and allow human experts to retrace decisions and use their judgment [6].”
Deep Learning for Image Segmentation: A Focus on Medical Imaging
Khalifa, A. F., & Badr, E. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.035888 · Cited by 25
Durur-Subasi, İ., & Özçelik, Ş. B. (2023) · Eurasian Journal of Medicine · doi:10.5152/eurasianjmed.2023.23360 · Cited by 20
Optimizing medical image analysis through MViTX on multiple datasets with explainable AI
Sheakh, M. A., Tahosin, M. S., Alam, M. J., & Begum, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11014-x · Cited by 17
“In our analysis, we employ three prominent XAI methods: SHAP (SHapley Additive exPlanations) [47], and Saliency Map [48].”
Recommendations for machine learning benchmarks in neuroimaging
Leenings, R., Winter, N. R., Dannlowski, U., & Hahn, T. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119298 · Cited by 19
Epilepsy Detection using Bi-LSTM with Explainable Artificial Intelligence
Rathod, P., Bhalodiya, J. M., & Naik, S. T. (2022) · IEEE India Conference · doi:10.1109/indicon56171.2022.10039816 · Cited by 17
“Several researchers have also noted the same observations and difficulties in translating such models for clinical practice [17].”
BackgroundECLAD: Extracting Concepts with Local Aggregated Descriptors
Posada-Moreno, A. F., Surya, N., & Trimpe, S. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110146 · Cited by 23
Li, H., Xu, Y., Lei, Y., Wang, Q., & Gao, X. (2022) · Diagnostics · doi:10.3390/diagnostics12061359 · Cited by 18
Differentiating spinal pathologies by deep learning approach
Haim, O., Agur, A., Gabay, S., Azolai, L., Shutan, I., Chitayat, M., Katirai, M., Sadon, S., Artzi, M., & Lidar, Z. (2023) · The Spine Journal · doi:10.1016/j.spinee.2023.09.019 · Cited by 16
Yu, S. (2023) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3616961.3616965 · Cited by 17
Sokolov, A. V., & Schiöth, H. B. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02992-y · Cited by 17
Shu, K., Mao, S., Zhang, Z., Coyle, J. L., & Sejdić, E. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108505 · Cited by 16
Lu, S.-C., Chen, G. H., Liu, A.-S., Sun, J., Gao, J.-W., Huang, C., Tsai, C., & Fu, L. (2025) · Journal of Medical Internet Research · doi:10.2196/67576 · Cited by 18
Explainability for Regression CNN in Fetal Head Circumference Estimation from Ultrasound Images
Zhang, J., Petitjean, C., Yger, F., & Aïnouz, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61166-8_8 · Cited by 16
Hüllebrand, M., Ivantsits, M., Tautz, L., Kelle, S., & Hennemuth, A. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.829512 · Cited by 17
“Explainability approaches for convolutional neural networks support the identification of image regions, which contribute to classification results (32) and provide information for plausibility checks as demonstrated for the interpretation of echocardiograms (33).”
BackgroundCodCAM: A new ensemble visual explanation for classification of medical thermal images
Örnek, A. H., & Ceylan, M. (2023) · Quantitative InfraRed Thermography Journal · doi:10.1080/17686733.2023.2167459 · Cited by 19
“Singh et al. [54] show how deep learning models are used in explainable medical image analysis.”
MethodKerber, B., Hepp, T., Küstner, T., & Gatidis, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292993 · Cited by 20
Explainable AI (XAI) for AI-Acceptability: The Coming Age of Digital Management 5.0
Gamoura, S. C. (2023) · IEEE International Conference on Networking, Sensing and Control (ICNSC) · doi:10.1109/icnsc58704.2023.10319030 · Cited by 17
“The main works in this field are [4], [48], [49] and [50].”
BackgroundLamprou, V., Kallipolitis, A., & Maglogiannis, I. G. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108238 · Cited by 21
Evaluating artificial intelligence for medical imaging: a primer for clinicians
Keni, S. (2024) · British Journal of Hospital Medicine · doi:10.12968/hmed.2024.0312 · Cited by 20
Ghosh, S., Sindhujaa, P., Kesavan, D. K., Gulyás, B. Z., & Máthé, D. (2025) · Surgeries · doi:10.3390/surgeries6030050 · Cited by 17
“Recent studies have focused on developing interpretable DL models in medical image analysis to enhance trust and facilitate clinical adoption [18].”
COVID-19 Diagnosis from Chest CT Scans: A Weakly Supervised CNN-LSTM Approach
Kara, M., Öztürk, Z., Akpek, S., & Turupcu, A. (2021) · AI · doi:10.3390/ai2030020 · Cited by 16
“There are many methods developed to explain deep learning methods trained on medical images [34], such as Grad-CAM [35].”
MethodPhilipp, M., Alperovich, A., Gutt‐Will, M., Mathis, A., Saur, S. C., Raabe, A., & Mathis-Ullrich, F. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00179 · Cited by 15
“This addresses the need for transparency and explain-ability in medical computer vision applications [26].”
BackgroundRodriguez-Sampaio, M., Rincón, M., Valladares‐Rodríguez, S., & Bachiller, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-06242-1_55 · Cited by 16
Editorial: Interpretable and explainable machine learning models in oncology
Hrinivich, W. T., Wang, T., & Wang, C. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1184428 · Cited by 16
Accurate Explanation Model for Image Classifiers using Class Association Embedding
Xie, R., Chen, J., Jiang, L., Xiao, R., Pan, Y., & Cai, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00180 · Cited by 18
“Nevertheless, the “black-box” nature of deep neural networks [5], namely, the lack of explainable inferences, creates a major and sustained challenge.”
BackgroundChauhan, T., & Sonawane, S. S. (2022) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v10i4.5538 · Cited by 14
Metadata
- Authors
- Singh, A., Sengupta, S., & Lakshminarayanan, V.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Singh, Amitojdeep
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2020-06-20
- Volume
- 6
- Issue
- 6
- Pages
- 52
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC8321083
- OpenAlex
- W3031116957
- Semantic Scholar
- af6af40155860ac2fd36680e63686473803b5f52
- Cited by
- 718
- References
- 75
- Citations by year
- 2026: 82 · 2025: 135 · 2024: 132 · 2023: 154 · 2022: 130 · 2021: 77 · 2020: 8
- FWCI
- 3.17
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Compute Canada
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable deep learning, medical image analysis, model interpretability, black-box models, clinical deployment, feature importance, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.