Explainable Deep Learning Models in Medical Image Analysis
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Attributions of VGG-16 with images from Imagenet using the methods implemented in [19].](https://figures.cochat.ai/paper-figures/a357c2dc02ed4a63a419e6c762416bfb/explainable-deep-learning-models-in-medical-image-analysis-figure-3.jpg)
![Figure 4 Example of heat maps from a retinal OCT image [40].](https://figures.cochat.ai/paper-figures/002116f86db043c7aaeedcab11546bae/explainable-deep-learning-models-in-medical-image-analysis-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More details about attention mechanisms in neural networks is discussed in [51].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An earlier study for breast mass classification from mammograms [40] using two different CNNs - AlexNet [41] and GoogleNet [42] - employed saliency maps to visualize the image features.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Both approaches are beneficial for medical imaging datasets which tend to be relatively smaller than computer vision benchmarks like ImageNet [33].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“rey patch and hence occluding them from the system’s view. It can provide both positive and negative evidence by highlighting the responsible features. This technique was applied by Zeiler and Fergus [18] to the CNN for the image classification task. Occlusion is the benchmark for any attribution study as it is a simpl…”
MethodWold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606 · cited 1×
“Principal component analysis (PCA) [12], t-Distributed Stochastic Neighbor Embedding (t-SNE) [13] are some common examples of these methods.”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 2×
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
Background
Research works printed but not linked: 2
- El-Baz, A.S. (2020). Recent Advances in Deep Learning Applications for Retinal Diagnosis using OCT. State of the Art in Neural Networks, Elsevier. in press.
- Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
Cited by
Items 401–500 of 772
Shang, F., Song, T., Wang, Z., Wu, M., Yan, C., & Wang, X. (2024) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.23-0519 · Cited by 8
A Multi-Label Deep Learning Model for Detailed Classification of Alzheimer's Disease
Yang, M., Zhao, Y., Yu, H., Chen, S., Gao, G., Li, D., Wu, X., Huang, L., & Ye, S. (2025) · Actas Españolas de Psiquiatría · doi:10.62641/aep.v53i1.1728 · Cited by 7
Voxel-Level Importance Maps for Interpretable Brain Age Estimation
Bintsi, K.-M., Baltatzis, V., Hammers, A., & Rueckert, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87444-5_7 · Cited by 7
Hidayah, F., Paidi, P., Maulidiya, A., Al-Farisi, F. R., & Anshori, M. (2023) · INTERNATIONAL JOURNAL OF MULTIDISCIPLINARY RESEARCH AND ANALYSIS · doi:10.47191/ijmra/v6-i7-46 · Cited by 7
“According to (Singh et al., 2020) The use of less varied learning models can lead to lower learning outcomes because learners are less active during the learning process.”
Naumova, K., Devos, A., Karimireddy, S. P., Jaggi, M., & Hartley, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01226-1 · Cited by 7
Explainable AI in Manufacturing
Zeiser, T., Ehret, D., Lutz, T., & Saar, J. (2024) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc61926.2024.10794363 · Cited by 6
“In the current literature, XAI is mainly used in industries such as finance, healthcare or law [5], [8], [39], [40].”
Scalco, E., Rizzo, G., Bertolino, N., & Mastropietro, A. (2025) · Physica Medica · doi:10.1016/j.ejmp.2025.104978 · Cited by 6
Jiang, C., Xing, X., Yang, N., Fang, Y., Zhang, S., Walsh, S. L. F., Yang, G., & Shen, D. G. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103604 · Cited by 6
Visualizing UNet Decisions: An Explainable AI Perspective for Brain MRI Segmentation
Mala, D. J., Chattopadhyay, M., Mukhopadhyay, P., & Sinha, R. (2025) · IEEE Access · doi:10.1109/access.2025.3592239 · Cited by 7
“One of the most straightforward techniques in the explainable AI family is Vanilla Gradient [20].”
Pomohaci, M. D., Grasu, M. C., Dumitru, R. L., Toma, M., & Lupescu, I. G. (2023) · Diagnostics · doi:10.3390/diagnostics13091663 · Cited by 6
Panigrahi, G. R., Sethy, P. K., Borra, S. P. R., Barpanda, N. K., & Behera, S. K. (2023) · Revue d intelligence artificielle · doi:10.18280/ria.370318 · Cited by 7
“vision applications, including image categorization, with unrivaled accuracy and robustness [6, 11, 25-27].”
BackgroundRecent Challenges in Medical Science using Machine Learning Techniques: A Review
Shah, M., Padhiyar, P., Parmar, K., Patel, J. A., Panesar, S. F., & Raja, K. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290424 · Cited by 6
“In this review article, Amitojdeep Singh et al. (2020) [10] offers a comprehensive summary of the existing uses of explainable deep learning for various medical imaging tasks.”
BackgroundKujime, K., Seno, H., Nakajima, K., Yamazaki, M., Sakuma, I., Yamagata, K., KUSANO, K. F., & Tomii, N. (2024) · Journal of Arrhythmia · doi:10.1002/joa3.13096 · Cited by 6
“14 Explainable models that show which features of the image were focused on using gradients of the neural network have been proposed in the field of medical image analysis.”
BackgroundInternet of Things and Machine Learning for Smart Healthcare
da Costa, C. A., Zeiser, F. A., Righi, R. D. R., Antunes, R. S., Alegretti, A. P., Bertoni, A. P. S., Ramos, G. D. O., de Mello, B. H., Vanin, F. N. D. S., Bertoletti, O. A., et al. (2024) · Studies in computational intelligence · doi:10.1007/978-981-97-5624-7_3 · Cited by 9
Applications and Integration of Radiomics for Skull Base Oncology
Patel, R. V., Groff, K. J., & Bi, W. L. (2024) · Advances in experimental medicine and biology · doi:10.1007/978-3-031-64892-2_17 · Cited by 6
Deep Learning Research Directions in Medical Imaging
Simionescu, C., & Iftene, A. (2022) · Mathematics · doi:10.3390/math10234472 · Cited by 8
Decision Explanation: Applying Contextual Importance and Contextual Utility in Affect Detection.
Fouladgar, N., Alirezaie, M., & Främling, K. (2020) · Aaltodoc (Aalto University) · Cited by 7
Topo-Net: Retinal Image Analysis with Topological Deep LearningPreprint
Ahmed, F., & Coşkunuzer, B. (2024) · medRxiv · doi:10.1101/2024.02.03.24302291 · Cited by 7
“Machine Learning in Retinal Image Analysis: In the past decade, ML tools have been widely employed in medical image analysis [7], [8].”
MethodPathmakumara, H. C., & Perera, G. (2025) · medRxiv · doi:10.1101/2025.10.08.25337634 · Cited by 6
Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive SurveyPreprint
Dagnaw, G. H., Zhu, Y., Maqsood, M. H., Yang, W., Dong, X., Yin, X., & Liew, A. W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07148 · Cited by 17
“Biomedical image analysis is fundamental to modern medicine and life science research, enabling the extraction of critical information from diverse imaging modalities [203].”
Length-scale study in deep learning prediction for non-small cell lung cancer brain metastasis
Zhou, H., Lin, S., Watson, M., Bernadt, C. T., Zhang, O., Liao, L., Govindan, R., Côté, R. J., & Yang, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-73428-2 · Cited by 6
Vo, H.-T., Thien, N. N., Mui, K. C., Le, H. L., & Tien, P. P. (2024) · CTU Journal of Innovation and Sustainable Development · doi:10.22144/ctujoisd.2024.320 · Cited by 5
“XAI has succeeded in various fields such as medicine (Singh et al., 2020; Schutte et al., 2021; Gaur et al., 2022) agriculture (Quach et al., 2023) and traffic classification (Ahn et al., 2020), making XAI more transparent and accessible in these areas.”
BackgroundSolving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks
Mamalakis, M., Mamalakis, A., Agartz, I., Mørch-Johnsen, L. E., Murray, G. K., Suckling, J., & Lió, P. (2025) · AI Open · doi:10.1016/j.aiopen.2025.02.001 · Cited by 10
“Perturbation techniques assess the sensitivity of an AI prediction to certain input features (Mamalakis et al., 2022), by systematically perturbing sub-groups of the input data (Tjoa and Guan, 2020; Singh et al., 2020).”
Explainable AI in Medical Imaging, Personalized Medicine, and Bias Reduction
Komal, Sethi, G. K., Sharma, S., & Kumar, R. (2025) · Wiley eBooks · doi:10.1002/9781394249312.ch21 · Cited by 7
Kumar, M. P., & Balakrishnan, S. M. (2025) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.70121 · Cited by 5
“Critical applications such as medical image analysis and autonomous cars require explainable AI components to explain the fairness of the decision-making process [7].”
InfluentialExplainable AI for raising confidence in deep learning‐based tumor tracking models
Grama, D. I., Dahele, M. R., Slotman, B. J., & Verbakel, W. F. A. R. (2025) · Medical Physics · doi:10.1002/mp.17940 · Cited by 6
“10 The former consists in removing or modifying certain input features and measuring the impact on the model’s output, while the latter uses the model’s gradients to generate attribution maps, which is less computationally expensive.”
Towards a transparent and interpretable AI model for medical image classifications
Wen, B., Wu, Y., Daqqaq, T. S., & Chaddad, A. (2025) · Cognitive Neurodynamics · doi:10.1007/s11571-025-10343-w · Cited by 6
“Furthermore, in [29], Grad-CAM and attention techniques were introduced to improve the visual interpretability of deep models, highlighting advances in disease detection and diagnosis.”
Mazhar, K., Dwivedi, P., & Kant, V. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70548 · Cited by 5
Wang, H. J., Shi, G., Liu, X., Zhang, J., Chen, S., Shi, J., Li, Y., Yue, G., & Wu, Y. (2026) · Expert Systems · doi:10.1111/exsy.70259 · Cited by 8
“…of feature weights: For diabetes prediction, after training a random forest model based on patient physiological indicators like blood glucose and BMI, LIME can output the contribution weights of each indicator, helping clinicians quickly identify key diagnostic factors (Singh et al. 2020).”
XAI Applications in Medical Imaging: A Survey of Methods and Challenges
Tulsani, V., Sahatiya, P., Parmar, J., & Parmar, J. (2023) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v11i9.8332 · Cited by 6
Advancements in Diabetic Retinopathy and Cataract Identification Through Deep Learning
Raman, R., Kumar, V., Saini, D., Rabadiya, D., Patre, S., & Meenakshi, R. (2024) · International Conference on Data Science and Network Security (ICDSNS) · doi:10.1109/icdsns62112.2024.10691108 · Cited by 5
Explainable Cancer Segmentation Through Classification
Dugăeșescu, A., Chiru, C., Nan, M., Trăşcău, M., & Florea, A. M. (2024) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp63557.2024.10793024 · Cited by 5
“Among the widely used attribution-based methods for ex-plainability [24], the Class Activation Maps (CAM) methodology provides a valuable approach for constructing visual explanations as a linear combination of feature maps from a target layer – usually the last convolutional layer as it has the…”
Ayoob, M., Nettasinghe, O., Sylvester, V., Bowala, H., & Mohideen, H. (2025) · Applied Computer Systems · doi:10.2478/acss-2025-0002 · Cited by 5
“Secondly, we plan to delve deeper into the model’s inner workings by exploring more advanced explainable AI techniques [22].”
Exploring the Power of Dual Deep Learning for Fake News Detection
Moalla, H., Abid, H., Sallami, D., Aïmeur, E., & Hamed, B. B. (2025) · Informatica · doi:10.31449/inf.v48i4.5977 · Cited by 12
Willem, T., Wollek, A., Cheslerean-Boghiu, T., Kenney, M., & Buyx, A. (2025) · JMIR Medical Informatics · doi:10.2196/59452 · Cited by 5
“…to helping choose which data to include for a model, data feature selection methods have been used to create new insights about certain statistical connections of variables and, thus, strengthen the explainability of a model [47] or find clues about the root causes of certain diseases [48].”
Bayesian Quantum Orthogonal Neural Networks for Anomaly Detection
Mathur, N., Coyle, B., Jain, N., Raj, S., Tandon, A., Krauser, J. S., & Stoessel, R. (2025) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce65121.2025.00242 · Cited by 6
“Online monitoring of additive manufacturing (AM) generates vast amounts of data that must also be analysed for potential deviations and defect detection.”
Shi, X., Silva, D., Aslan, Y., Ekmekcioǧlu, E., & KONDOZ, A. M. (2021) · Entropy · doi:10.3390/e24010067 · Cited by 5
Ultrasound COVID-19 Classification Based on the Novel Module-Based Dual-Path Network
Fang, L., & Wang, X. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3208217 · Cited by 6
Distance-Aware eXplanation Based Learning
Hagos, M. T., Belton, N., Curran, K. M., & Namee, B. M. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00048 · Cited by 7
“While model explanation methods have been proposed and continue to be used to tackle the “ black-box ” nature of deep learning models [5], [6], they can also be used in an interactive learning approach to promote a more transparent model training process [7], [8].”
MethodZeng, Y., Zhang, X., Wang, J., Usui, A., Ichiji, K., Bukovský, I., Chou, S., Funayama, M., & Homma, N. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-00974-6 · Cited by 7
“Perturbation-based methods are intuitive and simple to implement but are also time consuming, and gradient-based methods could be of low quality and noisy or show false confidence in visualization results due to the global pooling operation and the gradient vanishing issue in models [16, 26].”
BackgroundMethodAnnotation and segmentation of diabetic retinopathy lesions: an explainable AI application
Kheradfallah, H., Balaji, J. J., Jayakumar, V., Rasheed, M. A., & Lakshminarayanan, V. (2022) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2612576 · Cited by 5
“XAI has been used in various clinical imaging studies.(12) XAI methods give relatively good results when compared with segmentation based on clinician response.”
MethodSelf-citationAn Explainable AI-Based Skin Disease Detection
Ballari, G. S., Giraddi, S. G., Chickerur, S., & Kanakareddi, S. G. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-5331-6_30 · Cited by 5
Cardoen, B., Wong, T., Alan, P., Lee, S., Matsubara, J. A., Nabi, I. R., & Hamarneh, G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0276726 · Cited by 5
“(ii) Explainable AI, where neural networks can, for example, produce the regions of the image that provide the most decisive information supporting the predicted image level label, are covered in more detail in recent reviews [16], [17].”
BackgroundCancer Cell Detection and Classification from Digital Whole Slide Image
Gavade, A. B., Nerli, R. B., Ghagane, S. C., Gavade, P. A., & Bhagavatula, V. S. P. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-19-6880-8_31 · Cited by 5
Explicable AI for surveillance and interpretation of Coronavirus using X-ray imaging
Chauhan, T., & Sonawane, S. S. (2023) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci56872.2023.10099633 · Cited by 5
“It provides an overview of explainable deep learning’s current applications in medical imaging tasks such as neuroimaging, retinal image analysis, chest imaging, CT imaging and tissue imaging [8].”
BackgroundExplainable few-shot learning with visual explanations on a low resource pneumonia dataset
Madan, S., Chaudhury, S., & Gandhi, T. K. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.10.013 · Cited by 5
Jabarulla, M. Y., Uden, T., Jack, T., Beerbaum, P., & Oeltze‐Jafra, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.10255 · Cited by 9
“Attribution maps can be generated using perturbation-based or backpropagation-based methods [19], both commonly used as post-hoc visualization approaches.”
Dong, S., Fu, A., & Liu, J. (2024) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-24-100 · Cited by 5
“However, neural networks are usually seen as powerful “black boxes” with weak interpretability (41).”
BackgroundAttention-Guided U-Net With Grad-CAM for Explainable Polyp Segmentation in Colonoscopy Images
Gupta, D., & Benila, S. (2025) · IEEE Access · doi:10.1109/access.2025.3626159 · Cited by 5
“Patel and Singh [32] provided a comprehensive review of explainable deep learning methods, including Grad-CAM and attention visualization techniques—both integral components of our proposed framework.”
Civit-Masot, J., Luna-Perejón, F., Muñoz-Saavedra, L., Corral, J. M. R., Domínguez-Morales, M., & Civit, A. (2026) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-026-03514-2 · Cited by 6
“These technologies provide information on the classification criteria used in the automatic system [ 15, 16].”
Hasan, E., Fuad, M. T. H., Sharif, O., & Wagler, A. E. (2026) · Journal of Imaging · doi:10.3390/jimaging12030098 · Cited by 4
“Incorporating explainable AI (XAI) techniques will also be essential to improve interpretability and foster clinical trust in automated cellular analysis systems [57,58].”
Li, X., Gao, X., Wang, C., Wang, Q., Li, B., & Wan, K. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3390426 · Cited by 4
“LRP has been applied to interpret DL models in gene analysis, medical diagnosis and other fields [66], [67].”
Data driven cost-sensitive boosted tree for interpretable banking systemic risk prediction
Xia, M., Wang, Z., & Liu, W. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.115664 · Cited by 4
Validation of Artificial Intelligence Computer-Aided Detection of Colonic Neoplasm in Colonoscopy
Lee, H., Chung, J., Kim, K. O., Kwon, K. A., Kim, J. H., Yun, S.-C., Jung, S. W., Sheeraz, A., Yoon, Y. J., Kim, J. H., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14232762 · Cited by 4
“Deep learning in artificial intelligence enables key information to be taken and learnt from typical features of a large amount of data in the medical field [22,23].”
Ogundipe, O., Zhai, B., Kurt, Z., & Woo, W. L. (2025) · Current Bioinformatics · doi:10.2174/0115748936309582240907160359 · Cited by 4
Classification of Histopathology Images by Siamese and Triplet Network Embeddings
Amato, D., Calderaro, S., Bosco, G. L., Rizzo, R., & Vella, F. (2025) · IEEE Access · doi:10.1109/access.2025.3613448 · Cited by 4
“Such classifiers, sometimes indicated as ‘‘inherently explainable’’, are also the ones preferred in the development of interpretable Machine Learning systems [17], [18], [19].”
Deep learning in hepatic oncology imaging: a narrative review of computed tomography applications
Wan, Z., Zhang, X., Jiang, Y., Chai, S., Lyu, C., & Hu, Y. (2025) · ONCOLOGIE · doi:10.1515/oncologie-2025-0279 · Cited by 4
On Explainability in AI-Solutions: A Cross-Domain Survey
Antón, S. D. D., Schneider, D., & Schotten, H. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14862-0_17 · Cited by 6
“In medical applications, compliance and liability play important roles [38,62,60], meaning someone has to be responsible for a decision.”
Sanity Check for Shapley values-based explanations of Deep Neural Networks Predictions
Tatano, R., Mastropietro, A., Busto, E., & Vaccarino, F. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00130 · Cited by 4
“A review of the XAI in the medical field can be found in [5], [6].”
BackgroundSingh, G., & Pal, A. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-2839-8_12 · Cited by 4
Interpretable diagnosis of breast lesions in ultrasound imaging using deep multi-stage reasoning
Cui, K., Liu, W., & Wang, D. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad869f · Cited by 4
“To enhance the interpretability of auxiliary diagnostic methods, many approaches have been applied, such as feature visualization, feature importance analysis, expert systems, and others (Singh et al 2020, Van der Velden et al 2022).”
Syed, M. G., Doney, A. S. F., George, G., Mordi, I. R., & Trucco, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87000-3_12 · Cited by 4
Deep learning for epileptogenic zone delineation from the invasive EEG: challenges and lookouts
Hoogteijling, S., & Zijlmans, M. (2021) · Brain Communications · doi:10.1093/braincomms/fcab307 · Cited by 4
“Explainable AI (xAI) refers to such methods that aim to provide an explanation along with the output of an ML model.(12) These methods can provide a saliency map that highlights important aspects in the input according to the DL model.”
MethodHolmes, G., Frank, E., Fletcher, D., & Sterling, C. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511110 · Cited by 4
“Another avenue is to make use of advances in methods that explain deep model decision-making [18], making them part of the human-in-the-loop experience.”
BackgroundBastida, J. P. C., Pearson, E. A., & Al‐Hallaq, H. (2022) · Journal of medical imaging · doi:10.1117/1.jmi.9.4.045002 · Cited by 5
“To address these concerns, several techniques have been developed to explain or visualize key elements involved in the CNN decision process; 10,11 however, such techniques are not always integrated to DL model development.”
MethodAre Graph Neural Network Explainers Robust to Graph Noises?
Li, Y., Verma, S., Yang, S., Zhou, J. J., & Chen, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22695-3_12 · Cited by 4
Machine learning approaches for COVID-19 detection from chest X-ray imaging: A Systematic Review
Arteaga-Arteaga, H. B., delaPava, M., Mora-Rubio, A., Bravo-Ortíz, M. A., Alzate-Grisales, J. A., Arias-Garzón, D., López-Murillo, L. H., Buitrago-Carmona, F., Villa-Pulgarín, J. P., Mercado-Ruiz, E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.05615 · Cited by 5
“One of the main drawbacks of deep learning is the lack of interpretability, which has imitated its application in some areas [Singh et al., 2020].”
BackgroundRath, B., Panigrahy, V., & Mishra, T. K. (2023) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit59427.2023.10431366 · Cited by 4
Unveiling the Depths of Explainable AI: A Comprehensive Review
Khan, W., & Ishrat, M. (2024) · Advances in systems analysis, software engineering, and high performance computing book series · doi:10.4018/979-8-3693-0968-1.ch004 · Cited by 5
Explainable artificial intelligence (XAI) for healthcare decision-making
Zeng, Z. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/155849 · Cited by 6
Fixing confirmation bias in feature attribution methods via semantic matchPreprint
Cinà, G., Fernández-Llaneza, D., Deponte, L., Mishra, N., Röber, T. E., Pezzelle, S., Calixto, I., Goedhart, R., & Birbil, Ş. İ. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.00897 · Cited by 5
“We refer to Abhishek and Kamath [2022] and Singh et al. [2020] for a more detailed account of different methods.”
MethodWang, R., Zhao, J., Liu, X., Tang, X., Shi, Y., & Wei, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4051336/v1 · Cited by 4
Mamalakis, M., Macfarlane, S. C., Notley, S. V., Gad, A. K. B., & Panoutsos, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109052 · Cited by 6
An AI Decision System to Predict Lung Nodules through Localization from Chest X-ray Images
Koyyada, S. P., & Singh, T. P. (2023) · doi:10.1109/icsc60394.2023.10441301 · Cited by 4
“XAI methods are adopted in [45]–[49] and [50] while identifying several lung diseases.”
MethodExplainable Anomaly Detection of 12-Lead ECG Signals Using Denoising Autoencoder
Hribar, R., & Torkar, D. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-54049-3_7 · Cited by 6
Systematic Review of XAI Tools for AI-HCI Research
Alaqsam, A., & Sas, C. (2024) · Electronic workshops in computing · doi:10.14236/ewic/bcshci2024.6 · Cited by 3
Comparison of CNNs and Transformer Models in Diagnosing Bone Metastases in Bone Scans Using Grad-CAM
Pak, S., Son, H. J., Kim, D., Woo, J. Y., Yang, I., Hwang, H. S., Rim, D., Choi, M. S., & Lee, S. H. (2025) · Clinical Nuclear Medicine · doi:10.1097/rlu.0000000000005898 · Cited by 5
“37,38 In our study, we encountered severe class imbalance between the bone metastasis-positive and metastasis-neg-ative groups.”
Panigrahi, I., Kim, S. S. Y., Liaqat, A., Jinturkar, R., Russakovsky, O., Fong, R., & Abtahi, P. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719730 · Cited by 5
Enhancing Interpretability in CNN Models for Medical Imaging using Attention Maps
Saraswathi, S. (2025) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc65190.2025.11140850 · Cited by 3
“XAI for MI [14]From a review by Tjoa and Guan [14], of XAI in MI, centred on future directions for user-centred design in medicine.”
Liang, N., Hui, Z., & Xia, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329487 · Cited by 3
“It processes input data using a multi-head self-attention mechanism and a feedforward neural network (FNN) [25].”
Aljaddouh, B., Malathi, D., & Alaswad, F. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00535-y · Cited by 5
Mustapha, R., Ganeshan, B., Ellis, S., Dolcetti, L., Tharmakulasingam, M., DeSouza, K., Jiang, X., Savage, C., Lim, S. K. X., Chan, E., et al. (2026) · EBioMedicine · doi:10.1016/j.ebiom.2026.106173 · Cited by 4
“63 Autoencoders are able to learn semantically meaningful representations of images in an unsupervised manner and offer some advantages: training on an external data set and transparency through the exploration of the latent space parameters.”
Mewada, H. K., Pires, I. M., Thakkar, H. K., & Patel, A. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110104 · Cited by 3
Kim, M., & Joo, S. (2022) · Sensors · doi:10.3390/s22155896 · Cited by 3
A Practical Upper Bound for the Worst-Case Attribution Deviations
Wang, F., & Kong, A. W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02358 · Cited by 6
“They have been widely used in a number of critical realworld applications, such as risk management [2], medical imaging [24, 29] and drug discovery [13].”
BackgroundXiong, F., & Wei, Y. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01231-6 · Cited by 3
“Typical model-specific interpretation modules include attention mechanism, expert knowledge, textual proof, etc. [55].”
Expert-level diagnosis of pediatric posterior fossa tumors via consistency calibration
Sun, C., Yan, Z., Zhang, Y., Tian, X., & Gong, J. F. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111919 · Cited by 3
Exploring Contextual Importance and Utility in Explaining Affect Detection
Fouladgar, N., Alirezaie, M., & Främling, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77091-4_1 · Cited by 3
Na, Y., Joo, H., Trang, L. T., Quan, L. D. A., & Woo, J. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.906616 · Cited by 3
“An occlusion sensitivity map was used to select sensitive EEG channels to predict speech intelligibility scores (Esmaeilzadeh et al., 2018; Singh et al., 2020).”
MethodExplainability of deep learning models in medical image classification
Kolárik, M., Sarnovský, M., Paralič, J., & Butka, P. (2022) · IEEE International Symposium on Computational Intelligence and Informatics (CINTI) · doi:10.1109/cinti-macro57952.2022.10029502 · Cited by 3
Pal, M., & Mistry, S. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0085-5_38 · Cited by 3
Zaunseder, E., Mütze, U., Garbade, S. F., Haupt, S., Kölker, S., & Heuveline, V. (2023) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci52147.2023.10371991 · Cited by 3
“Recently, eXplainable Artificial Intelligence (XAI) techniques have been applied to provide human-understandable interpretations of deep learning algorithms in the medical domain [22].”
BackgroundChandrashekhar, K., Leela, P., Madhavi, K. R., Avanija, J., & Tangudu, N. (2024) · International Conference on Computing for Sustainable Global Development · doi:10.23919/indiacom61295.2024.10498958 · Cited by 3
Объяснительный искусственный интеллект в моделях поддержки принятия решений для Здравоохранения 5.0
Averkin, A. N., & Yarushev, S. (2023) · Computer Tools in Education · doi:10.32603/2071-2340-2023-2-41-61 · Cited by 5
Parra-Mora, E., & Cruz, L. A. D. S. (2022) · IEEE Ecuador Technical Chapters Meeting (ETCM) · doi:10.1109/etcm56276.2022.9935750 · Cited by 3
Matheny, M. E., Ohno-Machado, L., Davis, S. E., & Nemati, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00031-0 · Cited by 3
Matsuyama, E., Takehara, M., Takahashi, N., & Watanabe, H. (2023) · Open Journal of Medical Imaging · doi:10.4236/ojmi.2023.133007 · Cited by 4
Graph-Based 3-Dimensional Spatial Gene Neighborhood Networks of Single Cells in Gels and Tissues
Fang, Z., Krusen, K., Priest, H., Wang, M., Kim, S., Sriram, A., Yellanki, A., Singh, A. K., Horwitz, E. M., & Coskun, A. F. (2025) · BME Frontiers · doi:10.34133/bmef.0110 · Cited by 2
“Explainability has been especially important in the biomedical field as it can find connections between original data and the prediction, as well as provide transparent and tracible predictions [ 47 , 48 ].”
Self-citationYang, Y., Cheng, W.-H., Lin, E.-T., Liu, A.-S., Ko, C., Huang, C., Tsai, C., & Fu, L. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01534-2 · Cited by 2
Wei, J., Chen, B., Jia, B., Gao, Z., Xia, P. C., Peng, K.-D., Ren, J., Ji, X.-H., & Sui, Z.-J. (2025) · Case Studies in Construction Materials · doi:10.1016/j.cscm.2025.e05243 · Cited by 6
Metadata
- Authors
- Singh, A., Sengupta, S., & Lakshminarayanan, V.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Singh, Amitojdeep
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2020-06-20
- Volume
- 6
- Issue
- 6
- Pages
- 52
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC8321083
- OpenAlex
- W3031116957
- Semantic Scholar
- af6af40155860ac2fd36680e63686473803b5f52
- Cited by
- 718
- References
- 75
- Citations by year
- 2026: 82 · 2025: 135 · 2024: 132 · 2023: 154 · 2022: 130 · 2021: 77 · 2020: 8
- FWCI
- 3.17
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Compute Canada
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable deep learning, medical image analysis, model interpretability, black-box models, clinical deployment, feature importance, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.