Explainable Deep Learning Models in Medical Image Analysis
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Attributions of VGG-16 with images from Imagenet using the methods implemented in [19].](https://figures.cochat.ai/paper-figures/a357c2dc02ed4a63a419e6c762416bfb/explainable-deep-learning-models-in-medical-image-analysis-figure-3.jpg)
![Figure 4 Example of heat maps from a retinal OCT image [40].](https://figures.cochat.ai/paper-figures/002116f86db043c7aaeedcab11546bae/explainable-deep-learning-models-in-medical-image-analysis-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More details about attention mechanisms in neural networks is discussed in [51].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An earlier study for breast mass classification from mammograms [40] using two different CNNs - AlexNet [41] and GoogleNet [42] - employed saliency maps to visualize the image features.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Both approaches are beneficial for medical imaging datasets which tend to be relatively smaller than computer vision benchmarks like ImageNet [33].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“rey patch and hence occluding them from the system’s view. It can provide both positive and negative evidence by highlighting the responsible features. This technique was applied by Zeiler and Fergus [18] to the CNN for the image classification task. Occlusion is the benchmark for any attribution study as it is a simpl…”
MethodWold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606 · cited 1×
“Principal component analysis (PCA) [12], t-Distributed Stochastic Neighbor Embedding (t-SNE) [13] are some common examples of these methods.”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 2×
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
Background
Research works printed but not linked: 2
- El-Baz, A.S. (2020). Recent Advances in Deep Learning Applications for Retinal Diagnosis using OCT. State of the Art in Neural Networks, Elsevier. in press.
- Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
Cited by
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Explanations for deep learningbased methods are more urged in this field than in other fields, to gain the trust of physicians, regulators as well as the patients [154].”
BackgroundAdvances in medical image analysis with vision Transformers: A comprehensive review
Azad, R., Kazerouni, A., Heidari, M., Aghdam, E. K., Molaei, A., Jia, Y., Jose, A., Roy, R., & Merhof, D. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103000 · Cited by 485
Deep Learning for Medical Anomaly Detection – A Survey
Fernando, T., Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2022) · ACM Computing Surveys · doi:10.1145/3464423 · Cited by 399
“Finally, as per [103] model interpretation techniques in the medical domain can be broadly categorised into attribution based and non-attribution based methods.”
BackgroundMethodInfluentialSchwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328
“Application domain natural language processing [34] medicine [124,131] recommendation systems [101,148]”
BackgroundDeep Learning for Medical Image-Based Cancer Diagnosis
Jiang, X., Hu, Z., Wang, S., & Zhang, Y. (2023) · Cancers · doi:10.3390/cancers15143608 · Cited by 345
Explainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372
Muhammad, D., & Bendechache, M. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.08.005 · Cited by 258
“The attribution and non-attribution methodologies are utilised as XAI tools for dissecting and understanding the predictive decision-making process of ML and DL models [31].”
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283
“It is easy to understand a model ’ s prediction by a user if similar example images can be provided from the dataset [28].”
BackgroundMethodCTformer: convolution-free Token2Token dilated vision transformer for low-dose CT denoising
Wang, D., Fan, F., Wu, Z., Liu, R., Wang, F., & Yu, H. (2023) · Physics in Medicine & Biology · doi:10.1088/1361-6560/acc000 · Cited by 250
“Other than the effectiveness, a transformer is more desirable for physicians because it is self-explanatory [26], e.”
BackgroundIqbal, S., N. Qureshi, A., Li, J., & Mahmood, T. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09899-9 · Cited by 210
“Ultimately, there will likely be several rules and laws governing artificial intelligence morality on healthcare systems [442, 443].”
BackgroundEnnab, M., & Mcheick, H. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1444763 · Cited by 238
Abbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236
“This was followed by oncology, where XAI techniques such as SHAP and concept bottlenecks supported cancer prognosis, recurrence prediction, and treatment planning [106,114,117].”
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
“XAI is also increasingly being researched and applied in various domains, including but not limited to healthcare [5, 71, 75, 100, 117, 119, 135, 141], autonomous driving [9, 77, 94], energy and”
BackgroundA Survey on Deep Learning for Neuroimaging-Based Brain Disorder Analysis
Zhang, L., Wang, M., Liu, M., & Zhang, D. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00779 · Cited by 220
“Some studies begin to explore the interpretability of deep learning in medical image analysis, and aim to show the features that most influence the predictions (Singh et al., 2020).”
BackgroundA survey on the interpretability of deep learning in medical diagnosis
Teng, Q., Liu, Z., Song, Y., Han, K., & Lu, Y. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00960-4 · Cited by 198
“[11] analyzed the interpretability of deep learning models from the applications of different medical imaging tasks (CT imaging, X-ray imaging, etc.”
BackgroundA Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“In [33], a survey of explainable deep learning applications for medical imaging tasks from the perceptive of deep learning researcher, developer, and end-user is presented.”
BackgroundExplainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2024) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 161
Allgaier, J., Mulansky, L., Draelos, R. L., & Pryss, R. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102616 · Cited by 170
Guidelines and evaluation of clinical explainable AI in medical image analysis
Jin, W., Li, X., Fatehi, M., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102684 · Cited by 199
From Blackbox to Explainable AI in Healthcare: Existing Tools and Case Studies
Srinivasu, P. N., Sandhya, N., Jhaveri, R. H., & Raut, R. (2022) · Mobile Information Systems · doi:10.1155/2022/8167821 · Cited by 171
“*is field benefits AI for various activities, including decision making, predictive evaluation, analysis, risk evaluation, and regulation [21].”
BackgroundDeep learning for Alzheimer's disease diagnosis: A survey
Khojaste-Sarakhsi, M., Haghighi, S. S., Ghomi, S. M. T. F., & Marchiori, E. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102332 · Cited by 194
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110787 · Cited by 169
Explainable AI: current status and future directionsPreprint
Gohel, P., Singh, P., & Mohanty, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07045 · Cited by 248
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“Research was conducted on how saliency maps can enhance the interpretability of convolutional neural networks (CNNs) by visually displaying feature importance, leading to improved comprehension in fields such as medical image analysis [56].”
Gadermayr, M., & Tschuchnig, M. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102337 · Cited by 226
Adak, A., Pradhan, B., & Shukla, N. (2022) · Foods · doi:10.3390/foods11101500 · Cited by 193
“According to Singh et al. [50], DL is significant in medical diagnostic tasks and outperformed human experts.”
BackgroundA survey on graph-based deep learning for computational histopathology
Ahmedt-Aristizabal, D., Armin, M. A., Denman, S., Fookes, C., & Petersson, L. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.102027 · Cited by 147
“Thus, more work should be conducted to investigate how a pathologist could refine a graph model decision via a human-in-the-loop system [128], [129] Such approaches provide an important safety mechanism for detecting and correcting algorithmic errors that may occur.”
BackgroundArtificial Intelligence for Alzheimer’s Disease: Promise or Challenge?
Fabrizio, C., Termine, A., Caltagirone, C., & Sancesario, G. (2021) · Diagnostics · doi:10.3390/diagnostics11081473 · Cited by 146
Müller, D., Soto‐Rey, I., & Krämer, F. (2022) · IEEE Access · doi:10.1109/access.2022.3182399 · Cited by 183
“For computer vision tasks like image classification, deep convolutional neural networks are state-of-the-art and unmatched in accuracy and robustness [5], [42]–[44].”
BackgroundAbdullah, T. A. A., Zahid, M. S. M., & Ali, W. (2021) · Symmetry · doi:10.3390/sym13122439 · Cited by 179
Domain Generalization for Medical Image Analysis: A Review
Yoon, J. S., Oh, K., Shin, Y., Mazurowski, M. A., & Suk, H. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3507831 · Cited by 212
“For a general overview of interpretable AI for MedIA, readers are referred to the survey by Singh et al. [163] and van der Velden et al. [164].”
A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging
Gulum, M. A., Trombley, C. M., & Kantardzic, M. (2021) · Applied Sciences · doi:10.3390/app11104573 · Cited by 145
“In the review by [76], the authors provide a review of explainable deep learning models for medical imaging analysis.”
Interpretable machine learning for dementia: a systematic review
Martin, S. A., Townend, F. J., Barkhof, F., & Cole, J. H. (2023) · Alzheimer's & Dementia · doi:10.1002/alz.12948 · Cited by 149
Application of Artificial Intelligence in Orthodontics: Current State and Future Perspectives
Liu, J., Zhang, C., & Shan, Z. (2023) · Healthcare · doi:10.3390/healthcare11202760 · Cited by 177
“Explainable AI (XAI) techniques aim to demystify the underlying logic and make the AI algorithms more transparent, explainable and trustable [182,183].”
Saheb, T., Saheb, T., & Carpenter, D. O. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104660 · Cited by 162
Houssein, E. H., Gamal, A. M., Younis, E. M. G., & Mohamed, E. (2025) · Cluster Computing · doi:10.1007/s10586-025-05281-5 · Cited by 172
Machine learning and orthodontics, current trends and the future opportunities: A scoping review
Mohammad‐Rahimi, H., Nadimi, M., Rohban, M. H., Shamsoddin, E., Lee, V. Y., & Motamedian, S. R. (2021) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2021.02.013 · Cited by 174
A Distinctive Explainable Machine Learning Framework for Detection of Polycystic Ovary Syndrome
Khanna, V. V., Chadaga, K., Sampathila, N., Prabhu, S., Bhandage, V., & Hegde, G. K. (2023) · Applied System Innovation · doi:10.3390/asi6020032 · Cited by 129
“Nevertheless, Important insights into the classifier predictions makes them, meaningful [47].”
Deep Learning in Medical Image Analysis
Zhang, Y., Górriz, J. M., & Dong, Z. (2021) · Journal of Imaging · doi:10.3390/jimaging7040074 · Cited by 138
“Singh and Sengupta (2020) [22] from Canada present a review of the current applications of explainable deep learning for different medical imaging tasks.”
BackgroundExplainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135
Fraiwan, M., & Khasawneh, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.00237 · Cited by 56
Sadr, S., Mohammad‐Rahimi, H., Motamedian, S. R., Zahedrozegar, S., Motie, P., Vinayahalingam, S., Dianat, O., & Nosrat, A. (2022) · Journal of Endodontics · doi:10.1016/j.joen.2022.12.007 · Cited by 143
A review of explainable and interpretable AI with applications in COVID‐19 imaging
Fuhrman, J. D., Gorre, N., Hu, Q., Li, H., El Naqa, I., & Giger, M. L. (2022) · Medical Physics · doi:10.1002/mp.15359 · Cited by 120
An analysis of explainability methods for convolutional neural networks
Vonderhaar, L., Elvira, T., & Ochoa, O. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105606 · Cited by 151
Hassan, S. U., Abdulkadir, S. J., Zahid, M. S. M., & Al-Selwi, S. M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109569 · Cited by 138
A Survey on XAI for 5G and Beyond Security: Technical Aspects, Challenges and Research Directions
Senevirathna, T., La, V. H., Marchal, S., Siniarski, B., Liyanage, M., & Wang, S. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3437248 · Cited by 148
“The XAI methods can be divided into multiple categories based on various criteria [37], [57].”
A Comprehensive Review of Explainable Artificial Intelligence (XAI) in Computer Vision
Cheng, Z. Q., Wu, Y. P., Li, Y., Cai, L., & Ihnaini, B. (2025) · Sensors · doi:10.3390/s25134166 · Cited by 144
“Building upon the quantitative evaluation results, this section offers a method-by-method comparative analysis [41,42].”
A Review of Recent Advances in Deep Learning Models for Chest Disease Detection Using Radiography
Ait Nasser, A., & Akhloufi, M. A. (2023) · Diagnostics · doi:10.3390/diagnostics13010159 · Cited by 121
“It is very important to highlight the features that the model extracts to provide a decision, as performed by Singh et al. [164] and Chetoui et al. [140] who proposed explainable approaches for COVID-19 detection based on Grad-CAM algorithm [141] to output the heatmaps and show how the most…”
MethodMultimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 146
Wang, N., Zhang, H., Dahal, A., Cheng, W., Zhao, M. J., & Lombardo, L. (2024) · Geoscience Frontiers · doi:10.1016/j.gsf.2024.101800 · Cited by 123
“We recall here that SHAP values can be estimated using a number of approaches ranging from Kernel SHAP (e.g., Roshan and Zafar, 2022), Tree SHAP (e.g., Wang et al. , 2022a), and Deep SHAP (e.g., Singh et al. , 2020).”
MethodExplainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review
de Vries, B. M., Zwezerijnen, G. J. C., Burchell, G. L., van Velden, F. H. P., Menke-van der Houven van Oordt, C. W., & Boellaard, R. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1180773 · Cited by 106
“CAM is one of the most well-known model specific attribution methods (28, 29).”
MethodRaghavan, K., Sivaselvan, B., & Kamakoti, V. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17776-7 · Cited by 116
Żuraw, A., & Aeffner, F. (2021) · Veterinary Pathology · doi:10.1177/03009858211040484 · Cited by 120
“112,121 Similar to traditional image analysis, image analysis teams can apply deep learning–based approaches using commercially available tools (no AI programming skills required) or independently code custom solutions using flexible programming languages.”
BackgroundSkeletal Fracture Detection with Deep Learning: A Comprehensive Review
Su, Z., Adam, A., Nasrudin, M. F., Ayob, M. B., & Punganan, G. (2023) · Diagnostics · doi:10.3390/diagnostics13203245 · Cited by 119
“Compared with traditional methods, deep learning algorithms have the strength to extract features automatically [3,4].”
BackgroundMethodMohammad‐Rahimi, H., Nadimi, M., Ghalyanchi‐Langeroudi, A., Taheri, M., & Ghafouri‐Fard, S. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.638011 · Cited by 116
Civit-Masot, J., Bañuls-Beaterio, A., Domínguez-Morales, M., Rivas-Perez, M., Muñoz-Saavedra, L., & Corral, J. M. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107108 · Cited by 111
Explainable deep learning in healthcare: A methodological survey from an attribution view
Jin, D., Sergeeva, E., Weng, W., Chauhan, G., & Szolovits, P. (2022) · WIREs Mechanisms of Disease · doi:10.1002/wsbm.1548 · Cited by 72
Zou, L., Goh, H. L., Liew, C. J. Y., Quah, J. L., Gu, G. T., Chew, J. J., Kumar, M. P., Ang, C. G. L., & Ta, A. W. A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3153754 · Cited by 92
“However, this approach is unable to distinguish between positive and negative evidence due to absolute values of the partial derivative [31].”
BackgroundArtificial Intelligence and Deep Learning for Upper Gastrointestinal Neoplasia
Sharma, P., & Hassan, C. (2021) · Gastroenterology · doi:10.1053/j.gastro.2021.11.040 · Cited by 102
Shrivastava, A. D., Swainston, N., Samanta, S., Roberts, I., Muelas, M. W., & Kell, D. B. (2021) · Biomolecules · doi:10.3390/biom11121793 · Cited by 107
“The existence of recognisable ‘features’ that may be extracted and combined to create meaningful images is what lies at the heart of the kinds of deep learning systems applied in modern, generative image processing [99].”
BackgroundMontero, A. M. B., Bibal, A., Huet‐Dastarac, M., Draguet, C., Valdés, G., Nguyen, D. T. D., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac678a · Cited by 105
“ and ( 2 ) whatneedstobeexplained ( thewholemodelorparticular decisions ) ? performance of ML models ( Singh et al 2020, Huff et al 2021 ) .”
BackgroundGroen, A. M., Kraan, R., Amirkhan, S. F., Daams, J. G., & Maas, M. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110592 · Cited by 89
Deep Learning in Biomedical Optics
Tian, L., Hunt, B., Bell, M. A. L., Yi, J., Smith, J. T., Ochoa, M., Intes, X., & Durr, N. J. (2021) · Lasers in Surgery and Medicine · doi:10.1002/lsm.23414 · Cited by 93
The Role of Generative Adversarial Network in Medical Image Analysis: An In-depth Survey
Alamir, M., & Alghamdi, M. (2022) · ACM Computing Surveys · doi:10.1145/3527849 · Cited by 102
“DL medical systems have produced much interest over the last few years and have been employed deeply in all fields of medicine, from drug identification to medical decision-making, significantly changing the way medicine is practiced [92, 177].”
BackgroundHamida, S. U., Chowdhury, M. J. M., Chakraborty, N. R., Biswas, K., & Sami, S. K. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110149 · Cited by 95
Explainable deep convolutional neural networks for insect pest recognition
Coulibaly, S., Kamsu-Foguem, B., Kamissoko, D., & Traoré, D. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.133638 · Cited by 85
Musulin, J., Štifanić, D., Zulijani, A., Ćabov, T., Dekanić, A., & Car, Z. (2021) · Cancers · doi:10.3390/cancers13081784 · Cited by 91
Ahmed, F., Naz, N. S., Khan, S., Rehman, A. U., Ismael, W. M., & Khan, M. A. (2026) · BMC Medical Imaging · doi:10.1186/s12880-025-02118-w · Cited by 101
Jin, W., Li, X., & Hamarneh, G. (2022) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v36i11.21452 · Cited by 88
“Recently, Singh et al. (2020) evaluated 13 XAI algorithms on classifying eye diseases using retina images and asked 14 clinicians to rate the heatmaps regarding their clinical relevance ( plausibility ).”
BackgroundExplainable Artificial Intelligence for Medical Applications: A Review
Sun, Q., Akman, A., & Schuller, B. W. (2024) · ACM Transactions on Computing for Healthcare · doi:10.1145/3709367 · Cited by 140
Ennab, M., & Mcheick, H. (2022) · Diagnostics · doi:10.3390/diagnostics12071557 · Cited by 76
Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare
Procter, R., Tolmie, P., & Rouncefield, M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3577009 · Cited by 92
“(Singh et al. 2020) In this paper we have sought to explore what trustworthy AI means by reflecting on the nature and the roles of accounts and accountability as practiced within organisational settings and illustrating this with case studies drawn from our earlier work in clinical decision-making.”
Hussain, T., & Shouno, H. (2023) · Information · doi:10.3390/info14120642 · Cited by 89
“Many researchers have applied attribution-based explainability approaches to interpret DL [52].”
Review of medical image processing using quantum-enabled algorithms
Yan, F., Huang, H., Pedrycz, W., & Hirota, K. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10932-x · Cited by 94
Grad-CAM (Gradient-weighted Class Activation Mapping): A systematic literature review
Fayyaz, A. M., Abdulkadir, S. J., Talpur, N., Al-Selwi, S. M., Hassan, S. U., & Sumiea, E. H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111200 · Cited by 94
A review of microscopic analysis of blood cells for disease detection with AI perspective
Deshpande, N. M., Gite, S., & Aluvalu, R. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.460 · Cited by 87
Mohammad‐Rahimi, H., Sohrabniya, F., Ourang, S. A., Dianat, O., Aminoshariae, A., Nagendrababu, V., Dummer, P. M. H., Duncan, H. F., & Nosrat, A. (2024) · International Endodontic Journal · doi:10.1111/iej.14128 · Cited by 88
Liu, S.-C., Lai, J., Huang, J.-Y., Cho, C.-F., Lee, P. H., Lu, M.-H., Yeh, C., Yu, J., & Lin, W. (2021) · Cancer Imaging · doi:10.1186/s40644-021-00425-3 · Cited by 78
“In addition to prediction, explaining or interpreting how a deep learning model makes predictions has become more important recently to obtain trust from physicians, patients, regulators and other stakeholders involved [24].”
BackgroundEvaluation of Explainable Deep Learning Methods for Ophthalmic Diagnosis
Singh, A., Balaji, J. J., Rasheed, M. A., Jayakumar, V., Raman, R., & Lakshminarayanan, V. (2021) · Clinical ophthalmology · doi:10.2147/opth.s312236 · Cited by 79
Self-citationUsing generative AI to investigate medical imagery models and datasets
Lang, O., Stupp, D., Traynis, I., Cole-Lewis, H. J., Bennett, C. R., Lyles, C. R., Lau, C., Irani, M., Semturs, C., Webster, D. R., et al. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.105075 · Cited by 79
“Among explainability methods in computer vision, 21 the output of most techniques 22 – 24 is a heatmap of per-pixel importance, for instance based on the saliency of the image features.”
MethodPreston, F. G., Meng, Y., Burgess, J., Ferdousi, M., Azmi, S., Petropoulos, I. N., Kaye, S., Malik, R. A., Zheng, Y., & Alam, U. (2022) · Diabetologia · doi:10.1007/s00125-021-05617-x · Cited by 70
“The primary modality used to explicate AI-based DLA within medical imaging diagnostics is attribution based, where the contribution to the output decision of each input feature is determined, allowing the generation of heat-maps known as attribution maps [24].”
BackgroundMethodInterpretable AI for bio-medical applications
Sathyan, A., Weinberg, A. I., & Cohen, K. D. O. (2022) · Complex Engineering Systems · doi:10.20517/ces.2022.41 · Cited by 80
“There are ways to classify the XAI approaches by several criteria[13] such as: model dependency, sample particularity, explainability timing and the interaction between the explanation to the model itself.”
MethodGunashekar, D. D., Bielak, L., Hägele, L., Oerther, B., Benndorf, M., Grosu, A., Brox, T., Zamboglou, C., & Bock, M. (2022) · Radiation Oncology · doi:10.1186/s13014-022-02035-0 · Cited by 76
“However, these methods are widely used for the interpretation of classification decisions [29], but have rarely been applied for segmentation tasks.”
MethodCollins, T., Maktabi, M., Barberio, M., Bencteux, V., Jansen‐Winkeln, B., Chalopin, C., Marescaux, J., Hostettler, A., Diana, M., & Gockel, I. (2021) · Diagnostics · doi:10.3390/diagnostics11101810 · Cited by 70
“Furthermore, we aim to analyze the features learned by the 3DCNN using ‘explainable AI’ techniques [53], especially to determine the spectral bands that had the strongest influence on the prediction.”
MethodA review of explainable AI in the satellite data, deep machine learning, and human poverty domain
Hall, O., Ohlsson, M., & Rognvaldsson, T. S. (2022) · Patterns · doi:10.1016/j.patter.2022.100600 · Cited by 78
Video-Based Driver Drowsiness Detection With Optimised Utilization of Key Facial Features
Yang, L., Yang, H., Wei, H., Hu, Z., & Lv, C. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3346054 · Cited by 77
“Due to the advantages of non-contact, low cost, and convenience to use, the methods based on image processing are gradually applied to various fields, such as intelligent driving [26], [27], security monitoring [28], [29], intelligent manufacturing [30], [31], and medical diagnosis [32], [33].”
MethodBharadwaj, U. U., Christine, M., Li, S., Chou, D., Pedoia, V., Link, T. M., Chin, C. T., & Majumdar, S. (2023) · European Radiology · doi:10.1007/s00330-023-09483-6 · Cited by 75
“Notwithstanding the apparent advantages, adoption of deep learning in clinical settings is limited due to concerns regarding the black-box nature of the algorithms and lack of clinical interpretability [11, 14, 15].”
BackgroundHe, T., Fong, J. N., Moore, L. W., Ezeana, C. F., Victor, D., Divatia, M., Vasquez, M., Ghobrial, R. M., & Wong, S. T. C. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.101894 · Cited by 63
Jiménez-Mesa, C., Arco, J. E., Martínez-Murcia, F. J., Suckling, J., Ramírez, J., & Górriz, J. M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106984 · Cited by 67
“Furthermore, these algorithms hold the promise of enhancing image reconstruction, reducing noise, and mitigating artifacts, thereby elevating both the overall image quality and its inter-pretability [13 – 15].”
Interpretability of deep neural networks: A review of methods, classification and hardware
Antamis, T., Drosou, A., Vafeiadis, T., Nizamis, A., Ioannidis, D., & Tzovaras, D. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128204 · Cited by 70
Ahmed, M., Bibi, T., Khan, R. A., & Nasir, S. (2024) · International Multi-Topic Conference · doi:10.1109/inmic64792.2024.11004362 · Cited by 75
“The emergence of XAI aims to address these issues of transparency and explainability, advocating the creation of accurate AI models that are comprehensible and trustworthy to users [2].”
Colour and Texture Descriptors for Visual Recognition: A Historical Overview
Bianconi, F., Fernández, A., Smeraldi, F., & Pascoletti, G. (2021) · Journal of Imaging · doi:10.3390/jimaging7110245 · Cited by 64
Cetin, I., Stephens, M., Cámara, Ó., & Ballester, M. Á. G. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102158 · Cited by 72
Severn, C., Suresh, K., Görg, C., Choi, Y. S., Jain, R., & Ghosh, D. (2022) · Sensors · doi:10.3390/s22145205 · Cited by 69
“Recent works on explainable deep learning-based imaging models commonly use methods such as integrated gradients and saliency maps to highlight areas of an image which are influential to a given model’s prediction [20–24].”
BackgroundLaBoone, P. A., & Marques, O. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100294 · Cited by 79
Revisiting the Trustworthiness of Saliency Methods in Radiology AI
Zhang, J., Chao, H., Dasegowda, G., Wang, G., Kalra, M. K., & Yan, P. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220221 · Cited by 57
El–Sappagh, S., Alonso, J. M., Abuhmed, T., Ali, F., & Bugarín, A. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10415-5 · Cited by 69
Metadata
- Authors
- Singh, A., Sengupta, S., & Lakshminarayanan, V.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Singh, Amitojdeep
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2020-06-20
- Volume
- 6
- Issue
- 6
- Pages
- 52
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC8321083
- OpenAlex
- W3031116957
- Semantic Scholar
- af6af40155860ac2fd36680e63686473803b5f52
- Cited by
- 718
- References
- 75
- Citations by year
- 2026: 82 · 2025: 135 · 2024: 132 · 2023: 154 · 2022: 130 · 2021: 77 · 2020: 8
- FWCI
- 3.17
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Compute Canada
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable deep learning, medical image analysis, model interpretability, black-box models, clinical deployment, feature importance, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.