Explainable Deep Learning Models in Medical Image Analysis
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Attributions of VGG-16 with images from Imagenet using the methods implemented in [19].](https://figures.cochat.ai/paper-figures/a357c2dc02ed4a63a419e6c762416bfb/explainable-deep-learning-models-in-medical-image-analysis-figure-3.jpg)
![Figure 4 Example of heat maps from a retinal OCT image [40].](https://figures.cochat.ai/paper-figures/002116f86db043c7aaeedcab11546bae/explainable-deep-learning-models-in-medical-image-analysis-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More details about attention mechanisms in neural networks is discussed in [51].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An earlier study for breast mass classification from mammograms [40] using two different CNNs - AlexNet [41] and GoogleNet [42] - employed saliency maps to visualize the image features.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Both approaches are beneficial for medical imaging datasets which tend to be relatively smaller than computer vision benchmarks like ImageNet [33].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“rey patch and hence occluding them from the system’s view. It can provide both positive and negative evidence by highlighting the responsible features. This technique was applied by Zeiler and Fergus [18] to the CNN for the image classification task. Occlusion is the benchmark for any attribution study as it is a simpl…”
MethodWold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606 · cited 1×
“Principal component analysis (PCA) [12], t-Distributed Stochastic Neighbor Embedding (t-SNE) [13] are some common examples of these methods.”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 2×
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
Background
Research works printed but not linked: 2
- El-Baz, A.S. (2020). Recent Advances in Deep Learning Applications for Retinal Diagnosis using OCT. State of the Art in Neural Networks, Elsevier. in press.
- Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
Cited by
Items 301–400 of 772
Cheng, L., Bosch, P. B. J., Hofman, R. F. H., Brakenhoff, T. B., Bruggemans, E. F., van der Geest, R. J., & Holman, E. R. (2022) · Journal of the American Heart Association · doi:10.1161/jaha.121.024168 · Cited by 14
Liver Tumour Segmentation based on ResNet Technique
Sirco, A., Almisreb, A. A., Tahir, N. M., & Bakri, J. (2022) · IEEE International Conference on Control System, Computing and Engineering · doi:10.1109/iccsce54767.2022.9935636 · Cited by 14
“Input from the network is 224 bytes by 224 bytes by 3 [20].”
BackgroundHasenstab, K. A., Huynh, J., Masoudi, S., Cunha, G. M., Pazzani, M. J., & Hsiao, A. (2023) · IEEE Access · doi:10.1109/access.2023.3236575 · Cited by 15
“Several methods have been proposed to interpret network decisions [6], [7], [8], the majority of which being attribution methods, which highlight salient areas on an input image important for CNN prediction.”
BackgroundMethodZhang, J., Chen, Y., Zeng, P., Liu, Y., Diao, Y., & Liu, P. (2023) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2022.12.016 · Cited by 14
Competencies of Teachers in Game-based Pedagogy
due to ethical violation., T. A. I. R. F. T. I. (2023) · Pegem Journal of Education and Instruction · doi:10.47750/pegegog.13.02.39 · Cited by 24
Wang, Y., Rahman, A., Duggar, W. N., Thomas, T. V., Roberts, P. R., Vijayakumar, S., Jiao, Z., Bian, L., & Wang, H. (2023) · Medical Physics · doi:10.1002/mp.16680 · Cited by 14
“In recent years, the studies conducted on explainability aim to identify the features that mostly influence the decision of a model (Singh et al., 2020).”
BackgroundMulticlass Medical X-ray Image Classification using Deep Learning with Explainable AI
Roshan, M. K. G. (2022) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2022.44541 · Cited by 13
Leukocyte deep learning classification assessment using Shapley additive explanations algorithm
Michalski, A., Duraj, K. M., & Kupcewicz, B. (2023) · International Journal of Laboratory Hematology · doi:10.1111/ijlh.14031 · Cited by 13
“Several black box explanatory tools have been developed over the past few years.(13) One of them is SHAP DeepExplainer—an approach to explain any machine learning model.”
BackgroundDeep Learning-Based Diagnosis of Fatal Hypothermia Using Post-Mortem Computed Tomography
Zeng, Y., Zhang, X., Yoshizumi, I., Zhang, Z., Mizuno, T., Sakamoto, S., Kawasumi, Y., Usui, A., Ichiji, K., Bukovský, I., et al. (2023) · The Tohoku Journal of Experimental Medicine · doi:10.1620/tjem.2023.j041 · Cited by 13
Nothing Is Harder to Resist Than the Temptation of AI
Park, A., Kietzmann, J., Killoran, J., Cui, Y. G., van Esch, P., & Dabirian, A. (2023) · IT Professional · doi:10.1109/mitp.2023.3340529 · Cited by 13
Bachanek, S., Würzberg, P., Biggemann, L., Janssen, T. Y., Nietert, M. M., Lotz, J., Zeuschner, P., Maßmann, A., Uhlig, A., & Uhlig, J. (2024) · European Radiology · doi:10.1007/s00330-024-11026-6 · Cited by 13
“This may be partially attributable to the underlying architecture of ENNs, which are generally known to improve model calibration and performance in deep learning tasks without requiring large changes to the model or training procedures [14, 15].”
Automated System for the Detection of Heart Anomalies Using Phonocardiograms: A Systematic Review
Gudigar, A., Raghavendra, U., Maithri, M., Samanth, J., Inamdar, M. A., Vidhya, V., Vicnesh, J., Prabhu, M. A., Tan, R., Yeong, C. H., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3465511 · Cited by 19
“XAI plays a crucial role in enhancing the transparency, interpretability, and trustworthiness of the diagnostic pre-VOLUME dictions made by these complex AI systems [134], [135].”
BackgroundSex differences in brain MRI using deep learning toward fairer healthcare outcomes
Dibaji, M., Ospel, J. M., de Souza, R. M., & Bento, M. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1452457 · Cited by 17
Manz, R., Bäcker, J., Cramer, S., Meyer, P., Müller, D., Muzalyova, A., Rentschler, L., Wengenmayr, C., Hinske, L. C., Huss, R., et al. (2025) · The Journal of Pathology Clinical Research · doi:10.1002/2056-4538.70023 · Cited by 18
“For image-based classi fi cation systems such as those used for digitized histopathological slides (whole slide images), XAI methods that highlight the areas relevant to the decision are particularly suitable to provide additional insights [13].”
Pohlmeyer-Esch, G., Halsey, C., Boisclair, J., Ram, S., Kitz, S., Knight, B., Moulin, P., & Frisk, A.-L. (2025) · Toxicologic Pathology · doi:10.1177/01926233251340622 · Cited by 17
“43,55,58 Saliency maps highlight the contribution of individual pixels to a classification decision.”
Mohagheghi, S., & Foruzan, A. H. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105106 · Cited by 16
Measuring the Left Ventricular Ejection Fraction using Geometric Features
Lagopoulos, A., & Hristu‐Varsakelis, D. (2022) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms55023.2022.00008 · Cited by 13
“Furthermore, regulations such as the European General Data Protection Regulation (GDPR) are making it somewhat harder to justify the use of black-box models in commercial ventures, including healthcare, because re-traceability of the model’s decisions and output is now a requirement [9].”
BackgroundVaidyanathan, A., Guiot, J., Zerka, F., Belmans, F., Peufflik, I. V., Deprez, L., Danthine, D., Canivet, G., Lambin, P., Walsh, S., et al. (2022) · ERJ Open Research · doi:10.1183/23120541.00579-2021 · Cited by 13
Jin, W., Li, X., & Hamarneh, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.06487 · Cited by 14
“Recently, Singh et al. (2020) evaluated 13 XAI algorithms on classifying eye diseases using retina images and asked 14 clinicians to rate the heatmaps regarding their clinical relevance ( plausibility ).”
BackgroundA Systematic Review of Real-Time Deep Learning Methods for Image-Based Cancer Diagnostics
Sriraman, H., Saleena, B., Vats, S., & Balasubramanian, P. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s446745 · Cited by 13
“Explainable DL models 88–91 can ensure doctors that the DL system is performing as expected.”
BackgroundFayyad, M. F., & Mustakim, M. (2024) · International Electronics Symposium (IES) · doi:10.1109/ies63037.2024.10665856 · Cited by 12
“This procedure generates weighted gradient activation maps using the gradient concept as it progresses towards the last convolutional layer [3].”
BackgroundMethodKong, S. H. (2025) · Endocrinology and Metabolism · doi:10.3803/enm.2025.2518 · Cited by 12
“Most current imaging AI models for fracture risk prediction employ supervised learning, where outcomes guide the model to recognize and prioritize predictive features [12].”
Chen, C., Ren, Z.-Y., Zong, H., Xiong, Y.-T., & Yu, H. (2025) · Frontiers in Neurology · doi:10.3389/fneur.2025.1678815 · Cited by 12
“However, the widespread clinical adoption of ML models has been hindered by their “black box” nature, which limits interpretability and transparency in clinical decision-making (11, 12).”
Zhai, L., Zhao, M., Zhang, J., Jamil, M., Naz, R., & Li, C. Y. (2025) · BMC Psychiatry · doi:10.1186/s12888-025-07589-3 · Cited by 18
“Many ML models, especially deep learning models, lack interpretability, which hinders their adoption in real-world settings [8, 14].”
Applications, image analysis, and interpretation of computer vision in medical imaging
Matsuzaka, Y., & Iyoda, M. (2026) · Frontiers in Radiology · doi:10.3389/fradi.2025.1733003 · Cited by 14
Intelligent host engineering for metabolic flux optimisation in biotechnology
Munro, L. J., & Kell, D. B. (2021) · Biochemical Journal · doi:10.1042/bcj20210535 · Cited by 12
“The success of these messages will be judged by the rapidity with which the strategies they contain are adopted.”
BackgroundMeirelles, A. L. S., Kurç, T., Kong, J., Ferreira, R., Saltz, J. H., & Teodoro, G. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.894430 · Cited by 15
“Also, applications developed with deep learning should consider explainability to improve confidence in their use (13, 14).”
BackgroundAn interpretable feature-learned model for overall survival classification of High-Grade Gliomas
Malhotra, R., Saini, B. S., & Gupta, S. (2023) · Physica Medica · doi:10.1016/j.ejmp.2023.102591 · Cited by 13
Analysis of Explainable Artificial Intelligence Methods on Medical Image Classification
Shivhare, I., Jogani, V., Purohit, J., & Shrawne, S. C. (2023) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect57570.2023.10118312 · Cited by 12
Feldkamp, N., & Straßburger, S. (2023) · ACM/IEEE/SCS Workshop on Principles of Advanced and Distributed Simulation (PADS) · doi:10.1145/3573900.3591114 · Cited by 14
Semi-Supervised Detection Model Based on Adaptive Ensemble Learning for Medical Images
Li, J., Shi, H., Chen, W., Liu, N., & Hwang, K. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3282809 · Cited by 15
A Study on Object Detection Using Artificial Intelligence and Image Processing–Based Methods
Nain, V., Shyam, H. S., Kumar, N., Tripathi, P., & Rai, M. (2024) · Wiley eBooks · doi:10.1002/9781394200733.ch6 · Cited by 12
Oh, J.-H., Kim, H., Lee, K. M., Ryu, C., Park, S., Jang, J. H., Choi, H. S., & Kim, E. J. (2021) · Yonsei Medical Journal · doi:10.3349/ymj.2021.62.12.1125 · Cited by 14
Mallio, C. A., Quattrocchi, C. C., Zobel, B. B., & Parizel, P. M. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-20-1306 · Cited by 12
“To improve this process, an increased transparency of deep learning AI algorithms, the so-called explainable AI, is highly advisable not only to obtain systems that are directly interpretable and trustworthy but also to give the end users the opportunity to improve their accuracy (17).”
BackgroundAn investigation into the deep learning approach in sentimental analysis using graph-based theories
Kentour, M., & Lu, J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0260761 · Cited by 11
“The majority of works on graph explainable DL have addressed CAPs problem from specific angles, usually referred to as “model specific” [41]; however, only few attempts have tried to position a DNN as a compositional unit [42] and the best way to assign input values which refers to the historic CAPs.”
BackgroundTowards a neuro-symbolic cycle for human-centered explainability
Mileo, A. (2025) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240740 · Cited by 17
“…Image Analysis, the majority of approaches to explainability are post-model and based on attribution of input features to output (producing partial explanations which are subject to human interpretation) or attention based, sometimes coupled with rule extraction (producing local explanations) [44].”
Luo, L., Wu, Y., Li, S., Li, F., Wang, X., & Wei, X. (2025) · Journal of Nursing Management · doi:10.1155/jonm/5578698 · Cited by 12
Machine learning detection of epileptic seizure onset zone from iEEG
Broti, N. M., Iwasaki, M., & Ono, Y. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00480-w · Cited by 11
“Explainability tools like saliency maps [ 81], Shapley addi‑ tive explanations SHAP [ 82], and attention based visual‑ izations [83] can enhance model transparency, supporting clinical validation.”
Deep learning-based approach to third molar impaction analysis with clinical classifications
Balel, Y., & Sağtaş, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-93783-y · Cited by 11
Comparative evaluation of CAM methods for enhancing explainability in veterinary radiography
Dusza, P., Banzato, T., Burti, S., Bendazzoli, M., Müller, H., & Wodziński, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-14060-6 · Cited by 12
“12, 13, 14.”
Cai, X., Zhang, Z., Zhao, S., Liu, W., & Fan, X. (2025) · Bioengineering · doi:10.3390/bioengineering12101058 · Cited by 11
“Among them, the features that have the greatest impact on the output results are considered the most important [9].”
InfluentialMalpractice in the machine age: Legal and ethical responses to machine learning in medical imaging
Chau, M., Spuur, K. M., White, S. R., Pyper, A., & Crossman, M. H. (2026) · Radiography · doi:10.1016/j.radi.2026.103339 · Cited by 13
Deep convolutional neural networks for regular texture recognition
Liu, N., Rogers, M., Cui, H. L., Liu, W., Li, X., & Delmas, P. J. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.869 · Cited by 11
Tampu, I. E., Eklund, A., Johansson, K. J. M., Gimm, O., & Haj‐Hosseini, N. (2022) · Journal of Biophotonics · doi:10.1002/jbio.202200227 · Cited by 12
Liu, J., Yuan, G., Yang, C., Song, H., & Luo, L. (2022) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2022.023195 · Cited by 15
Chin, L. C., Ling, A. H. W., Chong, Y. F., Mashor, . Y., Alshantti, K., & Aziz, M. E. (2023) · Diagnostics · doi:10.3390/diagnostics13142377 · Cited by 11
“Debelee et al. (2020) [10] surveyed several deep learning-based techniques for identifying and segmenting tumors in breast cancer, cervical cancer, brain tumors, colon cancer, and lung cancer.”
BackgroundHan, K., Yu, Y., & Lü, T. (2024) · Processes · doi:10.3390/pr12010182 · Cited by 14
Koyyada, S. P., & Singh, T. P. (2024) · SN Computer Science · doi:10.1007/s42979-023-02573-8 · Cited by 16
“For example, explainable AI (XAI) [136] techniques, such as Guided Grad CAM, Gradient Back Propagation, and Class activation maps (CAM), would be used to identify how features are selected based on expert knowledge.”
TTDCapsNet: Tri Texton-Dense Capsule Network for complex and medical image recognition
Akoto-Adjepong, V., Appiah, O., Mensah, P. K., & Appiahene, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300133 · Cited by 11
Diagnostic and Prognostic Electrocardiogram-Based Models for Rapid Clinical Applications
Islam, M. S., Kalmady, S. V., Hindle, A., Sandhu, R. K., Sun, W., Sepehrvand, N., Greiner, R., & Kaul, P. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.07.003 · Cited by 21
Advanced Artificial Intelligence Technologies Transforming Contemporary Pharmaceutical Research
Kumar, P., Chaudhary, B., Arya, P., Chauhan, R., Devi, S., Parejiya, P. B., & Gupta, M. M. (2025) · Bioengineering · doi:10.3390/bioengineering12040363 · Cited by 14
““A single disease, numerous targets” refers to the polypharmacology phenomenon [154].”
Gopalan, R., Onniyil, D., Viswanathan, G., & Samdani, G. (2025) · World Journal of Advanced Engineering Technology and Sciences · doi:10.30574/wjaets.2025.15.2.0635 · Cited by 12
Wahid, K. A., He, R., Dede, C., Mohamed, A., Abdelaal, M. A., van Dijk, L. V., Fuller, C. D., & Naser, M. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98253-9_28 · Cited by 10
Evaluating perceptual and semantic interpretability of saliency methods: A case study of melanoma
Bokadia, H., Yang, S. C., Li, Z., Folke, T., & Shafto, P. (2022) · Applied AI Letters · doi:10.1002/ail2.77 · Cited by 15
RETRACTED ARTICLE: Robust adversarial uncertainty quantification for deep learning fine-tuning
Ahmed, U., & Lin, J. C. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05087-5 · Cited by 10
“To address this problem, researchers are exploring various explanatory methods, including saliency and counterfactuality, to make DL networks more transparent and explainable [25, 26].”
BackgroundRetractedClassification of kidney abnormalities using deep learning with explainable AI
Jhumka, K., Khan, M. H.-M., Mungloo-Dilmohamud, Z., & Fedally, S. A. (2023) · International Women in Data Science Conference at Prince Sultan University (WiDS PSU) · doi:10.1109/wids-psu57071.2023.00037 · Cited by 10
Glänzer, L., Masalkhi, H. E., Roeth, A. A., Schmitz‐Rode, T., & Slabu, I. (2023) · Cancers · doi:10.3390/cancers15153773 · Cited by 12
“…for several computer vision tasks and has been used for medical imaging tasks, like the classification of Alzheimer’s, lung cancer detection, retinal disease detection, detection of abnormalities in modalities, like MRI, CT, X-ray, or ultrasound, and semantic segmentation of medical images [1,2].”
MethodTeo, Y. X., Lee, R. E., Nurzaman, S. G., Tan, C. P., & Chan, P. Y. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108957 · Cited by 10
Zhang, H., Zhu, Y., Jin, L., Yang, H., Wang, J., Ablameyko, S., Liu, X., & Xu, Y. (2025) · Small Methods · doi:10.1002/smtd.202401616 · Cited by 10
An Explainable Computer Vision in Histopathology: Techniques for Interpreting Black Box Model
Bhattacharjee, S., Hwang, Y.-B., Ugli, I. K. K., Sumon, R. I., Kim, H., & Choi, H. (2022) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic54071.2022.9722656 · Cited by 12
Quantified Explainability: Convolutional Neural Network Focus Assessment in Arrhythmia Detection
Varandas, R. S. P., Gonçalves, B., Gamboa, H. F. S., & Vieira, P. M. (2022) · BioMedInformatics · doi:10.3390/biomedinformatics2010008 · Cited by 11
XAI Method Properties: A (Meta-)study.Preprint
Schwalbe, G., & Finzel, B. (2021) · arXiv (Cornell University) · Cited by 10
Pyramid Network With Quality-Aware Contrastive Loss for Retinal Image Quality Assessment
Yue, G., Zhang, S., Zhou, T., Jiang, B., Liu, W., & Wang, T. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3501405 · Cited by 13
“On the other hand, interpretability techniques [67], [68] will be incorporated into our RIQA network for better understanding its working mechanism.”
Chong, L., Cao, T., Tang, M., Pu, W., & Lei, P. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112060 · Cited by 9
Exploring the Impact of Explainability in Large Language Model (LLM) Applications on User Experience
Wang, Y., Fang, X., Xu, Z., Li, J., & Wang, L. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719941 · Cited by 9
Network Occlusion Sensitivity Analysis Identifies Regional Contributions to Brain Age Prediction
He, L., Wang, S., Chen, C., Wang, Y., Fan, Q., Chu, C., Fan, L., & Xu, J. (2025) · Human Brain Mapping · doi:10.1002/hbm.70239 · Cited by 10
“…the methods that have been shown to be effective on naturalistic images may not retain good interpretability on actual neuroimages, so their results may not be persuasive enough to be generalizable (Gu et al. 2018; Liang et al. 2021; Shrikumar et al. 2017; Singh et al. 2020; Teng et al. 2022).”
Salmanpour, M. R., Amiri, S., Gharibi, S., Shariftabrizi, A., Xu, Y., Weeks, W. B., Rahmim, A., & Hacihaliloglu, I. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01585-5 · Cited by 14
“Lundberg et al. [73] demonstrated SHAP’s clinical use, and Timmeren [74] underscored the need for a comprehensive RF dictionary to bridge AI and clinical applications.”
Rule Extraction in the Assessment of Brain MRI Lesions in Multiple Sclerosis: Preliminary Findings
Nicolaou, A., Loizou, C. P., Pantzaris, M., Kakas, A. C., & Pattichis, C. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89128-2_27 · Cited by 9
Early Esophageal Malignancy Detection Using Deep Transfer Learning and Explainable AI
Shaw, P., Sankaranarayanan, S., & Lorenz, P. (2022) · International Conference on Communication and Information Systems (ICCIS) · doi:10.1109/iccis56375.2022.9998162 · Cited by 9
“The studies that were carried out in some of the research papers [11-15] discuss the influence of Explainable AI on neural-network-based systems.”
BackgroundBullying Tweets Detection using CNN- Attention
Alhlou, A., & Alam, A. (2023) · International Journal on Cybernetics & Informatics · doi:10.5121/ijci.2023.120106 · Cited by 9
Perera, R. D., Savas, P. S., Senanayake, D., Salgado, R., Joensuu, H., O’Toole, S. A., Li, J., Loi, S., & Halgamuge, S. K. (2024) · Communications Engineering · doi:10.1038/s44172-024-00246-9 · Cited by 9
“The lack of interpretability and explainability of DNNs can lead to mistrust and hinder their adoption in real-world settings 10,32,33 .”
Rapid classification of glaucomatous fundus images
Singh, H., Saini, S. S., & Lakshminarayanan, V. (2021) · Journal of the Optical Society of America A · doi:10.1364/josaa.415395 · Cited by 9
“The reader is referred to Singh et al., (2021) [23] for a review of the new field of explainable AI.”
BackgroundSelf-citationLim, M. J. R. (2021) · Neurosurgery · doi:10.1093/neuros/nyab337 · Cited by 9
QUANTIFYING EXPLAINABLE AI METHODS IN MEDICAL DIAGNOSIS: A STUDY IN SKIN CANCERPreprint
Sangwan, H. (2024) · medRxiv · doi:10.1101/2024.12.08.24318158 · Cited by 10
UbiQTree: Uncertainty quantification in XAI with tree ensembles
Dubey, A., Anžel, A., İlgen, B., & Hattab, G. (2026) · Patterns · doi:10.1016/j.patter.2025.101454 · Cited by 12
“53 Recent frameworks continue to adopt SHAP as a technique for improving explainability in healthcare.”
Abraham, L. A., Palanisamy, G., Goutham, V., & Nisha, J. S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11410-8 · Cited by 8
“AI can help interpret PET data to improve tumor detection (Singh et al. 2020 ).”
Chandola, Y., Virmani, J., Bhadauria, H. S., & Kumar, P. (2021) · Elsevier eBooks · doi:10.1016/b978-0-323-90184-0.00001-1 · Cited by 8
Artificial Intelligence for Global Healthcare
Parsa, A. D., Hakkim, S., Vinnakota, D., Mahmud, I., Bulsari, S., Dehghani, L., Pulikkottil, A. T., Sivasubramanian, M., & Kabir, R. (2023) · Medical virology · doi:10.1007/978-981-99-0369-6_1 · Cited by 11
Luo, Y., Wu, G., Liu, Y., Liu, W., & Han, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3436714 · Cited by 8
“I NTRODUCTION R ECENT advances in medical image analysis have witnessed the roaring success of the extensive deployment of deep learning techniques in this research field [1], [2], [3], [4], [5].”
Hasenstab, K. A., Hahn, L. D., Chao, N., & Hsiao, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-75886-0 · Cited by 8
“Outside of generative and commonly used gradient-or perturbation-based attribution methods, there are several alternative approaches for explainable AI in medical imaging 35–37 , including attention networks 38 and feature analytic methods 37,39,40 .”
Topo-CNN: Retinal Image Analysis with Topological Deep Learning
Ahmed, F., Bhuiyan, M. A. N., & Coşkunuzer, B. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01575-7 · Cited by 24
Lee, S., Lee, E., Park, S., Park, M. S., Jung, J., Min, G. J., Park, S., Lee, S., Cho, B., Eom, K., et al. (2022) · Bone Marrow Transplantation · doi:10.1038/s41409-022-01583-z · Cited by 10
Brini, I., Mehri, M., Ingold, R., & Amara, N. E. B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16014-1_10 · Cited by 8
DeepSHAP Summary for Adversarial Example Detection
Lin, Y.-C., & Yu, F. (2023) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest59248.2023.00010 · Cited by 9
“It has been applied in various domains, including medical [28]–[30], engineering [31], [32], aerospace [33], transportation [34], manufacturing [35], [36], and security [37], [38].”
BackgroundAL-XAI-MERS: Unveiling Alzheimer's Mysteries with Explainable AI
Deshmukh, A., Kallivalappil, N., D'souza, K., & Kadam, C. (2024) · International Conference on Emerging Trends in Information Technology and Engineering (ICETITE) · doi:10.1109/ic-etite58242.2024.10493489 · Cited by 8
“The study underscored the importance of model-agnostic XAI techniques in enhancing the interpretability of complex AI models, particularly in critical areas of medical science characterised by the black-box nature of deep learning [10].”
BackgroundMedical Image Classification for Disease Prediction with the Aid of Deep Learning Approaches
Pandiyarajan, M., Raja, J. T., Ramasamy, J., Tiwari, M., Shinde, S., & Chakravarthi, M. K. (2022) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite53722.2022.9823417 · Cited by 8
A systematic review of automated temporal bone segmentation methods
Li, W., Wijewickrema, S. N. R., Margeta, J., Kamraoui, R. A., Hussain, R., & Gerard, J.-M. (2025) · Biomedical Engineering Advances · doi:10.1016/j.bea.2025.100195 · Cited by 7
Guided relevance attention mapping: Explainable artificial intelligence reimagined
Yadav, A., Srivastava, V. P., & Yadav, A. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112925 · Cited by 8
Towards precision medicine for otology and neurotology: Machine learning applications and challenges
Adcock, K. S., Arulchelvan, E., Shields, N., & Vanneste, S. (2025) · Hearing Research · doi:10.1016/j.heares.2025.109473 · Cited by 7
Din, S. U., Kemna, R., Ket, J. C. F., Iqbal, M., Bohoudi, O., Hoogendoorn, M., Beretta, E., & Lisowska, A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1819422 · Cited by 8
“Others (Champendal et al., 2023; Bhati et al., 2024; Patrício et al., 2023; Ennab and Mcheick, 2024; Van der Velden et al., 2022; Yang et al., 2022; Singh et al., 2020a; Chaddad et al., 2023; Salahuddin et al., 2022; Nazir et al., 2023; Band et al., 2023; Borys et al., 2023; Bharati et al., 2023a)…”
MIDRC CRP10 AI interface—an integrated tool for exploring, testing and visualization of AI models
Gorre, N., Carranza, E., Fuhrman, J. D., Li, H., Madduri, R., Giger, M. L., & Naqa, I. E. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acb754 · Cited by 8
The effect of head motion on brain age prediction using deep convolutional neural networks
Vakli, P., Weiss, B., Rozmann, D., Eross, G., Nárai, Á., Hermann, P., & Vidnyánszky, Z. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120646 · Cited by 8
Li, Y., Xiong, X., Liu, X., Wu, Y., Li, X., Liu, B., Lin, B., Yu, L., & Xu, B. (2024) · PeerJ · doi:10.7717/peerj.18098 · Cited by 9
“However, these features frequently exhibit inaccuracies, inconsistency, and a lack of transparency, making it dif fi cult to explain the model predictions ( Singh, Sengupta & Lakshminarayanan, 2020 ).”
BackgroundVieira, C., da Rocha, L. C. D., Guimarães, M. D. P., & Dias, D. R. C. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109556 · Cited by 7
Alrabie, S., & Barnawi, A. (2025) · Bioengineering · doi:10.3390/bioengineering12060558 · Cited by 7
“This is achieved by leveraging the gradients of the target class flowing into the last convolutional layer and weighting the corresponding feature maps to identify and visualize areas of high importance [45].”
Ensembling to Leverage the Interpretability of Medical Image Analysis Systems
Zafeiriou, A., Kallipolitis, A., & Maglogiannis, I. G. (2023) · IEEE Access · doi:10.1109/access.2023.3291610 · Cited by 8
“Interpetability is a critical aspect of medical image analysis and related applications [13], [14] and highly sought after in recent works.”
BackgroundMohan, A., Asghar, Z., Abid, R., Subedi, R., Kumari, K., Kumar, S., Majumder, K., Bhurgri, A. I., Tejwaney, U., & Kumar, S. (2023) · Annals of Medicine and Surgery · doi:10.1097/ms9.0000000000001175 · Cited by 11
“In theory, any decision-making process in the clinical setting should be driven by combinations of appropriate data features in the context of GI endoscopy([10]).”
BackgroundKlein, N., Carr, A., Hampel-Arias, Z., Zastrow, A., Ziemann, A., & FLYNN, E. B. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10283350 · Cited by 7
KİRAZ, A. H., Djibrillah, F. O., & Yüksel, M. E. (2023) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.4037 · Cited by 8
Mohamadi, H. E., & Hassouni, M. E. (2023) · International Conference on Wireless Networks and Mobile Communications (WINCOM) · doi:10.1109/wincom59760.2023.10322952 · Cited by 7
Metadata
- Authors
- Singh, A., Sengupta, S., & Lakshminarayanan, V.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Singh, Amitojdeep
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2020-06-20
- Volume
- 6
- Issue
- 6
- Pages
- 52
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC8321083
- OpenAlex
- W3031116957
- Semantic Scholar
- af6af40155860ac2fd36680e63686473803b5f52
- Cited by
- 718
- References
- 75
- Citations by year
- 2026: 82 · 2025: 135 · 2024: 132 · 2023: 154 · 2022: 130 · 2021: 77 · 2020: 8
- FWCI
- 3.17
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Compute Canada
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable deep learning, medical image analysis, model interpretability, black-box models, clinical deployment, feature importance, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.