Explainable Deep Learning Models in Medical Image Analysis
Open accessThe DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Attributions of VGG-16 with images from Imagenet using the methods implemented in [19].](https://figures.cochat.ai/paper-figures/a357c2dc02ed4a63a419e6c762416bfb/explainable-deep-learning-models-in-medical-image-analysis-figure-3.jpg)
![Figure 4 Example of heat maps from a retinal OCT image [40].](https://figures.cochat.ai/paper-figures/002116f86db043c7aaeedcab11546bae/explainable-deep-learning-models-in-medical-image-analysis-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More details about attention mechanisms in neural networks is discussed in [51].”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“An earlier study for breast mass classification from mammograms [40] using two different CNNs - AlexNet [41] and GoogleNet [42] - employed saliency maps to visualize the image features.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Both approaches are beneficial for medical imaging datasets which tend to be relatively smaller than computer vision benchmarks like ImageNet [33].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2008) · Journal of Machine Learning Research · Cited by 33,051
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 2×
“rey patch and hence occluding them from the system’s view. It can provide both positive and negative evidence by highlighting the responsible features. This technique was applied by Zeiler and Fergus [18] to the CNN for the image classification task. Occlusion is the benchmark for any attribution study as it is a simpl…”
MethodWold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606 · cited 1×
“Principal component analysis (PCA) [12], t-Distributed Stochastic Neighbor Embedding (t-SNE) [13] are some common examples of these methods.”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 2×
“Different kinds of taxonomies and classification methods are discussed briefly here and a detailed analysis of the taxonomies can be found in [8,9] and a flow chart for them is shown in 1.”
Background
Research works printed but not linked: 2
- El-Baz, A.S. (2020). Recent Advances in Deep Learning Applications for Retinal Diagnosis using OCT. State of the Art in Neural Networks, Elsevier. in press.
- Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
Cited by
Items 101–200 of 772
Self-eXplainable AI for medical image analysis: A survey and new outlooks
Hou, J., Liu, S., Bie, Y., Wang, H., Tan, A., Luo, L., & Chen, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104791 · Cited by 66
Diagnosis and detection of bone fracture in radiographic images using deep learning approaches
Aldhyani, T. H. H., Ahmed, Z. A. T., Alsharbi, B. M., Ahmad, S., Al-Adhaileh, M. H., Kamal, A. H., Almaiah, M. A., & Nazeer, J. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2024.1506686 · Cited by 67
Artificial Intelligence in Cardiac MRI: Is Clinical Adoption Forthcoming?
Fotaki, A., Anton, E. P., Chiribiri, A., Botnar, R. M., Pushparajah, K., & Prieto, C. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.818765 · Cited by 58
“While AI can extract novel insights from existing data, it is often difficult to justify why the network reached a certain output; the so-called “black-box problem” (38).”
BackgroundHaj‐Hosseini, N., Lindblad, J., Hasséus, B., Kumar, V. V., Subramaniam, N., & Hirsch, J. (2022) · Journal of Maxillofacial and Oral Surgery · doi:10.1007/s12663-022-01710-9 · Cited by 66
“XAI directly in the interface [62], it may be possible to offer an interactive tool for the best possible diagnostics at a minimal cost [63].”
MethodBordekar, H., Cersullo, N., Brysch, M., Philipp, J., & Hühne, C. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02272-4 · Cited by 58
“Firstly, CNNs often demand substantial amounts of data for effective training (Singh et al., 2020).”
Sherwani, M. K., & Gopalakrishnan, S. (2024) · Frontiers in Radiology · doi:10.3389/fradi.2024.1385742 · Cited by 61
Wang, Z., Li, Y., Zhai, J., Yang, S., Sun, B., & Liang, P. (2024) · Talanta · doi:10.1016/j.talanta.2024.126138 · Cited by 55
Integrated explainable deep learning prediction of harmful algal blooms
Lee, D.-H., Kim, M., Lee, B., Chae, S., Kwon, S., & Kang, S. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122046 · Cited by 60
Al‐Kababji, A., Bensaali, F., Dakua, S. P., & Himeur, Y. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105532 · Cited by 61
“To that end, an increasing interest has been shown recently in developing explainable and interpretable liver tumor delineation frameworks [185].”
BackgroundHasan, M. M., Hossain, M. M., Rahman, M. M., Azad, A., Alyami, S. A., & Moni, M. A. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107407 · Cited by 56
Salehmin, M. N. I., Kiong, T. S., Mohamed, H., Umar, D. A., Yu, K. L., Ong, H. C., Nomanbhay, S. M., & Lim, S. S. (2024) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2024.07.045 · Cited by 58
U-Net Based Chest X-ray Segmentation with Ensemble Classification for Covid-19 and Pneumonia
Kumarasinghe, H., Kolonne, S., Fernando, C., & Meedeniya, D. (2022) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v18i07.30807 · Cited by 54
Skin cancer classification using explainable artificial intelligence on pre-extracted image features
Khater, T., Ansari, S., Mahmoud, S. A., Hussain, A. J., & Tawfik, H. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200275 · Cited by 76
Explainable deep neural networks for novel viral genome prediction
Dasari, C. M., & Bhukya, R. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02572-3 · Cited by 58
Toledo-Cortés, S., Useche, D. H., Müller, H., & Gonzalez, F. A. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105472 · Cited by 56
Kalsi, D., Thomas, L., Hyde, C., Dan, M., Greenhalgh, J., & Ko, J. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1264846 · Cited by 57
Unraveling the Black Box: A Review of Explainable Deep Learning Healthcare Techniques
Murad, N. Y., Hasan, M. H., Azam, M. H., Yousuf, N., & Yalli, J. S. (2024) · IEEE Access · doi:10.1109/access.2024.3398203 · Cited by 61
“The integration of modern technology, notably deep learning, holds the promise of altering diagnoses and treatment procedures in the fast-expanding healthcare scene [16].”
BackgroundOh, K., Yoon, J. S., & Suk, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3197845 · Cited by 66
“a model’s internal workings [14], [15], [16].”
Counterfactual Graphs for Explainable Classification of Brain Networks
Abrate, C., & Bonchi, F. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467154 · Cited by 74
“On the other hand, although providing post-hoc explanations of a black-box classifier is a very active research area [13, 14], most of the proposed methods focus on tabular data [22, 26], images [31, 33, 34] or time series [3].”
BackgroundExplainable Machine Learning for Lung Cancer Screening Models
Kobylińska, K., Orłowski, T., Adamek, M., & Biecek, P. (2022) · Applied Sciences · doi:10.3390/app12041926 · Cited by 45
Interpretable Classification of Pneumonia Infection Using eXplainable AI (XAI-ICP)
Sheu, R.-K., Pardeshi, M. S., Pai, K.-C., Chen, L.-C., Wu, C.-L., & Chen, W.-C. (2023) · IEEE Access · doi:10.1109/access.2023.3255403 · Cited by 43
“Recent advances in the computer based medical diagnosis (CAD) has improved the quality control process within the hospitals [1], [2], [3], [4].”
BackgroundXML-GBM lung: An explainable machine learning-based application for the diagnosis of lung cancer
Rikta, S. T., Uddin, K. M. M., Biswas, N., Mostafiz, R., Sharmin, F., & Dey, S. K. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100307 · Cited by 56
Hepp, T., Blum, D., Armanious, K., Schölkopf, B., Štern, D., Yang, B., & Gatidis, S. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.101967 · Cited by 55
A survey on privacy for B5G/6G: New privacy challenges, and research directions
Sandeepa, C., Siniarski, B., Kourtellis, N., Wang, S., & Liyanage, M. (2022) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2022.100405 · Cited by 61
“One such is the pre-model, in-model and post-model explanations [135].”
BackgroundNhlapho, W., Atemkeng, M., Brima, Y., & Ndogmo, J. C. (2024) · Information · doi:10.3390/info15040182 · Cited by 55
Termine, A., Fabrizio, C., Strafella, C., Caputo, V., Petrosini, L., Caltagirone, C., Giardina, E., & Cascella, R. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11040280 · Cited by 50
The Robustness of Counterfactual Explanations Over Time
Ferrario, A., & Loi, M. (2022) · IEEE Access · doi:10.1109/access.2022.3196917 · Cited by 64
CXAI: Explaining Convolutional Neural Networks for Medical Imaging Diagnostic
Rguibi, Z., Hajami, A., Zitouni, D., Elqaraoui, A., & Bedraoui, A. (2022) · Electronics · doi:10.3390/electronics11111775 · Cited by 57
“In our implementation, we will use the post hoc explanation to explain CNN models due to these difficulties [32].”
MethodDeep Learning for Orthopedic Disease Based on Medical Image Analysis: Present and Future
Lee, J. H., & Chung, S. W. (2022) · Applied Sciences · doi:10.3390/app12020681 · Cited by 48
“In a similar context, there are attempts to improve the explainability by improving the existing CNN architecture [11].”
BackgroundVeritti, D., Rubinato, L., Sarao, V., Nardin, A. D., Foresti, G. L., & Lanzetta, P. (2023) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-023-06245-4 · Cited by 52
“One such strategy is to use explainable AI algorithms, which are designed to provide insight into the decision-making process of AI-based algorithms [5, 47].”
MethodRoy, S. S., Pal, D., & Meena, T. (2023) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-023-00437-y · Cited by 47
“Recent papers use attribution approaches for explaining DL network in precision diagnostics (Singh et al. 2020).”
Umapathy, S., Murugappan, M., Deepa, B., & Thakur, M. (2023) · Diagnostics · doi:10.3390/diagnostics13182987 · Cited by 45
“Several researchers have used explainable Artificial Intelligence (AI) and deep learning techniques in their medical imaging applications [6–9].”
BackgroundPapandrianos, N. I., Feleki, A., Moustakidis, S., Papageorgiou, E. I., Apostolopoulos, I. D., & Apostolopoulos, D. J. (2022) · Applied Sciences · doi:10.3390/app12157592 · Cited by 45
“Regardless of their remarkable advantages, CNNs are structured by default as black boxes [16] since they do not offer transparency in their internal functionalities.”
González-Alday, R., García-Cuesta, E., Maojo, & Kulikowski, C. (2023) · Applied Sciences · doi:10.3390/app131910778 · Cited by 54
Mir, A. N., Rizvi, D. R., & Ahmad, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110519 · Cited by 49
The effect of machine learning explanations on user trust for automated diagnosis of COVID-19
Goel, K., Sindhgatta, R., Kalra, S., Goel, R., & Mutreja, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105587 · Cited by 54
“With increased used of machine learning for medical diagnosis, there has been an increased focus on using ML explanations for understanding pre-dictions [14].”
BackgroundThe promise of explainable deep learning for omics data analysis: Adding new discovery tools to AI
Santorsola, M., & Lescai, F. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.06.002 · Cited by 49
Muñoz-Saavedra, L., Escobar-Linero, E., Civit-Masot, J., Luna-Perejón, F., Civit, A., & Domínguez-Morales, M. (2023) · Sensors · doi:10.3390/s23167134 · Cited by 47
“These technologies provide information about the objective classification criteria used in the automatic system [20,21].”
MethodMR Images, Brain Lesions, and Deep Learning
Castillo, D., Lakshminarayanan, V., & Rodríguez-Álvarez, M. J. (2021) · Applied Sciences · doi:10.3390/app11041675 · Cited by 52
“This is being alleviated by the development of explainable AI [84].”
BackgroundSelf-citationHan, B., Zhao, N., Zeng, C., Mu, Z., & Gong, X. (2022) · Scientific Reports · doi:10.1038/s41598-022-24404-1 · Cited by 43
Deep learning-enhanced nuclear medicine SPECT imaging applied to cardiac studies
Apostolopoulos, I. D., Papandrianos, N. I., Feleki, A., Moustakidis, S., & Papageorgiou, E. I. (2023) · EJNMMI Physics · doi:10.1186/s40658-022-00522-7 · Cited by 41
“Deep Learning methods face the explainability issue [76].”
MethodAgibetov, A., Kammerlander, A. A., Duca, F., Nitsche, C., Koschutnik, M., Donà, C., Dachs, T.-M., Rettl, R., Stria, A., Schrutka, L., et al. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11121268 · Cited by 40
“Abbreviations and Acronyms CA cardiac amyloidosis CMR cardiac magnetic resonance HFpEF heart failure with preserved ejection fraction HF heart failure AL amyloid light chains LV left ventricle HF heart failure CNN convolutional neural network ECG electrocardiogram EMB endomyocardial biopsy LGE late gadolinium enhancem…”
BackgroundGündoğan, E. (2025) · Applied Sciences · doi:10.3390/app15105412 · Cited by 40
Acevedo, A., Merino, A., Boldú, L., Molina, Á., Alférez, S., & Rodellar, J. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104479 · Cited by 47
The Usefulness of Gradient-Weighted CAM in Assisting Medical Diagnoses
Chien, J.-C., Lee, J.-D., Hu, C.-S., & Wu, C. (2022) · Applied Sciences · doi:10.3390/app12157748 · Cited by 39
Deep Learning for Brain MRI Confirms Patterned Pathological Progression in Alzheimer's Disease
Pan, D., Zeng, A., Yang, B., Lai, G., Hu, B., Song, X., Jiang, T., & Initiative, A. D. N. (2022) · Advanced Science · doi:10.1002/advs.202204717 · Cited by 42
Medical slice transformer for improved diagnosis and explainability on 3D medical images with DINOv2
Müller‐Franzes, G. A., Khader, F., Siepmann, R. M., Han, T., Kather, J. N., Nebelung, S., & Truhn, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-09041-8 · Cited by 57
“This improvement is important since improved saliency maps aid in understanding the model’s decision-making process and increase the trustworthiness of deep learning models in clinical settings 5–7 .”
Zhang, K., Wang, D., Lin, F., Xie, J., & Zhou, W. (2026) · iScience · doi:10.1016/j.isci.2026.115026 · Cited by 48
“Random Forest 10 GBM 10 LGBM 10 XGBoost 10 AdaBoost 10 EBM 50 GAM 10 Saliency Maps Saliency Maps 28,31,115 They are simple and fast to compute, only requiring a single backward pass to visually highlight important input pixels for the prediction.”
InfluentialA graph-based interpretability method for deep neural networks
Wang, T., Zheng, X., Zhang, L., Cui, Z., & Xu, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126651 · Cited by 44
Jiang, J., Peng, H.-H., Yang, Z., Ma, X., Sahakijpijarn, S., Moon, C., Ouyang, D., & Williams, R. O. (2022) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2022.122179 · Cited by 41
Prediction of bone mineral density in CT using deep learning with explainability
Kang, J.-W., Park, C.-Y., Lee, D.-E., Yoo, J.-H., & Kim, M. (2023) · Frontiers in Physiology · doi:10.3389/fphys.2022.1061911 · Cited by 30
“XAI is currently used in medical fi elds to clarify DL features to assist clinicians in understanding and trusting decisions (Amann et al., 2020; Singh et al., 2020).”
MethodLee, M. R. (2023) · Processes · doi:10.3390/pr11113171 · Cited by 47
“Moreover, deep learning is computationally Beyond robotics, deep learning finds applications in diverse domains including image analysis [156–158], video analysis [159–161], natural language generation [162,163], speech recognition [164–166], biometrics [167,168], text analytics [169,170], and…”
BackgroundRadhakrishnan, M., Sampathila, N., Muralikrishna, H., & Swathi, K. S. (2024) · IEEE Access · doi:10.1109/access.2024.3448219 · Cited by 46
“XAI techniques, such as integrated gradient, saliency map, grad-cam, and DeepLift are employed [52].”
MethodGabor capsule network with preprocessing blocks for the recognition of complex images
Ayidzoe, M. A., Yu, Y., Mensah, P. K., Cai, J., Adu, K., & Tang, Y. (2021) · Machine Vision and Applications · doi:10.1007/s00138-021-01221-6 · Cited by 36
Applications of interpretability in deep learning models for ophthalmology
Hanif, A. M., Beqiri, S., Keane, P. A., & Campbell, J. P. (2021) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000000780 · Cited by 40
A Deep Learning Approach for Detecting Otitis Media From Wideband Tympanometry Measurements
Sundgaard, J. V., Bray, P., Laugesen, S., Harte, J., Kamide, Y., Tanaka, C., Christensen, A. N., & Paulsen, R. R. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3159263 · Cited by 38
An interpretable convolutional neural network for nuclear power plant abnormal events
Shin, J. H., Bae, J., Kim, J., & Lee, S. J. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109792 · Cited by 40
Yengec-Tasdemir, S. B., Aydın, Z., Akay, E., Doğan, S., & Yılmaz, B. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107441 · Cited by 37
Toward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 50
Valizadeh, G., Elahi, R., Hasankhani, Z., Rad, H. S., & Shalbaf, A. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10176-6 · Cited by 43
Liu, Y., Hsu, H. Y., Lin, T., Peng, B., Saqi, A., Salvatore, M. M., & Jambawalikar, S. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-023-04798-w · Cited by 39
“This could help researchers to better understand how a model reaches decisions, and identify features associated with it [18, 19].”
BackgroundAffective Design Analysis of Explainable Artificial Intelligence (XAI): A User-Centric Perspective
Bernardo, E. L., & Seva, R. R. (2023) · Informatics · doi:10.3390/informatics10010032 · Cited by 48
“It centers on the AI’s purpose, process, and performance to grant cognitive resources [16,17] to help developers [18] (e.g., troubleshooting, understanding limitations, and improving performance) and to answer critical questions raised by an end-user [19].”
Wang, R., Kuo, P., Chen, L.-C., Seastedt, K. P., Gichoya, J. W., & Celi, L. A. (2024) · EBioMedicine · doi:10.1016/j.ebiom.2024.105047 · Cited by 39
“65 However, interpreting the model ’ s operations presents a signi fi cant challenge.”
BackgroundMir, A. N., & Rizvi, D. R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111413 · Cited by 43
A lightweight xAI approach to cervical cancer classification
Civit-Masot, J., Luna-Perejón, F., Muñoz-Saavedra, L., Domínguez-Morales, M., & Civit, A. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03063-6 · Cited by 37
“These technologies, through various and varied subsequent analyses, provide information on the objective classification criteria used in the automatic system [27, 28].”
BackgroundGligorić, Z., Görçün, Ö. F., Gligorić, M., Pamucar, D. S., Šimić, V., & Küçükönder, H. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102079 · Cited by 31
“In medical di-agnostics, clarity, transparency, and comprehensibility of data derived from imaging technologies are imperative for accurate diagnoses (Singh et al., 2020).”
Using machine learning to enhance and accelerate synthetic biology
Rai, K., Wang, Y., O’Connell, R. W., Patel, A. B., & Bashor, C. J. (2024) · Current Opinion in Biomedical Engineering · doi:10.1016/j.cobme.2024.100553 · Cited by 47
State-of-the-Art in Responsible, Explainable, and Fair AI for Medical Image Analysis
Amirian, S., Gao, F., Littlefield, N. G., Hill, J. H., Yates, A. J., Plate, J. F., Pantanowitz, L., Rashidi, H. H., & Tafti, A. P. (2025) · IEEE Access · doi:10.1109/access.2025.3555543 · Cited by 37
“By explaining how the AI arrived at each conclusion, REF-AI also reduces the risk of errors or misdiagnosis, while it can also make these AI-enabled models transparent and free from biases [14], [19], [28], [140], [141], [142], [143], [144], [145].”
XctNet: Reconstruction network of volumetric images from a single X-ray image
Tan, Z., Li, J., Tao, H., Li, S., & Hu, Y. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102067 · Cited by 33
Role of radiomics in predicting immunotherapy response
Kothari, G. (2022) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13426 · Cited by 39
“Challenges of a deep learning approach, however, include the lack of interpretability and transparency of deep learning models, raising ethical and trust issues and limiting their clinical use, although solutions to partly overcome these issues are being investigated.(64) Conclusions The incorporation of immunotherapy…”
BackgroundDoubly Right Object Recognition: A Why Prompt for Visual Rationales
Mao, C., Teotia, R., Sundar, A., Menon, S., Yang, J., Wang, X., & Vondrick, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00267 · Cited by 50
“Learning models that can explain their own decision is important for building trustworthy systems, especially in applications that require humanmachine interactions [2, 15, 37, 50].”
BackgroundApplications of interpretable deep learning in neuroimaging: A comprehensive review
Munroe, L., da Silva, M., Heidari, F., Grigorescu, I., Dahan, S., Robinson, E. C., Deprez, M., & So, P. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00214 · Cited by 41
“…in the context of deep learning has become an important area of research as it can be easily incorporated into existing neural network architectures while also improving performance ( Brauwers & Frasincar, 2021; Niu et al., 2021) and providing explanations ( Samek et al., 2017; Singh et al., 2020).”
BackgroundiCVM: An Interpretable Deep Learning Model for CVM Assessment Under Label Uncertainty
Liao, N., Dai, J., Tang, Y., Zhong, Q., & Mo, S. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3179619 · Cited by 37
“Some of them [32]–[34] have investigated the interpretability of decisions made by the model.”
BackgroundGouabou, A. C. F., Collenne, J., Monnier, J., Iguernaissi, R., Damoiseaux, J.-L., Moudafi, A., & Merad, D. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232213838 · Cited by 40
“Furthermore, providing clinical end users with sufficient evidence to support the prediction builds confidence in the model’s decision and helps dermatologists identify potentially questionable decisions [41].”
BackgroundFarrag, A., Gad, G., Fadlullah, Z. M., Fouda, M. M., & Alsabaan, M. (2023) · IEEE Access · doi:10.1109/access.2023.3330465 · Cited by 33
“A number of these revamped models have found their place in disease categorization networks, offering justification for predictions made by such opaque DL models [19].”
BackgroundMethodChen, C., Guo, J., Zhang, L., Wu, X., & Yang, Z. Q. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114159 · Cited by 31
Kang, S., Shon, B., Park, E. Y., Jeong, S., & Kim, E. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310004 · Cited by 36
A 2.5D convolutional neural network for HPV prediction in advanced oropharyngeal cancer
Saint‐Esteven, A. L. G., Bogowicz, M., Konukoğlu, E., Riesterer, O., Balermpas, P., Gückenberger, M., Tanadini‐Lang, S., & van Timmeren, J. E. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105215 · Cited by 31
Real-world data mining meets clinical practice: Research challenges and perspective
Mandreoli, F., Ferrari, D., Guidetti, V., Motta, F., & Missier, P. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.1021621 · Cited by 33
Kim, S. Y. (2023) · Bioengineering · doi:10.3390/bioengineering10060701 · Cited by 37
Rotem, O., Schwartz, T., Maor, R., Tauber, Y., Shapiro, M. T., Meseguer, M., Gilboa, D., Seidman, D. S., & Zaritsky, A. (2024) · Nature Communications · doi:10.1038/s41467-024-51136-9 · Cited by 37
Rheumatoid Arthritis Diagnosis: Deep Learning vs. Humane
Avramidis, G., Avramidou, M. P., & Papakostas, G. A. (2021) · Applied Sciences · doi:10.3390/app12010010 · Cited by 33
DermX: An end-to-end framework for explainable automated dermatological diagnosis
Jalaboi, R., Faye, F., Orbes‐Arteaga, M., Jørgensen, D. R., Winther, O., & Galimzianova, A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102647 · Cited by 34
Explainable Image Similarity: Integrating Siamese Networks and Grad-CAM
Livieris, I. E., Pintelas, E. G., Kiriakidou, N., & Pintelas, P. (2023) · Journal of Imaging · doi:10.3390/jimaging9100224 · Cited by 34
“In computer vision, the concept of image similarity consists of a fundamental building block for various real-world applications, ranging from image retrieval [26] and pattern recognition [25] to anomaly detection [27].”
The Future of Orthodontics: Deep Learning Technologies
Surendran, A., Daigavane, P., Shrivastav, S., Kamble, R. H., Sanchla, A. D., Bharti, L., & Shinde, M. (2024) · Cureus · doi:10.7759/cureus.62045 · Cited by 42
“Approaches like gradient-weighted class activation mapping (Grad-CAM) and DeConvNet can uncover the underlying features influencing decision-making, which could improve the interpretability of AI models in orthodontics [82].”
MethodZhang, Y., Gao, Z., He, K., Li, C., & Mao, R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103027 · Cited by 39
Štifanić, J., Štifanić, D., Anđelić, N., & Car, Z. (2025) · Biology · doi:10.3390/biology14080909 · Cited by 35
“However, many DL models used for OC diagnosis are thought of as “black boxes”, which makes it challenging to comprehend their mechanisms to make decisions, especially in crucial applications like cancer diagnosis [14].”
An ISHAP-based interpretation-model-guided classification method for malignant pulmonary nodule
He, W., Li, B., Liao, R.-Q., Mo, H., & Tian, L. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107778 · Cited by 30
Towards a Connected Mobile Cataract Screening System: A Future Approach
Zaki, W. M. D. W., Mutalib, H. A., Ramlan, L. A., Hussain, A., & Mustapha, A. (2022) · Journal of Imaging · doi:10.3390/jimaging8020041 · Cited by 34
“Other than being the most widely used artificial intelligence (AI) method for a variety of tasks including medical imaging, DL has also been shown to be successful at detecting clinically significant features for the diagnosis and prognostic prediction of ocular diseases [12,13].”
MethodLISA : Enhance the explainability of medical images unifying current XAI techniques
Abeyagunasekera, S. H. P., Perera, Y., Chamara, K., Kaushalya, U., Haddela, P. S., & Senaweera, O. (2022) · IEEE International Conference for Convergence in Technology (I2CT) · doi:10.1109/i2ct54291.2022.9824840 · Cited by 36
Explainable AI in Orthopedics: Challenges, Opportunities, and Prospects
Amirian, S., Carlson, L., Gong, M. F., Lohse, I., Weiss, K. R., Plate, J. F., & Tafti, A. P. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00230 · Cited by 34
“This can facilitate more informed and shared decision-making about patient care by assisting healthcare providers, patients, and physicians in better understanding of the rationale behind AI models [16]–[18].”
BackgroundWatanabe, A., Ketabi, S., Namdar, K., & Khalvati, F. (2022) · Frontiers in Radiology · doi:10.3389/fradi.2022.991683 · Cited by 31
“Hence, the explainability of Artificial Intelligence (AI) models that shows the part of the model’s inputs or the kind of information the models focused on when making predictions is crucial for the radiology field to gain and increase radiologists’, patients’, and regulators’ trust in the use of the models for diagno…”
BackgroundMethodInfluentialExplainable screening of oral cancer via deep learning and case-based reasoning
Cimino, M. G. C. A., Campisi, G., Galatolo, F. A., Neri, P., Tozzo, P., Parola, M., Mantia, G. L., & Fede, O. D. (2025) · Smart Health · doi:10.1016/j.smhl.2024.100538 · Cited by 35
Precision medical image hash retrieval by interpretability and feature fusion
Guan, A., Liu, L., Fu, X., & Liu, L. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106945 · Cited by 31
Tempelaere, A., Ketelaere, B. D., He, J., Kalfas, I., Pieters, M., Saeys, W., Belleghem, R. V., Doorselaer, L. V., Verboven, P., & Nicolaï, B. M. (2023) · Postharvest Biology and Technology · doi:10.1016/j.postharvbio.2023.112576 · Cited by 28
MDF-Net for abnormality detection by fusing X-rays with clinical data
Hsieh, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Nascimento, J. C., Jorge, J. A., & Moreira, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-41463-0 · Cited by 33
Till, T., Tschauner, S., Singer, G., Lichtenegger, K., & Till, H. (2023) · Frontiers in Pediatrics · doi:10.3389/fped.2023.1291804 · Cited by 32
“These outcomes can take the form of visualizations, textual explanations, or examples (11, 12).”
BackgroundA Transfer Learning and Explainable Solution to Detect mpox from Smartphones images
Campana, M. G., Colussi, M., Delmastro, F., Mascetti, S., & Pagani, E. (2024) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2023.101874 · Cited by 29
Soe, N. N., Yu, Z. T., Latt, P. M., Lee, D. M., Ong, J. J., Ge, Z., Fairley, C. K., & Zhang, L. (2024) · BMC Medicine · doi:10.1186/s12916-024-03512-x · Cited by 28
Pediatric brain tumor classification using deep learning on MR images with age fusion
Tampu, I. E., Bianchessi, T., Blystad, I., Lundberg, P., Nyman, P., Eklund, A., & Haj‐Hosseini, N. (2024) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdae205 · Cited by 34
Metadata
- Authors
- Singh, A., Sengupta, S., & Lakshminarayanan, V.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Singh, Amitojdeep
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2020-06-20
- Volume
- 6
- Issue
- 6
- Pages
- 52
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC8321083
- OpenAlex
- W3031116957
- Semantic Scholar
- af6af40155860ac2fd36680e63686473803b5f52
- Cited by
- 718
- References
- 75
- Citations by year
- 2026: 82 · 2025: 135 · 2024: 132 · 2023: 154 · 2022: 130 · 2021: 77 · 2020: 8
- FWCI
- 3.17
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Nvidia
- Compute Canada
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- explainable deep learning, medical image analysis, model interpretability, black-box models, clinical deployment, feature importance, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.