The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Unveiling the black box: A systematic review of Explainable Artificial Intelligence in medical image analysis
Open accessAbstract
This systematic literature review examines state-of-the-art Explainable Artificial Intelligence (XAI) methods applied to medical image analysis, discussing current challenges and future research directions, and exploring evaluation metrics used to assess XAI approaches. With the growing efficiency of Machine Learning (ML) and Deep Learning (DL) in medical applications, there's a critical need for adoption in healthcare. However, their "black-box" nature, where decisions are made without clear explanations, hinders acceptance in clinical settings where decisions have significant medicolegal consequences. Our review highlights the advanced XAI methods, identifying how they address the need for transparency and trust in ML/DL decisions. We also outline the challenges faced by these methods and propose future research directions to improve XAI in healthcare. This paper aims to bridge the gap between cutting-edge computational techniques and their practical application in healthcare, nurturing a more transparent, trustworthy, and effective use of AI in medical settings. The insights guide both research and industry, promoting innovation and standardisation in XAI implementation in healthcare.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“This is further supported by the European Union’s General Data Protection Regulation (GDPR), which mandates transparency in algorithmic decision-making processes before their employment in patient-care settings [8].”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Engineering Design via Surrogate Modelling: A Practical Guide
Keane, A., Forrester, A., & Sobester, A. (2008) · American Institute of Aeronautics and Astronautics eBooks · doi:10.2514/4.479557 · Cited by 1,686
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Systematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“Our systematic literature review methodology comprises three distinct phases: (i) active planning, (ii) conducting and reporting the review results, and (iii) exploration of research challenges according to the widely accepted guidelines and processes outlined in [55] and [56,57].”
2020 International Joint Conference on Neural Networks (IJCNN)
(2020) · doi:10.1109/ijcnn48605.2020 · Cited by 774
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Explainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Medical Imaging 2012: Image Processing
Haynor, D. R., & Sebastien Ourselin, A. (2012) · SPIE eBooks · Cited by 563
DARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546
Explainable AI for Healthcare 5.0: Opportunities and Challenges
Saraswat, D., Bhattacharya, P., Verma, A., Prasad, V. K., Tanwar, S., Sharma, G., Bokoro, P. N., & Sharma, R. (2022) · IEEE Access · doi:10.1109/access.2022.3197671 · Cited by 523
2023 IEEE International Conference on Image Processing (ICIP)
(2023) · doi:10.1109/icip49359.2023 · Cited by 353
2018 IEEE Winter Conference on Applications of Computer Vision (WACV)
(2018) · doi:10.1109/wacv41190.2018 · Cited by 350
Hossain, M. S., Muhammad, G., & Guizani, N. (2020) · IEEE Network · doi:10.1109/mnet.011.2000458 · Cited by 347
Ranjbarzadeh, R., Caputo, A., Tirkolaee, E. B., Jafarzadeh Ghoushchi, S., & Bendechache, M. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106405 · Cited by 333
Self-citationBai, X., Wang, X., Liu, X., Liu, Q., Song, J., Sebe, N., & Kim, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108102 · Cited by 313
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283
An algorithmic approach to reducing unexplained pain disparities in underserved populations
Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S., & Obermeyer, Z. (2021) · Nature Medicine · doi:10.1038/s41591-020-01192-7 · Cited by 268
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 1×
“The SHAP value 𝜙 𝑖 for a feature 𝑖 , and calculate its average impact across every combination of features, represented in Eq.”
Explainable AI for Retinoblastoma Diagnosis: Interpreting Deep Learning Models with LIME and SHAP
Aldughayfiq, B., Ashfaq, F., Jhanjhi, N. Z., & Humayun, M. (2023) · Diagnostics · doi:10.3390/diagnostics13111932 · Cited by 211
Shen, Y., Wu, N., Phang, J., Park, J., Liu, K., Tyagi, S., Heacock, L., Kim, S. G., Moy, L., Cho, K., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101908 · Cited by 207
Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P. (2019) · Radiology · doi:10.1148/radiol.2018181422 · Cited by 206
Nafisah, S. I., & Muhammad, G. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07258-6 · Cited by 194
Ryo, M., Angelov, B., Mammola, S., Kass, J. M., Benito, B. M., & Hartig, F. (2021) · Ecography · doi:10.1111/ecog.05360 · Cited by 182
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2023) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 180
Game theory interpretation of digital soil mapping convolutional neural networks
Padarian, J., McBratney, A. B., & Minasny, B. (2020) · SOIL · doi:10.5194/soil-6-389-2020 · Cited by 178
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110787 · Cited by 169
Bai, J., Posner, R., Wang, T., Yang, C., & Nabavi, S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102049 · Cited by 167
Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review
Giuste, F., Shi, W., Zhu, Y., Naren, T., Isgut, M., Sha, Y., Li, T., Gupte, M., & Wang, M. D. (2022) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3185953 · Cited by 162
Explainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2024) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 161
Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions
Ibrahim, R., & Shafiq, M. O. (2022) · ACM Computing Surveys · doi:10.1145/3563691 · Cited by 160
xViTCOS: Explainable Vision Transformer Based COVID-19 Screening Using Radiography
Mondal, A. K., Bhattacharjee, A., Singla, P., & Prathosh, A. P. (2021) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2021.3134096 · Cited by 148
Clinical Interpretable Deep Learning Model for Glaucoma Diagnosis
Liao, W., Zou, B., Zhao, R., Chen, Y., He, Z., & Zhou, M. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2949075 · Cited by 144
A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, Á., Besa, C., Uribe, S. A., Andía, M. E., Cristián Tejos, N., Prieto, C., & Daniel Capurro, N. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 143
Explaining the Black-box Smoothly- A Counterfactual Approach
Singla, S., Eslami, M., Pollack, B., Wallace, S., & Batmanghelich, K. N. (2021) · Medical Image Analysis · doi:10.1016/j.media.2022.102721 · Cited by 128
Mertes, S., Huber, T., Weitz, K., Heimerl, A., & André, E. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.825565 · Cited by 119
Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis
Natekar, P., Kori, A., & Krishnamurthi, G. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00006 · Cited by 103
Explainable AI for Glaucoma Prediction Analysis to Understand Risk Factors in Treatment Planning
Kamal, M. S., Dey, N., Chowdhury, L., Hasan, S. I., & Santosh, K. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3171613 · Cited by 91
Assessment of knee pain from MR imaging using a convolutional Siamese network
Chang, G. H., Felson, D. T., Qiu, S., Guermazi, A., Capellini, T. D., & Kolachalama, V. B. (2020) · European Radiology · doi:10.1007/s00330-020-06658-3 · Cited by 75
Explainable Artificial Intelligence (XAI) for Deep Learning Based Medical Imaging Classification
Ghnemat, R., Alodibat, S., & Al-Haija, Q. S. A. (2023) · Journal of Imaging · doi:10.3390/jimaging9090177 · Cited by 70
Towards Trustworthy AI in Dentistry
Ma, J., Schneider, L., Lapuschkin, S., Achtibat, R., Duchrau, M., Krois, J., Schwendicke, F., & Samek, W. (2022) · Journal of Dental Research · doi:10.1177/00220345221106086 · Cited by 66
Explanation-Driven HCI Model to Examine the Mini-Mental State for Alzheimer’s Disease
Gaur, L., Bhandari, M., Shikhar, B. S., Jhanjhi, N. Z., Shorfuzzaman, M., & Masud, M. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3527174 · Cited by 65
An Explainable Brain Tumor Detection Framework for MRI Analysis
Yan, F., Chen, Y., Xia, Y., Wang, Z., & Xiao, R. (2023) · Applied Sciences · doi:10.3390/app13063438 · Cited by 63
Volumetric breast density estimation on MRI using explainable deep learning regression
van der Velden, B. H. M., Janse, M. H. A., Ragusi, M. A. A., Loo, C. E., & Gilhuijs, K. G. A. (2020) · Scientific Reports · doi:10.1038/s41598-020-75167-6 · Cited by 61
Stanley, E. A. M., Wilms, M., Mouchès, P., & Forkert, N. D. (2022) · Journal of medical imaging · doi:10.1117/1.jmi.9.6.061102 · Cited by 56
Kumar, A., Manikandan, R., Köse, U., Gupta, D. K., & Satapathy, S. C. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3457187 · Cited by 55
Li, M., Li, X., Jiang, Y., Zhang, J., Luo, H., & Yin, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109278 · Cited by 50
Yoon, K., Kim, J., Kim, J., Huh, J., Kim, J., & Choi, J. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107465 · Cited by 49
The effect of machine learning explanations on user trust for automated diagnosis of COVID-19
Goel, K., Sindhgatta, R., Kalra, S., Goel, R., & Mutreja, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105587 · Cited by 44
Wireless Capsule Endoscopy Image Classification: An Explainable AI Approach
Varam, D., Mitra, R., Mkadmi, M., Riyas, R. A., Abuhani, D. A., Dhou, S., & Alzaatreh, A. (2023) · IEEE Access · doi:10.1109/access.2023.3319068 · Cited by 40
Oberste, L., & Heinzl, A. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3227225 · Cited by 39
Sun, J., Shi, W., Giuste, F., Vaghani, Y. S., Tang, L., & Wang, M. D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46493-2 · Cited by 34
Farrag, A., Gad, G., Fadlullah, Z. M., Fouda, M. M., & Alsabaan, M. (2023) · IEEE Access · doi:10.1109/access.2023.3330465 · Cited by 33
Unboxing the Black Box of Attention Mechanisms in Remote Sensing Big Data Using XAI
Zaryabi, E. H., Moradi, L., Kalantar, B., Ueda, N., & Halin, A. B. A. (2022) · Remote Sensing · doi:10.3390/rs14246254 · Cited by 32
Using Causal Analysis for Conceptual Deep Learning Explanation
Singla, S., Wallace, S., Triantafillou, S., & Batmanghelich, K. N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_49 · Cited by 27
Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals
Resta, M., Monreale, A., & Bacciu, D. (2021) · Entropy · doi:10.3390/e23081064 · Cited by 25
A model-agnostic, network theory-based framework for supporting XAI on classifiers
Bonifazi, G., Cauteruccio, F., Corradini, E., Marchetti, M., Terracina, G., Ursino, D., & Virgili, L. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122588 · Cited by 25
Ali, S., Hussain, A., Bhattacharjee, S., Athar, A., Abdullah, & Kim, H. (2022) · Sensors · doi:10.3390/s22249983 · Cited by 22
Priya Dharshini, S., Ramkumar, K., Venkatesh, S., Narasimhan, K., & Adalarasu, K. (2023) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs57279.2023.10113001 · Cited by 22
Shin, H., Park, J. E., Jun, Y., Eo, T., Lee, J., Kim, J. E., Lee, D. H., Moon, H. H., Park, S. I., Kim, S., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09710-0 · Cited by 22
An Explainable System for Diagnosis and Prognosis of COVID-19
Lu, J., Jin, R., Song, E., Alrashoud, M. R., Al-Mutib, K. N., & Al-Rakhami, M. S. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3037915 · Cited by 12
Prediction of quality of life in people with ALS
Antoniadi, A. M., Galvin, M., Heverin, M., Hardiman, O. M., & Mooney, C. (2021) · ACM SIGAPP Applied Computing Review · doi:10.1145/3477127.3477128 · Cited by 11
Towards an Explainable AI-based Tool to Predict the Presence of Obstructive Coronary Artery Disease
Kyparissidis-Kokkinidis, I., Rigas, E. S., Logaras, E., Samaras, A., Rampidis, G. P., Giannakoulas, G. A., Κούσκουρας, Κ., Billis, A., & Bamidis, P. D. (2022) · Panhellenic Conference on Informatics · doi:10.1145/3575879.3576014 · Cited by 10
Predicting the Evolution of Pain Relief: Ensemble Learning by Diversifying Model Explanations
da Costa, A. B., Moreira, L. I., de Andrade, D. C., Veloso, A., & Ziviani, N. (2021) · ACM Transactions on Computing for Healthcare · doi:10.1145/3466781 · Cited by 5
Olar, A., Biricz, A., Bedőházi, Z., Sulyok, B., Pollner, P., & Csabai, I. (2023) · Scientific Reports · doi:10.1038/s41598-023-30505-2 · Cited by 5
Cited by
Explainable Artificial Intelligence in Radiological Cardiovascular Imaging—A Systematic Review
Haupt, M., Maurer, M. H., & Thomas, R. P. (2025) · Diagnostics · doi:10.3390/diagnostics15111399 · Cited by 58
Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review
Ebraheem, M., Toghranegar, J., Bridge2AI-Voice Consortium, Bensoussan, Y., & Templeton, J. M. (2026) · Journal of Medical Internet Research · doi:10.2196/83790
Metadata
- Authors
- Muhammad, D., & Bendechache, M.
- Venue
- Computational and Structural Biotechnology Journal
- Published
- 2024-12
- Volume
- 24
- Pages
- 542-560
- ISSN
- 2001-0370 (ISSN-L)
- OpenAlex
- W4401503494
- Cited by
- 258
- References
- 86
- FWCI
- 11.58
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.