Survey of Explainable AI Techniques in Healthcare
Open accessThe DOI matches its Crossref record; 119 of the 120 references it cites were found in a registry or an index, and 1 are not checked yet; 8 of the 50 sources its claims cite were checked for support.
Abstract
Artificial intelligence (AI) with deep learning models has been widely applied in numerous domains, including medical imaging and healthcare tasks. In the medical field, any judgment or decision is fraught with risk. A doctor will carefully judge whether a patient is sick before forming a reasonable explanation based on the patient’s symptoms and/or an examination. Therefore, to be a viable and accepted tool, AI needs to mimic human judgment and interpretation skills. Specifically, explainable AI (XAI) aims to explain the information behind the black-box model of deep learning that reveals how the decisions are made. This paper provides a survey of the most recent XAI techniques used in healthcare and related medical imaging applications. We summarize and categorize the XAI types, and highlight the algorithms used to increase interpretability in medical imaging topics. In addition, we focus on the challenging XAI problems in medical applications and provide guidelines to develop better interpretations of deep learning models using XAI concepts in medical image and text analysis. Furthermore, this survey provides future directions to guide developers and researchers for future prospective investigations on clinical topics, particularly on applications with medical imaging.
Verified 2026-10-05 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
This survey reviews recent explainable AI (XAI) techniques for healthcare and medical imaging, categorizing them by interpretation type, model specificity, explanation scope, and explanation form. It highlights commonly used methods such as Grad-CAM, LIME, SHAP, attention, prototypes, and radiomics, and discusses human-centered, deployment, quality, and future challenges.
Key findings
- XAI methods are categorized as intrinsic vs. post hoc, model-specific vs. model-agnostic, local vs. global, and feature-based/textual/example-based; post hoc methods such as CAM/Grad-CAM and LIME are most commonly used in medical imaging.
- The paper reports that clinician satisfaction with LIME chart interpretation reached 78%, though many low-satisfaction clinicians gave low scores despite recognizing the explanations.
- In an evaluation of saliency maps, eight interpretable methods (+ Grad-CAM, guided backpropagation, and guided Grad-CAM) were assessed, but performance on the testing datasets was not competitive.
- For Trojan attack detection, backpropagation had a high success rate, CAM detected the entire trigger region but not always accurate localization, and LIME had high success for small triggers; LIME and SHAP showed unstable explanations with limited examples.
Methods
- Design: narrative survey of recent XAI techniques used in healthcare and related medical imaging, summarizing and categorizing XAI types and algorithms.
- Data/sample: literature and tables (Table 1, Table 2, Table 3) covering XAI methods, modalities, and explanations; no primary dataset or systematic review protocol is reported.
- Analysis: qualitative synthesis of principles, deployment, performance, and challenges, including discussion of radiomics and future directions.
Limitations
- The paper states that the definition of XAI is not clear enough and that explainability depends on the prediction task and target users rather than uniform standards.
- The paper states that post hoc application in the medical field requires more effort; TCAV needs more investigation; trainable attention needs more work; SHAP applications are limited to specific problems; LIME’s low repeatability limits interpretability; and evaluating XAI explanations remains a challenge.
- Evident from the text: the survey does not report a systematic search strategy, inclusion/exclusion criteria, or quantitative meta-analysis; it is a narrative overview.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Flowchart of visual comparison between black-box and explainable artificial intelligence, and how the results affect the user. The top branch shows the process of a black-box model. Typically, it provides only results such as classes (e.g., COVID or non-COVID). The other two branches (middle and bottom) represent two XAI methods, referred to in Section 3.1. Specifically, the XAI model (middle) shows the example of saliency map, and the second one (bottom) is the prototype method, as explained in Section 4.1 and Section 4.6, respectively. An example of a CT image obtained from the COVID-19 CT scan dataset [10].](https://figures.cochat.ai/paper-figures/7410cd2efd4c4f5c87ab22ce2e01ec4c/survey-of-explainable-ai-techniques-in-healthcare-figure-1.jpg)

![Figure 3 Categorization of explainable AI methods used in this paper. These criteria and categories are summarized from [25,26,27]. The orange and blue grids represent the criteria and categories, respectively.](https://figures.cochat.ai/paper-figures/89dcecc050d5476e8f47abead15c4155/survey-of-explainable-ai-techniques-in-healthcare-figure-3.jpg)
Verification
Claims the paper makes
5 Valid · 2 Not valid · 36 Not checked · 19 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Notably, the bias may be derived from data, algorithms, and user interactions [24].”
§ Survey of Explainable AI Techniques in Healthcare > 2. XAI Techniques Related to Medical Imaging > 2.3. Bias and Fairness
What the sources say
- Supportedcites [24] Mehrabi, 2021, A survey on bias and fairness in machine learning, ACM Comput. Surv. (CSUR) source
Section 3 of Mehrabi et al. is titled 'BIAS IN DATA, ALGORITHMS, AND USER EXPERIENCES' (p. heading, passage [3]), and its opening text (passage [2]) states that biases in training data are learned and amplified by algorithms, that algorithms can display biased behaviour from design choices even on unbiased data, and that user interactions feed back into new biased data. Subsections 3.1.2 and 3.1.3 (passages [1], [6], [7]) explicitly define algorithmic bias and user interaction bias and describe user-generated data as a bias source, matching the claim's three origins. (LLM, conf=0.92)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In [15], the authors identified three types of security risks related to black-box AI: network attacks, system bias, and mismatch attacks.”
§ Survey of Explainable AI Techniques in Healthcare > 2. XAI Techniques Related to Medical Imaging > 2.1. Confidentiality and Privacy
What the sources say
- Partly supportedcites [15] Kiener, 2021, Artificial intelligence in medicine and the disclosure of risks, AI Soc. sourceExperimental — accuracy not yet measured
The “Three risks of black box AI” section identifies cyber-attacks, systematic bias, and mismatch as risks; page numbers are unavailable in our copy. But it describes mismatch as an AI recommendation that fails to fit a patient’s circumstances, not as an attack, and does not classify all three as security risks. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, there are many practical problems related to the speed of operation and implementation [106].”
§ Survey of Explainable AI Techniques in Healthcare > 6. Discussion, Challenges, and Prospects > 6.2. AI System Deployment
What the sources say
- Partly supportedcites [106] Sutton, 2020, An overview of clinical decision support systems: Benefits, risks, and strategies for success, NPJ Digit. Med. sourceExperimental — accuracy not yet measured
The Introduction says AI-based clinical decision support systems face challenges with explainability and have yet to reach widespread implementation. “Fragmented workflows” says such systems can increase the time clinicians need to complete tasks. These passages support practical implementation and time-related problems for clinical decision support, but do not establish a problem with the operating speed of XAI itself. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, CAM is effective in detecting the entire trigger region, but may not always provide the accurate localization of the trigger [112].”
§ Survey of Explainable AI Techniques in Healthcare > 6. Discussion, Challenges, and Prospects > 6.3. Quality of Explanation
What the sources say
- Partly supportedcites [112] Lin, Y.S., Lee, W.C., and Celik, Z.B. (2020). What do you see? Evaluation of explainable artificial intelligence (XAI) interpretability through neural backdoors. arXiv. sourceExperimental — accuracy not yet measured
Section 4.3 reports that Grad-CAM completely covers the trigger region for ResNet50 models, but highlights only a small area inside the trigger for VGG16 and AlexNet. This supports the localization caveat, not an unqualified claim that it detects the entire region. Section 2 also distinguishes Grad-CAM, the method evaluated, from CAM. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most of the recently proposed medical imaging works use post-interpretation rather than model-based interpretation (e.g., CAM and GCAM models are widely used).”
§ Survey of Explainable AI Techniques in Healthcare > 6. Discussion, Challenges, and Prospects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 62 claims and to verify the 42 references still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 13
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Explainable AI in healthcare has been surveyed extensively, yet no existing review addresses the challenges posed by decoder-based LLMs.”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These dimensions were categorized according to established and previously validated XAI taxonomies [12,17,18], which were adapted to the context of agentic AI systems as a provisional analytic schema to enable consistent classification across heterogeneous implementations.”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Nonetheless, these reviews consistently highlight persistent challenges relating to faithfulness, stability, and standardized evaluation of explanations.”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Survey of Explainable AI Techniques in Healthcare > 4. Introduction of the Explainable AI Method: A Brief Overview > 4.5. Triplet Networks
“the other two samples are randomly chosen from the training set in the same “Positive” and different “Negative” categories.”
§ Survey of Explainable AI Techniques in Healthcare > 4. Introduction of the Explainable AI Method: A Brief Overview > 4.5. Triplet Networks
“Since labeling is not necessary, this method can be used for unsupervised learning.”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 3×
“[28] BP * * * [29] Guided-BP * * * [30] Deconv Network * * * [31] LRP * * * [32] CAM * * * [33] Grad-CAM * * * [34] LIME * * * [35] GraphLIME * * * [36] SHAP * * * Feature”
BackgroundGauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“However, it is suggested to develop an interpretable model for any high-risk medical situation [8].”
BackgroundBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 1×
“Typically, XAI is an explainable model providing insights into how the predictions are made to achieve trustworthiness, causality, transferability, confidence, fairness, accessibility, and interactivity [5,6].”
Background
Research works printed but not linked: 6
- Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
- Yang, X., He, X., Zhao, J., Zhang, Y., Zhang, S., and Xie, P. (2020). COVID-CT-dataset: A CT scan dataset about COVID-19. arXiv.
- Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv.
- Jiang, C., Chen, Y., Chang, J., Feng, M., Wang, R., and Yao, J. (2021). Fusion of medical imaging and electronic health records with attention and multi-head machanisms. arXiv.
- Shapley, L.S. (2016). 17. A value for n-person games. Contributions to the Theory of Games (AM-28), Volume II, Princeton University Press.
- Nguyen, H.T.T., Cao, H.Q., Nguyen, K.V.T., and Pham, N.D.K. (2021, January 6–7). Evaluation of Explainable Artificial Intelligence: SHAP, LIME, and CAM. Proceedings of the FPT AI Conference 2021, Ha Noi, Viet Nam.
Cited by
A Review of Explainable Artificial Intelligence in Healthcare
Sadeghi, Z., Alizadehsani, R., CIFCI, M. A., Kausar, S., Rehman, R., Mahanta, P., Bora, P. K., Almasri, A., Alkhawaldeh, R. S., Hussain, S., et al. (2024) · Computers and Electrical Engineering · doi:10.1016/j.compeleceng.2024.109370 · Cited by 598
Deep learning: systematic review, models, challenges, and research directions
Talaei Khoei, T., Ould Slimane, H., & Kaabouch, N. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08957-4 · Cited by 411
Shaping the future of healthcare: ethical clinical challenges and pathways to trustworthy AI
Goktas, P., & Grzybowski, A. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14051605 · Cited by 336
“In practice, if an AI system recommends a particular treatment plan, LIME can provide a simplified explanation of that specific recommendation, enabling clinicians to evaluate its validity before implementation [11].”
Chaddad, A., Wu, Y., Kateb, R., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23146434 · Cited by 305
Self-citationŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
Metadata
- Authors
- Chaddad, A., Peng, J., Xu, J., & Bouridane, A.
- Affiliations
- Guilin University of Electronic Technology (CN)
- University of Sharjah (AE)
- Corresponding author
- Chaddad, Ahmad
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2023-01-05
- Volume
- 23
- Issue
- 2
- Pages
- 634
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC9862413
- OpenAlex
- W4313648826
- Cited by
- 591
- References
- 111
- Citations by year
- 2026: 144 · 2025: 211 · 2024: 159 · 2023: 76
- FWCI
- 12.33
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- NATIONAL NATURAL SCIENCE FOUNDATION (82260360, QN2021033002L)
- FOREIGN YOUNG TALENTS PROGRAM (82260360, QN2021033002L)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- explainable AI, health care, diagnostic imaging, deep learning interpretability, black-box models, medical image analysis, clinical decision support
- MeSH terms
- Algorithms, Artificial Intelligence, Humans, Judgment, Physicians, Research Personnel
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Survey of Explainable AI Techniques in Healthcare. https://app.cochat.ai/papers/10.3390/s23020634
- The paper
- Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023). Survey of Explainable AI Techniques in Healthcare. Sensors. https://doi.org/10.3390/s23020634
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.