Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 123 of the 123 references it cites were found in a registry or an index; 11 of the 16 sources its claims cite were checked for support.

Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges

Open access
Abbas, Qaiser, Jeong, Woonyoung, Lee, Seung Won · 2025 · Healthcare

Abstract

This meta-analysis synthesizes 62 peer-reviewed studies on XAI in CDSSs and reports gaps in user-friendly evaluation, methodological transparency, and ethical issues, alongside limited study of explanation fidelity, clinician trust, and usability in real-world settings.

Overview

CoChat Summary

TL;DR

This systematic review/meta-analysis synthesizes 62 peer-reviewed studies (2018–2025) on explainable AI in clinical decision support systems, finding that SHAP, LIME, and Grad-CAM dominate while evaluation, usability, and ethical integration remain underdeveloped. It proposes a Task–Modality–Explanation Alignment (TMEA) framework and calls for standardized metrics, participatory design, and longitudinal clinical validation.

Key findings

  • Across 62 studies, SHAP was the most frequently used XAI method, followed by LIME and Grad-CAM; model-agnostic techniques dominated tabular/EHR tasks, while model-specific methods such as Grad-CAM and attention mechanisms prevailed in imaging and sequential data. Concept-based explanations appeared in 4 studies.
  • Reported model performance had a median AUC of 0.87 (IQR 0.81–0.93), accuracy 86.4, sensitivity 84.1, and specificity 85.3. Studies combining high predictive performance with explanation fidelity ≥0.85 reported clinician trust scores 12–18 percentage points higher than those without quantitative explanation evaluation.
  • Explanation evaluation was limited: fidelity n = 16, consistency n = 10, human trust/agreement n = 9, localization accuracy n = 6, and qualitative case studies n = 12. Only 18 of 62 studies reported clinical validation through physician feedback, usability testing, or pilot deployment.
  • Methodological and ethical gaps were common: 87% of studies reported improved interpretability, but only 11 used robust statistical testing; 17.7% conducted formal significance tests; confidence intervals for interpretability metrics were reported in eight studies; 29% shared complete codebases; 18/62 (29%) reported clinician involvement; 22.5% conducted formal trust scoring; and 15% included fairness or bias mitigation strategies.

Methods

  • PRISMA-guided systematic review searching PubMed, IEEE Xplore, Scopus, and Web of Science from January 2018 to May 2025 using Boolean keyword combinations. The initial search returned 1,824 records; after removing 312 duplicates, 1,512 records were screened, 182 full texts assessed, and 62 studies included.
  • Inclusion criteria required peer-reviewed English primary studies applying XAI to real-world clinical data or simulations for CDSSs and evaluating interpretability, transparency, usability, or trust. Reviews, editorials, studies without XAI implementation/evaluation, non-healthcare domains, and non-peer-reviewed preprints were excluded. Quality was assessed with a 10-point checklist adapted from CONSORT-AI and STARD-AI; studies scoring below 5 were excluded.
  • Due to heterogeneity, no conventional effect-size meta-analysis was performed. The authors conducted a text-only quantitative synthesis of study-level proportions with 95% binomial confidence intervals (Wilson method) plus narrative synthesis grouped by XAI technique, clinical use case, and evaluation outcome. Software included Rayyan 1.6.1, Excel 2406, Python 3.11.5, and LaTeX 2024.

Limitations

  • The paper states: restriction to English-language peer-reviewed articles may introduce language and publication bias; inconsistent reporting across studies limited replicability; qualitative synthesis involved subjectivity; the evolving XAI landscape means LLM/foundation-model explainability was minimally represented; no formal quantitative meta-analysis was possible due to heterogeneous metrics and poor statistical reporting; model and domain diversity was skewed toward imaging and structured EHR data; and most studies lacked patient, nurse, or administrator perspectives.
  • Evident from the text: despite the title’s use of “Meta-Analysis,” the authors explicitly did not perform a conventional effect-size meta-analysis, relying instead on descriptive statistics and thematic synthesis. Several frequency results are reported narratively or vaguely (e.g., “many studies”) rather than with exact counts, and the review notes that explanation effectiveness, clinical impact, and usability interpretations remain inherently subjective.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/healthcare13172154

Cites

123 of 123 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Frequency of XAI method usage across the studies. SHAP, Grad-CAM, and LIME were the most frequently applied techniques, highlighting their popularity in interpretable AI research.
Figure 1 Frequency of XAI method usage across the studies. SHAP, Grad-CAM, and LIME were the most frequently applied techniques, highlighting their popularity in interpretable AI research.CC BY · PubMed Central
Figure 2 Systematic review workflow following the PRISMA guidelines. The flowchart outlines the key stages from the formulation of the research questions to final quality assessment in the study selection process.
Figure 2 Systematic review workflow following the PRISMA guidelines. The flowchart outlines the key stages from the formulation of the research questions to final quality assessment in the study selection process.CC BY · PubMed Central
Figure 3 Distribution of XAI techniques across clinical domains. This heatmap visualizes how frequently different explainable AI methods were applied across medical specialties such as radiology, oncology, and cardiology.
Figure 3 Distribution of XAI techniques across clinical domains. This heatmap visualizes how frequently different explainable AI methods were applied across medical specialties such as radiology, oncology, and cardiology.CC BY · PubMed Central
Figure 4 Domain-stratified prevalence of XAI methods with 95% confidence intervals. Points denote pooled proportions; whiskers indicate 95% CIs. Domains include radiology, pathology, EHR/tabular, time-series/physiology, text/NLP, and multimodal.
Figure 4 Domain-stratified prevalence of XAI methods with 95% confidence intervals. Points denote pooled proportions; whiskers indicate 95% CIs. Domains include radiology, pathology, EHR/tabular, time-series/physiology, text/NLP, and multimodal.CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 10 Not valid · 8 Not checked · 77 Own results and interpretations

59 more claims

Create a free account to see all 98 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 22
  • Probably not a substantial claim. 19
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 119 references

Research works printed but not linked: 4

  1. Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. (2020, January 13–18). Concept Bottleneck Models. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Virtual.
  2. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., and Viegas, F. (2018, January 10–15). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). Proceedings of the International Conference on Machine Learning, Stockholm, Sweden. PMLR.
  3. Chen, C.K., Li, O., Tao, D., Barnett, J., Rudin, C., and Su, J. (2019, January 8–14). This looks like that: Deep learning for interpretable image recognition. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
  4. Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6–11). Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia. PMLR.

Cited by

See all 236

Metadata

Authors
Abbas, Q., Jeong, W., & Lee, S. W.
Affiliations
  • Institute of Space Technology (PK)
  • Department of Metabiohealth, Institute for Cross-Disciplinary Studies, Sungkyunkwan University, Suwon 16419, Republic of Korea
  • Kangbuk Samsung Hospital (KR)
Corresponding author
Lee, Seung Won
Venue
Healthcare
Publisher
MDPI AG
Published
2025-08-29
Volume
13
Issue
17
Pages
2154
ISSN
2227-9032 (ISSN-L)
OpenAlex
W4413811448
Cited by
236
References
119
Citations by year
2026: 211 · 2025: 25
FWCI
22.53
Type
journal article
Language
en
Open access
Locations
  • Healthcare (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
Funders
Ministry of Education and Ministry of Science & ICT, Republic of Korea (NRF [2021-R1-I1A2 (059735)], RS [2024-0040 (5650)], RS [2024-0044 (0881)], RS [2019-II19 (0421)], RS [2025-2544 (3209)])
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
explainable AI, clinical decision support systems, Grad-CAM, attention mechanism, model-agnostic explanation, explanation fidelity, clinician trust, usability, interpretability evaluation, radiology, neurology, intensive care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges. https://app.cochat.ai/papers/10.3390/healthcare13172154
The paper
Abbas, Q., Jeong, W., & Lee, S. W. (2025). Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges. Healthcare. https://doi.org/10.3390/healthcare13172154

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.