Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 72 of the 128 sources its claims cite were checked for support.

Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis

Open access

Abstract

While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-centered framework to guide more effective and trustworthy XAI-CDSS development. Drawing on a comprehensive literature review, we detail a three-phase framework encompassing user-centered XAI method selection, interface co-design, and iterative evaluation and refinement. We demonstrate its application through a retrospective case study analysis of a published XAI-CDSS for sepsis care. Our synthesis highlights the importance of aligning XAI with clinical workflows, supporting calibrated trust, and deploying robust evaluation methodologies that capture real-world clinician–AI interaction patterns, such as negotiation. The case analysis shows how the framework can systematically identify and address user-centric gaps, leading to better workflow integration, tailored explanations, and more usable interfaces. We conclude that achieving trustworthy and clinically useful XAI-CDSS requires a fundamentally user-centered approach; our framework offers actionable guidance for creating explainable, usable, and trusted AI systems in healthcare.

Overview

CoChat Summary

TL;DR

This literature review synthesizes XAI for clinical decision support systems and argues that technical XAI often fails to address clinician needs, workflow integration, and evaluation. It proposes a three-phase user-centered framework and illustrates it retrospectively on a published sepsis XAI-CDSS case.

Key findings

  • The review identified approximately 1032 records; after removing 274 duplicates, 758 titles/abstracts were screened, 56 full texts reviewed, and 30 core studies were included (13 foundational pre-2021 and 17 recent 2021–2024).
  • A persistent CDS–clinician gap arises from three factors: clinicians’ needs and abilities, interface design, and evaluation of XAI methods; only a handful of papers included formal user evaluations of XAI components, and just one study explicitly investigated whether explanations helped users trust or use the model’s output.
  • The proposed three-phase framework comprises user-centered XAI method selection, user-centered interface design, and evaluation and iterative refinement.
  • In the retrospective sepsis case analysis of Sivaraman et al. (24 clinicians, mixed methods), SHAP explanations increased confidence but not concordance; a “negotiation” interaction pattern was dominant; trust was driven more by external validation than interface features or explanation types; and a mismatch remained between discrete AI recommendations and continuous clinical workflow. Related work found a disconnect between low self-reported trust and high behavioral reliance (WoA) in Laxar et al., and greater clinician trust for example-based over feature-importance explanations in Gomez et al.

Methods

  • Literature search from 2010 through December 2024 across PubMed, Scopus, Web of Science, Google Scholar, the ACM Digital Library, and IEEE Xplore, using combined XAI and CDSS keywords; PRISMA-style screening with inclusion criteria for peer-reviewed publications or highly cited preprints focused on XAI in healthcare CDSS.
  • Thematic synthesis of the final 30 core studies, focusing primarily on the 17 recent publications (2021–2024), with foundational works integrated for background and contrast.
  • The paper proposes a three-phase user-centered framework and applies it retrospectively to a published XAI-CDSS implementation for sepsis care (Sivaraman et al., CHI 2023), contextualized with Laxar et al. and Gomez et al.; the analysis is illustrative, not a formal validation.

Limitations

  • The paper states that the literature review may not have captured every relevant study; the case study is a retrospective, post hoc analysis of a single published work and not a prospective validation; the framework is conceptual and Phase 3 evaluation principles remain conceptual; the framework is primarily clinician-centric and not adapted for patient-facing XAI; it relies heavily on end-user involvement, and clinician attitudes vary (e.g., early vs. late believers, automation bias, confirmation bias); the framework requires further empirical validation and ongoing refinement.
  • Evident from the text: no new data were created or analyzed, and the case analysis is presented as an illustration of the framework’s analytical utility rather than a real-world prospective test.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/informatics12040119

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

54 Valid · 11 Not valid · 42 Not checked · 88 Own results and interpretations

59 more claims · 26 with their cited sources checked

Create a free account to see all 195 claims and to verify the 27 references still open.

See all claims — free account
  • Probably not a substantial claim. 93
  • It may state two separate claims as one. 69
  • Borderline: one of the tests for a substantial claim is unclear. 1

Create a free account to see these claims.

Citations

Cites

Show all 64 references

Research works printed but not linked: 9

  1. Hooker, S., Erhan, D., Kindermans, P.J., and Kim, B. (2019). A benchmark for interpretability methods in deep neural networks. Adv. Neural Inf. Process. Syst., 32.
  2. Lundberg, S.M., and Lee, S.I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30.
  3. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
  4. Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst., 29.
  5. Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., and Stewart, W. (2016). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Adv. Neural Inf. Process. Syst., 29.
  6. Che, Z., Purushotham, S., Khemani, R., and Liu, Y. (2017, January 4–8). Interpretable deep models for ICU outcome prediction. Proceedings of the AMIA Annual Symposium Proceedings, Washington, DC, USA.
  7. Katuwal, G.J., and Chen, R. (2016). Machine learning model interpretability for precision medicine. arXiv.
  8. Sedig, K., and Liang, H.N. (2007, January 25–29). On the design of interactive visual representations: Fitness of interaction. Proceedings of the EdMedia+ Innovate Learning, Vancouver, BC, Canada.
  9. Gadanidis, 2004, Designing online mathematical investigation, J. Comput. Math. Sci. Teach., 23, 275

Cited by

See all 39

Metadata

Authors
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T.
Affiliations
Western University (CA)
Corresponding author
Sedig, Kamran
Venue
Informatics
Publisher
MDPI AG
Published
2025-10-28
Volume
12
Issue
4
Pages
119
ISSN
2227-9709 (ISSN-L)
OpenAlex
W4415620636
Cited by
39
References
64
Citations by year
2026: 38 · 2025: 1
FWCI
15.70
Type
journal article
Language
en
Open access
Locations
  • Informatics (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
explainable AI, clinical decision support systems, model interpretability, clinician trust in AI, user-centered design, clinical workflow integration, explanation evaluation, sepsis care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis. https://app.cochat.ai/papers/10.3390/informatics12040119
The paper
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025). Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis. Informatics. https://doi.org/10.3390/informatics12040119

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.