The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 72 of the 128 sources its claims cite were checked for support.
Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis
Open accessAbstract
While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-centered framework to guide more effective and trustworthy XAI-CDSS development. Drawing on a comprehensive literature review, we detail a three-phase framework encompassing user-centered XAI method selection, interface co-design, and iterative evaluation and refinement. We demonstrate its application through a retrospective case study analysis of a published XAI-CDSS for sepsis care. Our synthesis highlights the importance of aligning XAI with clinical workflows, supporting calibrated trust, and deploying robust evaluation methodologies that capture real-world clinician–AI interaction patterns, such as negotiation. The case analysis shows how the framework can systematically identify and address user-centric gaps, leading to better workflow integration, tailored explanations, and more usable interfaces. We conclude that achieving trustworthy and clinically useful XAI-CDSS requires a fundamentally user-centered approach; our framework offers actionable guidance for creating explainable, usable, and trusted AI systems in healthcare.
Overview
CoChat Summary
TL;DR
This literature review synthesizes XAI for clinical decision support systems and argues that technical XAI often fails to address clinician needs, workflow integration, and evaluation. It proposes a three-phase user-centered framework and illustrates it retrospectively on a published sepsis XAI-CDSS case.
Key findings
- The review identified approximately 1032 records; after removing 274 duplicates, 758 titles/abstracts were screened, 56 full texts reviewed, and 30 core studies were included (13 foundational pre-2021 and 17 recent 2021–2024).
- A persistent CDS–clinician gap arises from three factors: clinicians’ needs and abilities, interface design, and evaluation of XAI methods; only a handful of papers included formal user evaluations of XAI components, and just one study explicitly investigated whether explanations helped users trust or use the model’s output.
- The proposed three-phase framework comprises user-centered XAI method selection, user-centered interface design, and evaluation and iterative refinement.
- In the retrospective sepsis case analysis of Sivaraman et al. (24 clinicians, mixed methods), SHAP explanations increased confidence but not concordance; a “negotiation” interaction pattern was dominant; trust was driven more by external validation than interface features or explanation types; and a mismatch remained between discrete AI recommendations and continuous clinical workflow. Related work found a disconnect between low self-reported trust and high behavioral reliance (WoA) in Laxar et al., and greater clinician trust for example-based over feature-importance explanations in Gomez et al.
Methods
- Literature search from 2010 through December 2024 across PubMed, Scopus, Web of Science, Google Scholar, the ACM Digital Library, and IEEE Xplore, using combined XAI and CDSS keywords; PRISMA-style screening with inclusion criteria for peer-reviewed publications or highly cited preprints focused on XAI in healthcare CDSS.
- Thematic synthesis of the final 30 core studies, focusing primarily on the 17 recent publications (2021–2024), with foundational works integrated for background and contrast.
- The paper proposes a three-phase user-centered framework and applies it retrospectively to a published XAI-CDSS implementation for sepsis care (Sivaraman et al., CHI 2023), contextualized with Laxar et al. and Gomez et al.; the analysis is illustrative, not a formal validation.
Limitations
- The paper states that the literature review may not have captured every relevant study; the case study is a retrospective, post hoc analysis of a single published work and not a prospective validation; the framework is conceptual and Phase 3 evaluation principles remain conceptual; the framework is primarily clinician-centric and not adapted for patient-facing XAI; it relies heavily on end-user involvement, and clinician attitudes vary (e.g., early vs. late believers, automation bias, confirmation bias); the framework requires further empirical validation and ongoing refinement.
- Evident from the text: no new data were created or analyzed, and the case analysis is presented as an illustration of the framework’s analytical utility rather than a real-world prospective test.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
54 Valid · 11 Not valid · 42 Not checked · 88 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“A central challenge in this domain is navigating the well-documented trade-off between model accuracy and inherent interpretability [6].”
What the sources say
- Disputedcites [6] Rudin, 2019, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell. sourceExperimental — accuracy not yet measured
In the section titled “It is a myth that there is necessarily a trade-off between accuracy and interpretability” (page not supplied), Rudin describes the trade-off as a widespread belief, not an established constraint. She says that, particularly with structured data and meaningful features, simpler interpretable models often perform comparably to more complex ones. (LLM, conf=0.94)
Checked by a reasoning model — may be wrong.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“XAI methods, therefore, hold significant potential to enhance the implementation, utility, and adoption of AI in CDSS.”
What the sources say
- Partly supportedcites [8] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceExperimental — accuracy not yet measured
The indexed Abstract describes explainability as crucial to the practical deployment of AI and says it can help professionals embrace AI in their fields. It does not address clinical decision support systems specifically or establish improved utility in that setting. (LLM, conf=0.83)
- Partly supportedcites [9] Tjoa, 2020, A survey on explainable artificial intelligence (XAI): Toward medical XAI, IEEE Trans. Neural Netw. Learn. Syst. sourceExperimental — accuracy not yet measured
The Abstract (page not provided) says explanations are needed to justify the reliability of medical AI, and §C, “Other Perspectives: Data-Driven” (page not provided), describes interpretable information helping users decide whether to follow a decision-support system. These passages support potential clinical utility, but the paper does not establish that XAI enhances implementation or adoption of AI in CDSS; §D, “Future Directions for Clinicians and Practitioners,” cautions that interpretability remains complementary support requiring human supervision. (LLM, conf=0.80)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Predictions from these ML models are often characterized by a lack of interpretability [5].”
What the sources say
- Supportedcites [5] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. source
The paper’s opening summary (p. 1) says complex algorithms often cannot provide insights into their behavior. The Introduction (p. 1) also describes widespread black-box models and the difficulty of understanding opaque models, supporting the claim that ML predictions often lack interpretability. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, surrogate models may only accurately approximate the original model in specific local regions [22].”
What the sources say
- Supportedcites [22] Magesh, P.R., Myloth, R.D., and Tom, R.J. (2020). An explainable machine learning model for early detection of Parkinson’s disease using LIME on DaTSCAN imagery. Comput. Biol. Med., 126. source
Section 4.2 describes LIME as a local surrogate model trained with greater weight on samples near the instance being explained. It says the learned model must plausibly estimate the original model’s predictions locally, supporting the claim’s qualified statement. (LLM, conf=0.90)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The effectiveness of such complex cognitive work depends on the characteristics of both the user and the CDSS, as well as the strength of the coupling between them.”
What the sources say
- Partly supportedcites [40] Sedig, 2017, Aligning information technologies with evidence-based health-care activities: A design and evaluation framework, Hum. Technol. Interdiscip. J. Humans Ict Environ. sourceExperimental — accuracy not yet measured
The joint cognitive-systems section says cognitive work emerges from the interaction of the stakeholder and health IT, and “The Mental Space” section links stronger coupling to better support for evidence-based health-care activities. The paper frames this as a principle about health IT generally, rather than making the claim specifically about CDSSs. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
59 more claims · 26 with their cited sources checked
Create a free account to see all 195 claims and to verify the 27 references still open.
- Probably not a substantial claim. 93
- It may state two separate claims as one. 69
- Borderline: one of the tests for a substantial claim is unclear. 1
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 3×
“Ultimately, the goal is to enable users to understand, and consequently trust, the models and their outputs [8,9].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 6×
“There are two general categories of XAI methods: ante hoc and post hoc [8,9,11,12].”
InfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · cited 2×
“Much of the literature still speaks in subjective terms of ‘good’ or ‘quality’ explanations [73], often without measuring how well those explanations integrate into clinicians’ decision-making processes.”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 1×
“These include propagation-based techniques like LRP, which backpropagate a quantity of interest to determine input contributions [30–32], and activation maximization, which identifies input patterns that maximally activate specific neurons [33].”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · cited 2×
“Ultimately, the goal is to enable users to understand, and consequently trust, the models and their outputs [8,9].”
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Research works printed but not linked: 9
- Hooker, S., Erhan, D., Kindermans, P.J., and Kim, B. (2019). A benchmark for interpretability methods in deep neural networks. Adv. Neural Inf. Process. Syst., 32.
- Lundberg, S.M., and Lee, S.I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30.
- Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
- Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst., 29.
- Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., and Stewart, W. (2016). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Adv. Neural Inf. Process. Syst., 29.
- Che, Z., Purushotham, S., Khemani, R., and Liu, Y. (2017, January 4–8). Interpretable deep models for ICU outcome prediction. Proceedings of the AMIA Annual Symposium Proceedings, Washington, DC, USA.
- Katuwal, G.J., and Chen, R. (2016). Machine learning model interpretability for precision medicine. arXiv.
- Sedig, K., and Liang, H.N. (2007, January 25–29). On the design of interactive visual representations: Fitness of interaction. Proceedings of the EdMedia+ Innovate Learning, Vancouver, BC, Canada.
- Gadanidis, 2004, Designing online mathematical investigation, J. Comput. Math. Sci. Teach., 23, 275
Cited by
EvoMDT: a self-evolving multi-agent system for structured clinical decision-making in multi-cancer
Liu, Q., Hu, Z., Huang, T., Niu, Y., Zhang, X., Ma, S., Lin, C., Goh, K. H., Kwon, H. E., Gao, F., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02304-8 · Cited by 31
“…trail;futuredeploymentswillintegratefull-textnotesandimagingreports via controlled parsers with human-in-the-loop veri fi cation 25 single-blind physician ratings and low safety-violation rates complement benchmark gains, aligning with CONSORT-AI/DECIDE-AI emphases on…”
Artificial Intelligence for Predicting Difficult Airways: A Review
Alatau, M., Bauer, J., & Sazonov, V. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14238600 · Cited by 12
“A growing body of literature emphasizes the importance of explainability in clinical decision support systems (CDSSs) beyond raw predictive performance [18].”
Artificial Intelligence-Based Models for Predicting Disease Course Risk Using Patient Data
Chowdhury, R. I., Bari, W., Hasan, M. T., Hossain, Z., & Rahman, M. (2026) · Computers · doi:10.3390/computers15020113 · Cited by 2
“ML (a subfield of AI) can learn from patients’ past follow-up data and recognize useful patterns [71,72].”
Sun, T., Liu, S., Qian, K., & Li, S. (2026) · Journal of Materials Chemistry B · doi:10.1039/d6tb00696e · Cited by 4
Ramos, F., dos Santos, A. M. P., Rocha, T., Petronilho, F. A. S., & Pereira, R. P. G. (2026) · Systems · doi:10.3390/systems14070764 · Cited by 3
“Recent evidence reinforces this gap, indicating that current explain-able AI approaches in clinical decision support still face unresolved challenges related to integration, usability, and real-world applicability [4].”
Metadata
- Authors
- Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T.
- Affiliations
- Western University (CA)
- Corresponding author
- Sedig, Kamran
- Venue
- Informatics
- Publisher
- MDPI AG
- Published
- 2025-10-28
- Volume
- 12
- Issue
- 4
- Pages
- 119
- ISSN
- 2227-9709 (ISSN-L)
- OpenAlex
- W4415620636
- Cited by
- 39
- References
- 64
- Citations by year
- 2026: 38 · 2025: 1
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Informatics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- explainable AI, clinical decision support systems, model interpretability, clinician trust in AI, user-centered design, clinical workflow integration, explanation evaluation, sepsis care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis. https://app.cochat.ai/papers/10.3390/informatics12040119
- The paper
- Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025). Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis. Informatics. https://doi.org/10.3390/informatics12040119
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.