CoChat

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index; 215 of the 303 sources its claims cite were checked for support.

Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives

Open access
Van Dort, Bethany A., Engelsma, Thomas, Sneekes, Pamela, Peute, Linda, Medlock, Stephanie · 2026 · PLOS Digital Health

Abstract

Explainable Artificial Intelligence (XAI) has the potential to enhance clinical decision support (CDS) systems however, it remains unclear how XAI systems are perceived by healthcare professionals in hospital settings, and if new challenges arise as a result of explanations. This scoping review aimed to understand healthcare professionals’ perceptions of CDS systems with XAI in the hospital setting; specifically, the drivers of acceptance and use, explainability needs, and design preferences. Databases were searched. Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems used in hospital settings. MEDLINE, Embase, and Web of Science were searched, and reference lists were screened for additional papers. Study characteristics and health professional perceptions were extracted and inductively coded. A quality assessment was performed using the CASP checklist. Sixteen studies were identified. Included studies primarily focused on ML-based CDS systems for predicting various clinical outcomes. Most studies used feature importance or model agnostic techniques like SHapley Additive exPlanations (SHAP). Overall, healthcare professionals perceived CDS systems with XAI as useful for supporting clinical tasks, decision-making, and teamwork. Acceptance was influenced by integration into workflows, performance, data quality, and alignment with clinical knowledge. Concerns were raised about overreliance and reduced professional autonomy. Health professionals predominantly used explanations to validate outputs, and desired actionable information from systems. SHAP plots and visualizations were difficult to interpret. Participants preferred explanation designs that included concise, high-level information, and simple plots for quick interpretation. Clear visual indicators such as colour coding, contextual patient data, and aggregating similar features also aided interpretation. Poorly designed XAI explanations can hinder understanding and increase cognitive burden in busy clinical settings. Future research should optimise the design and delivery of explanations, so clinicians can appropriately trust in XAI CDS systems and feel confident in their clinical decision making.

Overview

CoChat Summary

TL;DR

This scoping review synthesized 16 qualitative studies on healthcare professionals’ perceptions of XAI-enabled clinical decision support in hospitals, finding general perceived usefulness but acceptance tied to workflow integration, performance, data quality, and alignment with clinical knowledge. SHAP plots and values were often difficult to interpret, and clinicians preferred concise, actionable, simple explanations with contextual patient data.

Key findings

  • Across 16 studies and 338 extracted qualitative results, XAI CDS systems were perceived as useful for task completion, clinical decision-making, and teamwork; usefulness was user- or context-dependent, such as for less experienced doctors, night shifts, incomplete patient information, or patients unable to communicate.
  • Trust and acceptance were influenced by workflow/EHR integration, model performance, data quality, evidence, and alignment with clinical knowledge; concerns included overreliance, reduced professional autonomy, and deskilling.
  • Explanations were mainly used to validate outputs, and participants wanted actionable information, modifiable variables, and next steps. SHAP was used in five studies, with four reporting difficulty understanding SHAP plots or values; LIME was used in three studies, and example-based explanations in two.
  • Design preferences included high-level information with details on demand, colour coding, simple plots such as bar charts and line graphs, features grouped and ordered by contribution, contextual patient data, and whole-number percentages; decimals, negative values, and outputs not summing to 100% were confusing.

Methods

  • Scoping review following PRISMA-ScR; MEDLINE, Embase, and Web of Science were searched from database inception to 15 June 2024, with reference lists screened; the review protocol was not pre-registered.
  • Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems in hospital settings; reviews, commentaries, conference abstracts, non-English studies, community/primary care settings, and perceptions collected exclusively before system development were excluded.
  • Data were extracted by at least two researchers and inductively coded into themes covering the CDS system with XAI in general, the XAI techniques, and design/usability; quality was assessed with the CASP qualitative checklist.

Limitations

  • The authors state that three relevant studies were found only through reference-list screening, not the database search, due to inconsistent terminology, so the search may have missed other relevant studies and limited comprehensiveness.
  • The authors state that design recommendations are preliminary and require higher-level evidence; no studies were excluded or weighted by CASP quality, and some included studies reported explainability or user perceptions only as secondary outcomes, limiting depth.
  • Evident from the text: the review protocol was not pre-registered, and most included systems remained at prototype or pre-implementation stages, so post-implementation perceptions are limited.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pdig.0001417

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 PRISMA flow diagram of the study selection process.
Fig 1 PRISMA flow diagram of the study selection process.CC BY · PubMed Central
Fig 2 Themes identified from health professionals’ perceptions.
Fig 2 Themes identified from health professionals’ perceptions.CC BY · PubMed Central

Verification

Claims the paper makes

80 Valid · 6 Not valid · 17 Not checked · 56 Own results and interpretations

89 more claims · 52 with their cited sources checked

Create a free account to see all 159 claims and to verify the 54 references still open.

See all claims — free account
  • It may state two separate claims as one. 44
  • Probably not a substantial claim. 31

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 39 references

Research works printed but not linked: 1

  1. Veritas Health Innovation Ltd, 2025, Covidence systematic review software

Cited by

See all 9

Metadata

Authors
Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S.
Affiliations
  • Amsterdam Public Health (NL)
  • Amsterdam University Medical Centers (NL)
Venue
PLOS Digital Health
Published
2026-05-14
Volume
5
Issue
5
Pages
e0001417
ISSN
2767-3170 (ISSN-L)
OpenAlex
W7161139886
Cited by
9
References
39
Citations by year
2026: 9
FWCI
30.26
Type
journal article
Language
en
Open access
Locations
  • PLOS Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
explainable AI, clinical decision support systems, healthcare professionals' perceptions, hospital settings, technology acceptance, explanation design, SHAP, feature importance, clinical workflow integration, overreliance on AI, professional autonomy, trust in AI, cognitive burden, clinical decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives. https://app.cochat.ai/papers/10.1371/journal.pdig.0001417
The paper
Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S. (2026). Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives. PLOS Digital Health. https://doi.org/10.1371/journal.pdig.0001417

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.