The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index; 215 of the 303 sources its claims cite were checked for support.
Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives
Open accessAbstract
Explainable Artificial Intelligence (XAI) has the potential to enhance clinical decision support (CDS) systems however, it remains unclear how XAI systems are perceived by healthcare professionals in hospital settings, and if new challenges arise as a result of explanations. This scoping review aimed to understand healthcare professionals’ perceptions of CDS systems with XAI in the hospital setting; specifically, the drivers of acceptance and use, explainability needs, and design preferences. Databases were searched. Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems used in hospital settings. MEDLINE, Embase, and Web of Science were searched, and reference lists were screened for additional papers. Study characteristics and health professional perceptions were extracted and inductively coded. A quality assessment was performed using the CASP checklist. Sixteen studies were identified. Included studies primarily focused on ML-based CDS systems for predicting various clinical outcomes. Most studies used feature importance or model agnostic techniques like SHapley Additive exPlanations (SHAP). Overall, healthcare professionals perceived CDS systems with XAI as useful for supporting clinical tasks, decision-making, and teamwork. Acceptance was influenced by integration into workflows, performance, data quality, and alignment with clinical knowledge. Concerns were raised about overreliance and reduced professional autonomy. Health professionals predominantly used explanations to validate outputs, and desired actionable information from systems. SHAP plots and visualizations were difficult to interpret. Participants preferred explanation designs that included concise, high-level information, and simple plots for quick interpretation. Clear visual indicators such as colour coding, contextual patient data, and aggregating similar features also aided interpretation. Poorly designed XAI explanations can hinder understanding and increase cognitive burden in busy clinical settings. Future research should optimise the design and delivery of explanations, so clinicians can appropriately trust in XAI CDS systems and feel confident in their clinical decision making.
Overview
CoChat Summary
TL;DR
This scoping review synthesized 16 qualitative studies on healthcare professionals’ perceptions of XAI-enabled clinical decision support in hospitals, finding general perceived usefulness but acceptance tied to workflow integration, performance, data quality, and alignment with clinical knowledge. SHAP plots and values were often difficult to interpret, and clinicians preferred concise, actionable, simple explanations with contextual patient data.
Key findings
- Across 16 studies and 338 extracted qualitative results, XAI CDS systems were perceived as useful for task completion, clinical decision-making, and teamwork; usefulness was user- or context-dependent, such as for less experienced doctors, night shifts, incomplete patient information, or patients unable to communicate.
- Trust and acceptance were influenced by workflow/EHR integration, model performance, data quality, evidence, and alignment with clinical knowledge; concerns included overreliance, reduced professional autonomy, and deskilling.
- Explanations were mainly used to validate outputs, and participants wanted actionable information, modifiable variables, and next steps. SHAP was used in five studies, with four reporting difficulty understanding SHAP plots or values; LIME was used in three studies, and example-based explanations in two.
- Design preferences included high-level information with details on demand, colour coding, simple plots such as bar charts and line graphs, features grouped and ordered by contribution, contextual patient data, and whole-number percentages; decimals, negative values, and outputs not summing to 100% were confusing.
Methods
- Scoping review following PRISMA-ScR; MEDLINE, Embase, and Web of Science were searched from database inception to 15 June 2024, with reference lists screened; the review protocol was not pre-registered.
- Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems in hospital settings; reviews, commentaries, conference abstracts, non-English studies, community/primary care settings, and perceptions collected exclusively before system development were excluded.
- Data were extracted by at least two researchers and inductively coded into themes covering the CDS system with XAI in general, the XAI techniques, and design/usability; quality was assessed with the CASP qualitative checklist.
Limitations
- The authors state that three relevant studies were found only through reference-list screening, not the database search, due to inconsistent terminology, so the search may have missed other relevant studies and limited comprehensiveness.
- The authors state that design recommendations are preliminary and require higher-level evidence; no studies were excluded or weighted by CASP quality, and some included studies reported explainability or user perceptions only as secondary outcomes, limiting depth.
- Evident from the text: the review protocol was not pre-registered, and most included systems remained at prototype or pre-implementation stages, so post-implementation perceptions are limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
80 Valid · 6 Not valid · 17 Not checked · 56 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“while two studies described their systems using general ML terminology without referencing explainability or interpretability [23,30].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Discussion
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Disputedcites [30] KE Henry, 2022, Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system, NPJ Digit Med sourceExperimental — accuracy not yet measured
The paper describes its system in ML terms, but it also explicitly discusses interpretability of the model’s computations and feature-importance measures. That contradicts the claim that this study did not reference interpretability. The indexed copy provides no page number for the passage. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Colour coding was viewed as a helpful visual cue for conveying risk severity.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Not checked yetcites [28] BA Fritz, 2024, User-centered design of a machine learning dashboard for prediction of postoperative complications, Anesth Analg source
Checked before — its earlier verdict is held back until a new check runs.
- Supportedcites [31] S Jauk, 2021, Technology acceptance of a machine learning algorithm predicting delirium in a clinical setting: a mixed-methods study, J Med Syst source
In Results > Expert group meetings, high-risk patients are shown with a yellow symbol and very-high-risk patients with a red symbol; the experts appreciated this presentation, including its “traffic light symbol.” The Discussion likewise says users appreciated the yellow and red icons in the hospital interface. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For presentation of features, healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Partly supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak sourceExperimental — accuracy not yet measured
In Results > Explanation design preferences (page not available), providers preferred grouping related features to reduce information-processing effort. The Discussion says participants valued seeing which risk factors contributed most to predicted risk. Neither passage establishes a preference for ordering features by their contribution, though the proposed display shows each feature’s contribution. (LLM, conf=0.84)
- Not checked yetcites [28] BA Fritz, 2024, User-centered design of a machine learning dashboard for prediction of postoperative complications, Anesth Analg source
Checked before — its earlier verdict is held back until a new check runs.
- Partly supportedcites [24] SG Anjara, 2023, Examining explainable clinical decision support systems with think aloud protocols, PLoS One sourceExperimental — accuracy not yet measured
In Results, §3 (page unavailable), participants criticized the lack of priority ordering and the unintuitive display of attributes contributing to the relapse score. This supports ordering features by importance, but the paper does not report a preference for grouping them or say that this would enable quick interpretation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak source
The Results (Explanation design preferences) say providers found raw patient values and time-series plots helpful for interpreting predictions and explanations. The Discussion likewise says laboratory values, vital signs, and interventions provided useful context for assessing their clinical credibility and utility. (LLM, conf=0.94)
- Supportedcites [27] N Bienefeld, 2023, Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals, NPJ Digit Med source
In the Interview results passage (labeled Abstract in our copy), clinicians say they needed to check whether a prediction made clinical sense by considering CT scans and patient outcomes. The paper also says they consulted patient history, laboratory values, diagnostic tests, and imaging in parallel with the prediction system, and recommends integrating that context into future designs. (LLM, conf=0.88)
- Supportedcites [32] Z Jin, 2020, CarePre, ACM Trans Comput Healthc source
The evaluation sections record exactly this: in the case study (B. Case Study, p. 7) the physician found the similar-patient view 'very informative' and said it gave him more confidence in the prediction results because 'it provides specific evidence [to support the prediction results],' and in the expert interview (C. Domain Expert Interview, p. 8) E1 stated that 'comparing to the similar patients in detail not only gives me more confidence of the prediction results but also provides me with rich treatment examples,' while E2 called the records of similar patients 'an important reference.' This is grounded in the clinicians' stated requirement from the pilot study (III. Pilot Study, p. 3, R2/R3) that prediction results be illustrated within the context of the patient's historical medical record and comparable similar patients to verify and interpret them. The claim's example 'patient results' is a loose rendering of the patient's/similar patients' medical-record outcomes rather than lab test results per se, but the attributed point holds. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Flexibility in presentation formats was also valued, enabling alignment with individual user preferences.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak source
In Results > Explanation design preferences, the paper reports differing preferences across clinical roles and individuals, and says providers wanted control over how they organize and view explanation content. Table 3 likewise lists interactive options for different displays and organizations for various users. These passages support the paraphrase that flexible presentation was valued to accommodate user preferences. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
89 more claims · 52 with their cited sources checked
Create a free account to see all 159 claims and to verify the 54 references still open.
- It may state two separate claims as one. 44
- Probably not a substantial claim. 31
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“| Bienefeld(2023) [27] | Switzerland | University hospital (Neurosurgical intensive care) | Predict delayed cerebral ischemia in patients with aneurysmal subarachnoid haemorrhage | Prototype testing (low and high fidelity) | No | SHAP |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“| Pinto(2023) [34] | Portugal | Epilepsy refractory center, hospital | Prediction for EEG seizures | Prototype testing | No |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
““What you want are modifiable things... History of hypertension? You’re not going to be able to do anything about that” [21].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > References
“21. Veritas Health Innovation Ltd. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation Ltd; 2025.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
“participants in three studies reported wanting to see the variables that could be altered through interventions”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
“[23,25,28,32]”
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565 · cited 1×
“The PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guidelines [20] were followed in the reporting of this review and the checklist is included in the Supporting Information (S1 PRISMA Checklist).”
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · cited 1×
“A scoping review was conducted as XAI is a new field, and this method allowed for a broad mapping of existing literature and identification of key concepts related to health professionals’ perceptions of XAI in hospital settings [19].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Explainable Artificial Intelligence (XAI) aims to enhance transparency, trustwor-thiness, and accountability of AI systems by providing explanations for their decisions [4,5].”
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
Haenlein, M., & Kaplan, A. (2019) · California Management Review · doi:10.1177/0008125619864925 · Cited by 2,655
Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., Spurr, C., Khorasani, R., Tanasijevic, M., & Middleton, B. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1370 · Cited by 1,346
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897 · cited 3×
“Another commonly used method is Local Interpretable Model-agnostic Explanations (LIME), which approximates the predictions of a complex model locally using a simpler, interpretable model to explain individual predictions [9].”
InfluentialLoh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734 · cited 2×
“Previous reviews have explored XAI in healthcare, with a focus on explainability techniques [11,12], human computer interaction [13], and trust and intention to use systems [14].”
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“…the explanation (5) Express risk as whole-number percentages to reduce confusion Prior research has shown that certain cognitive biases can be amplified by specific explanation styles and designs [43] and that providing too much detail may reduce user performance due to information overload [44].”
Research works printed but not linked: 1
- Veritas Health Innovation Ltd, 2025, Covidence systematic review software
Cited by
From Explanations to Reflective Questions in Human-AI Decision-Making
Fischer, S. W. S., Holmberg, L., Thill, S., & Schraffenberger, H. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3831391
“Case information and the DSS predictions must be analysed and evaluated, diferent factors must be weighed up against one another, taking into account possible alternatives to the DSS prediction, as well as the consequences of the decision [36, 52, 53].”
Moghadam, S. T., & Jabin, M. S. R. (2026) · doi:10.2196/preprints.111956
Bearden, D. J. (2026) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2026.111295
Ramu, A., & D.A., T. (2026) · Научный журнал "Вестник Актюбинского регионального университета имени К Жубанова" · doi:10.70239/arsu.2026.t85.n3.06
Metadata
- Authors
- Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S.
- Affiliations
- Amsterdam Public Health (NL)
- Amsterdam University Medical Centers (NL)
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2026-05-14
- Volume
- 5
- Issue
- 5
- Pages
- e0001417
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC13175488
- OpenAlex
- W7161139886
- Cited by
- 9
- References
- 39
- Citations by year
- 2026: 9
- FWCI
- 30.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- explainable AI, clinical decision support systems, healthcare professionals' perceptions, hospital settings, technology acceptance, explanation design, SHAP, feature importance, clinical workflow integration, overreliance on AI, professional autonomy, trust in AI, cognitive burden, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives. https://app.cochat.ai/papers/10.1371/journal.pdig.0001417
- The paper
- Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S. (2026). Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives. PLOS Digital Health. https://doi.org/10.1371/journal.pdig.0001417
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

