Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index; 215 of the 303 sources its claims cite were checked for support.
Abstract
Explainable Artificial Intelligence (XAI) has the potential to enhance clinical decision support (CDS) systems however, it remains unclear how XAI systems are perceived by healthcare professionals in hospital settings, and if new challenges arise as a result of explanations. This scoping review aimed to understand healthcare professionals’ perceptions of CDS systems with XAI in the hospital setting; specifically, the drivers of acceptance and use, explainability needs, and design preferences. Databases were searched. Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems used in hospital settings. MEDLINE, Embase, and Web of Science were searched, and reference lists were screened for additional papers. Study characteristics and health professional perceptions were extracted and inductively coded. A quality assessment was performed using the CASP checklist. Sixteen studies were identified. Included studies primarily focused on ML-based CDS systems for predicting various clinical outcomes. Most studies used feature importance or model agnostic techniques like SHapley Additive exPlanations (SHAP). Overall, healthcare professionals perceived CDS systems with XAI as useful for supporting clinical tasks, decision-making, and teamwork. Acceptance was influenced by integration into workflows, performance, data quality, and alignment with clinical knowledge. Concerns were raised about overreliance and reduced professional autonomy. Health professionals predominantly used explanations to validate outputs, and desired actionable information from systems. SHAP plots and visualizations were difficult to interpret. Participants preferred explanation designs that included concise, high-level information, and simple plots for quick interpretation. Clear visual indicators such as colour coding, contextual patient data, and aggregating similar features also aided interpretation. Poorly designed XAI explanations can hinder understanding and increase cognitive burden in busy clinical settings. Future research should optimise the design and delivery of explanations, so clinicians can appropriately trust in XAI CDS systems and feel confident in their clinical decision making.
Overview
CoChat Summary
TL;DR
This scoping review synthesized 16 qualitative studies on healthcare professionals’ perceptions of XAI-enabled clinical decision support in hospitals, finding general perceived usefulness but acceptance tied to workflow integration, performance, data quality, and alignment with clinical knowledge. SHAP plots and values were often difficult to interpret, and clinicians preferred concise, actionable, simple explanations with contextual patient data.
Key findings
- Across 16 studies and 338 extracted qualitative results, XAI CDS systems were perceived as useful for task completion, clinical decision-making, and teamwork; usefulness was user- or context-dependent, such as for less experienced doctors, night shifts, incomplete patient information, or patients unable to communicate.
- Trust and acceptance were influenced by workflow/EHR integration, model performance, data quality, evidence, and alignment with clinical knowledge; concerns included overreliance, reduced professional autonomy, and deskilling.
- Explanations were mainly used to validate outputs, and participants wanted actionable information, modifiable variables, and next steps. SHAP was used in five studies, with four reporting difficulty understanding SHAP plots or values; LIME was used in three studies, and example-based explanations in two.
- Design preferences included high-level information with details on demand, colour coding, simple plots such as bar charts and line graphs, features grouped and ordered by contribution, contextual patient data, and whole-number percentages; decimals, negative values, and outputs not summing to 100% were confusing.
Methods
- Scoping review following PRISMA-ScR; MEDLINE, Embase, and Web of Science were searched from database inception to 15 June 2024, with reference lists screened; the review protocol was not pre-registered.
- Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems in hospital settings; reviews, commentaries, conference abstracts, non-English studies, community/primary care settings, and perceptions collected exclusively before system development were excluded.
- Data were extracted by at least two researchers and inductively coded into themes covering the CDS system with XAI in general, the XAI techniques, and design/usability; quality was assessed with the CASP qualitative checklist.
Limitations
- The authors state that three relevant studies were found only through reference-list screening, not the database search, due to inconsistent terminology, so the search may have missed other relevant studies and limited comprehensiveness.
- The authors state that design recommendations are preliminary and require higher-level evidence; no studies were excluded or weighted by CASP quality, and some included studies reported explainability or user perceptions only as secondary outcomes, limiting depth.
- Evident from the text: the review protocol was not pre-registered, and most included systems remained at prototype or pre-implementation stages, so post-implementation perceptions are limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
80 Valid · 6 Not valid · 17 Not checked · 56 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“while two studies described their systems using general ML terminology without referencing explainability or interpretability [23,30].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Discussion
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Disputedcites [30] KE Henry, 2022, Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system, NPJ Digit Med sourceExperimental — accuracy not yet measured
The paper describes its system in ML terms, but it also explicitly discusses interpretability of the model’s computations and feature-importance measures. That contradicts the claim that this study did not reference interpretability. The indexed copy provides no page number for the passage. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Colour coding was viewed as a helpful visual cue for conveying risk severity.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Not checked yetcites [28] BA Fritz, 2024, User-centered design of a machine learning dashboard for prediction of postoperative complications, Anesth Analg source
Checked before — its earlier verdict is held back until a new check runs.
- Supportedcites [31] S Jauk, 2021, Technology acceptance of a machine learning algorithm predicting delirium in a clinical setting: a mixed-methods study, J Med Syst source
In Results > Expert group meetings, high-risk patients are shown with a yellow symbol and very-high-risk patients with a red symbol; the experts appreciated this presentation, including its “traffic light symbol.” The Discussion likewise says users appreciated the yellow and red icons in the hospital interface. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For presentation of features, healthcare professionals preferred grouping features and ordering them by their contribution to predicted risk to allow easy and quick interpretation.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Partly supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak sourceExperimental — accuracy not yet measured
In Results > Explanation design preferences (page not available), providers preferred grouping related features to reduce information-processing effort. The Discussion says participants valued seeing which risk factors contributed most to predicted risk. Neither passage establishes a preference for ordering features by their contribution, though the proposed display shows each feature’s contribution. (LLM, conf=0.84)
- Not checked yetcites [28] BA Fritz, 2024, User-centered design of a machine learning dashboard for prediction of postoperative complications, Anesth Analg source
Checked before — its earlier verdict is held back until a new check runs.
- Partly supportedcites [24] SG Anjara, 2023, Examining explainable clinical decision support systems with think aloud protocols, PLoS One sourceExperimental — accuracy not yet measured
In Results, §3 (page unavailable), participants criticized the lack of priority ordering and the unintuitive display of attributes contributing to the relapse score. This supports ordering features by importance, but the paper does not report a preference for grouping them or say that this would enable quick interpretation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Health professionals valued the presentation of contextual information such as patient results alongside predictions and explanations”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Not checked — no full text of the sourcecites [23] J Abraham, 2023, Integrating machine learning predictions for perioperative risk management: towards an empirical design of a flexible-standardized risk assessment tool, J Biomed Inform source
- Supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak source
The Results (Explanation design preferences) say providers found raw patient values and time-series plots helpful for interpreting predictions and explanations. The Discussion likewise says laboratory values, vital signs, and interventions provided useful context for assessing their clinical credibility and utility. (LLM, conf=0.94)
- Supportedcites [27] N Bienefeld, 2023, Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals, NPJ Digit Med source
In the Interview results passage (labeled Abstract in our copy), clinicians say they needed to check whether a prediction made clinical sense by considering CT scans and patient outcomes. The paper also says they consulted patient history, laboratory values, diagnostic tests, and imaging in parallel with the prediction system, and recommends integrating that context into future designs. (LLM, conf=0.88)
- Supportedcites [32] Z Jin, 2020, CarePre, ACM Trans Comput Healthc source
The evaluation sections record exactly this: in the case study (B. Case Study, p. 7) the physician found the similar-patient view 'very informative' and said it gave him more confidence in the prediction results because 'it provides specific evidence [to support the prediction results],' and in the expert interview (C. Domain Expert Interview, p. 8) E1 stated that 'comparing to the similar patients in detail not only gives me more confidence of the prediction results but also provides me with rich treatment examples,' while E2 called the records of similar patients 'an important reference.' This is grounded in the clinicians' stated requirement from the pilot study (III. Pilot Study, p. 3, R2/R3) that prediction results be illustrated within the context of the patient's historical medical record and comparable similar patients to verify and interpret them. The claim's example 'patient results' is a loose rendering of the patient's/similar patients' medical-record outcomes rather than lab test results per se, but the attributed point holds. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Flexibility in presentation formats was also valued, enabling alignment with individual user preferences.”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
What the sources say
- Supportedcites [25] AJ Barda, 2020, A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare, BMC Med Inform Decis Mak source
In Results > Explanation design preferences, the paper reports differing preferences across clinical roles and individuals, and says providers wanted control over how they organize and view explanation content. Table 3 likewise lists interactive options for different displays and organizations for various users. These passages support the paraphrase that flexible presentation was valued to accommodate user preferences. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
89 more claims · 52 with their cited sources checked
Create a free account to see all 159 claims and to verify the 54 references still open.
- It may state two separate claims as one. 44
- Probably not a substantial claim. 31
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“| Bienefeld(2023) [27] | Switzerland | University hospital (Neurosurgical intensive care) | Predict delayed cerebral ischemia in patients with aneurysmal subarachnoid haemorrhage | Prototype testing (low and high fidelity) | No | SHAP |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Study characteristics
“| Pinto(2023) [34] | Portugal | Epilepsy refractory center, hospital | Prediction for EEG seizures | Prototype testing | No |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
““What you want are modifiable things... History of hypertension? You’re not going to be able to do anything about that” [21].”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > References
“21. Veritas Health Innovation Ltd. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation Ltd; 2025.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
“participants in three studies reported wanting to see the variables that could be altered through interventions”
§ Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives > Results > Results from synthesis of user perceptions
“[23,25,28,32]”
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565 · cited 1×
“The PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guidelines [20] were followed in the reporting of this review and the checklist is included in the Supporting Information (S1 PRISMA Checklist).”
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · cited 1×
“A scoping review was conducted as XAI is a new field, and this method allowed for a broad mapping of existing literature and identification of key concepts related to health professionals’ perceptions of XAI in hospital settings [19].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Explainable Artificial Intelligence (XAI) aims to enhance transparency, trustwor-thiness, and accountability of AI systems by providing explanations for their decisions [4,5].”
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
Haenlein, M., & Kaplan, A. (2019) · California Management Review · doi:10.1177/0008125619864925 · Cited by 2,655
Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., Spurr, C., Khorasani, R., Tanasijevic, M., & Middleton, B. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1370 · Cited by 1,346
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME
Salih, A. M., Raisi‐Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400304 · Cited by 897 · cited 3×
“Another commonly used method is Local Interpretable Model-agnostic Explanations (LIME), which approximates the predictions of a complex model locally using a simpler, interpretable model to explain individual predictions [9].”
InfluentialLoh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734 · cited 2×
“Previous reviews have explored XAI in healthcare, with a focus on explainability techniques [11,12], human computer interaction [13], and trust and intention to use systems [14].”
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“…the explanation (5) Express risk as whole-number percentages to reduce confusion Prior research has shown that certain cognitive biases can be amplified by specific explanation styles and designs [43] and that providing too much detail may reduce user performance due to information overload [44].”
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Ali, O., Abdelbaki, W., Shrestha, A., Elbasi, E., Alryalat, M. A. A., & Dwivedi, Y. K. (2023) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2023.100333 · Cited by 568
Barriers to and facilitators of artificial intelligence adoption in health care: scoping review
Hassan, M., Kushniruk, A., & Borycki, E. (2024) · JMIR Human Factors · doi:10.2196/48633 · Cited by 435 · cited 1×
“Many of these themes are long accepted requirements for CDS as outlined by Bates et al. in 2003 [39], and are not unique to XAI or hospital settings as described in a review of AI adoption in healthcare [40].”
A., S., & R., S. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100230 · Cited by 432
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312 · cited 1×
“Previous reviews have explored XAI in healthcare, with a focus on explainability techniques [11,12], human computer interaction [13], and trust and intention to use systems [14].”
Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024) · JMIR AI · doi:10.2196/53207 · Cited by 305 · cited 2×
“Previous reviews have explored XAI in healthcare, with a focus on explainability techniques [11,12], human computer interaction [13], and trust and intention to use systems [14].”
Perceptions and Needs of Artificial Intelligence in Health Care to Increase Adoption: Scoping Review
Chew, H. S. J., & Achananuparp, P. (2022) · Journal of Medical Internet Research · doi:10.2196/32939 · Cited by 243
Henry, K. E., Kornfield, R., Sridharan, A., Linton, R. C., Groh, C., Wang, T., Wu, A., Mutlu, B., & Saria, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00597-7 · Cited by 207 · cited 10×
“Participants in Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”
InfluentialA Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 203
Solving the Explainable AI Conundrum by Bridging Clinicians’ Needs and Developers’ Goals
Bienefeld, N., Boss, J. M., Lüthy, R., Brodbeck, D., Azzati, J., Blaser, M., Willms, J., & Keller, E. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00837-4 · Cited by 159
How to explain AI systems to end users: a systematic literature review and research agenda
Laato, S., Tiainen, M., Najmul Islam, A. K. M., & Mäntymäki, M. (2022) · Internet Research · doi:10.1108/intr-08-2021-0600 · Cited by 155 · cited 1×
“Previous reviews of XAI have also reported that explain-ability is sensitive to contextual factors and future research should investigate different groups of end users [15,16].”
Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey
Kök, İ., Okay, F. Y., Muyanlı, Ö., & Özdemir, S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3287678 · Cited by 151
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130 · cited 1×
“Previous reviews of XAI have also reported that explain-ability is sensitive to contextual factors and future research should investigate different groups of end users [15,16].”
How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129 · cited 1×
“…the explanation (5) Express risk as whole-number percentages to reduce confusion Prior research has shown that certain cognitive biases can be amplified by specific explanation styles and designs [43] and that providing too much detail may reduce user performance due to information overload [44].”
Barda, A. J., Horvat, C. M., & Hochheiser, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01276-x · Cited by 105 · cited 10×
“…or tabs to offer additional information , so clinicians could choose how much information Use simple to interpret plots to understand at a glance what features impacted the prediction [24,25,27,33,34,38] “But this information over here [referring to Shapley values of dynamic contributors, Fig.”
InfluentialSchwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · cited 10×
“Participants in Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”
InfluentialExplainability pitfalls: beyond dark patterns in explainable AI
Ehsan, U., & Riedl, M. O. (2024) · Patterns · doi:10.1016/j.patter.2024.100971 · Cited by 72 · cited 1×
“[33] inappropriate [14,41].”
CarePre: An Intelligent Clinical Decision Assistance System
Jin, Z., Cui, S., Guo, S., Gotz, D., Sun, J., & Cao, N. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3344258 · Cited by 70
Explainable AI (XAI): Explained
Reddy, G. P., & Kumar, Y. V. P. (2023) · IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream) · doi:10.1109/estream59056.2023.10134984 · Cited by 66 · cited 1×
“One involves using inherently interpretable models, such as decision trees, and the other focuses on applying explainability techniques to complex ‘black box’ models to help users understand the model’s process or output [6,7].”
Jauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01727-6 · Cited by 59 · cited 9×
“[28] Use tables and/or bar charts for visualization of risk [23,31] “Tables, grids or graphs, in general, are better [than paragraphs].””
InfluentialAbraham, J., Bartek, B., Meng, A., Ryan King, C., Xue, B., Lu, C., & Avidan, M. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104270 · Cited by 50 · cited 8×
“Participants in Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”
InfluentialMatthiesen, S., Diederichsen, S. Z., Hansen, M. K. H., Villumsen, C., Lassen, M. C. H., Jacobsen, P. K., Risum, N., Winkel, B. G., Philbert, B. T., Svendsen, J. H., et al. (2021) · JMIR Human Factors · doi:10.2196/26964 · Cited by 45 · cited 9×
“Participants in Several studies reported trust and acceptance of model outputs were facilitated by providing evidence for the model effectiveness on patients and/or evidence of the output [23,29,30,33,35,36].”
InfluentialPradelli, L., Risoli, C., Summer, E., Bellini, G., Mozzarelli, F., Anderson, G., Guasconi, M., Artioli, G., Bonacaro, A., & Sarli, L. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-087268 · Cited by 42
Brankovic, A., Cook, D., Rahman, J., Delaforce, A., Li, J., Magrabi, F., Cabitza, F., Coiera, E., & Bradford, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01764-2 · Cited by 37 · cited 1×
“The XAI-evaluation checklist by Brankovic and colleagues [42] recommends assessing, reporting and discussing the actionability of explanations.”
Samhammer, D., Roller, R., Hummel, P., Osmanodja, B., Burchardt, A., Mayrdorfer, M., Duettmann, W., & Dabrock, P. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.1016366 · Cited by 33
Zilker, S., Weinzierl, S., Kraus, M., Zschech, P., & Matzner, M. (2024) · Health Care Management Science · doi:10.1007/s10729-024-09673-8 · Cited by 29 · cited 7×
“…or tabs to offer additional information , so clinicians could choose how much information Use simple to interpret plots to understand at a glance what features impacted the prediction [24,25,27,33,34,38] “But this information over here [referring to Shapley values of dynamic contributors, Fig.”
InfluentialExamining explainable clinical decision support systems with think aloud protocols
Anjara, S. G., Janik, A., Dunford-Stenger, A., Mc Kenzie, K., Collazo-Lorduy, A., Torrente, M., Costabello, L., & Provencio, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0291443 · Cited by 28 · cited 9×
“…or tabs to offer additional information , so clinicians could choose how much information Use simple to interpret plots to understand at a glance what features impacted the prediction [24,25,27,33,34,38] “But this information over here [referring to Shapley values of dynamic contributors, Fig.”
InfluentialThe goal of explaining black boxes in EEG seizure prediction is not to explain models' decisions
Pinto, M. F., Batista, J., Leal, A., Lopes, F., Oliveira, A., Dourado, A., Abuhaiba, S. I., Sales, F., Martins, P., & Teixeira, C. A. (2023) · Epilepsia Open · doi:10.1002/epi4.12748 · Cited by 23
Shulha, M., Hovdebo, J., D’Souza, V., Thibault, F., & Harmouche, R. (2024) · JMIR Formative Research · doi:10.2196/50475 · Cited by 22 · cited 10×
“Local Interpretable Model-Agnostic Explanations (LIME) were also used by models in three studies [33,34,37], however, participants only described its usefulness in one.”
InfluentialHelman, S., Terry, M. A., Pellathy, T., Hravnak, M., George, E., Al-Zaiti, S., & Clermont, G. (2023) · Applied Clinical Informatics · doi:10.1055/s-0043-1775565 · Cited by 20
User-centered design of a machine learning dashboard for prediction of postoperative complications
Fritz, B. A., Pugazenthi, S., Budelier, T. P., Tellor Pennington, B. R., King, C. R., Avidan, M. S., & Abraham, J. (2024) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006577 · Cited by 15 · cited 8×
“[28] Use tables and/or bar charts for visualization of risk [23,31] “Tables, grids or graphs, in general, are better [than paragraphs].””
InfluentialBesculides, M., Mazumdar, M., Phlegar, S., Freeman, R., Wilson, S., Joshi, H., Kia, A., & Gorbenko, K. (2023) · JMIR Formative Research · doi:10.2196/42262 · Cited by 12 · cited 9×
“Perceptions on the model’s performance was reported in 13 studies [23–26 , 29– 37].”
InfluentialThe pace of hospital life: a mixed methods study
Long, J. C., Pomare, C., Ellis, L. A., Churruca, K., & Braithwaite, J. (2021) · PLOS ONE · doi:10.1371/journal.pone.0255775 · Cited by 8 · cited 1×
“Hospital settings are a particularly unique context, characterised by complex organisational structures, multidisciplinary teams, and high-pressure, fast-paced work [17,18].”
Chew, H. S. J., & Achananuparp, P. (2021) · doi:10.2196/preprints.32939 · Cited by 4
Research works printed but not linked: 1
- Veritas Health Innovation Ltd, 2025, Covidence systematic review software
Cited by
From Explanations to Reflective Questions in Human-AI Decision-Making
Fischer, S. W. S., Holmberg, L., Thill, S., & Schraffenberger, H. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3831391
“Case information and the DSS predictions must be analysed and evaluated, diferent factors must be weighed up against one another, taking into account possible alternatives to the DSS prediction, as well as the consequences of the decision [36, 52, 53].”
Moghadam, S. T., & Jabin, M. S. R. (2026) · doi:10.2196/preprints.111956
Bearden, D. J. (2026) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2026.111295
Ramu, A., & D.A., T. (2026) · Научный журнал "Вестник Актюбинского регионального университета имени К Жубанова" · doi:10.70239/arsu.2026.t85.n3.06
Metadata
- Authors
- Van Dort, B. A., Engelsma, T., Sneekes, P., Peute, L., & Medlock, S.
- Affiliations
- Amsterdam Public Health (NL)
- Amsterdam University Medical Centers (NL)
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2026-05-14
- Volume
- 5
- Issue
- 5
- Pages
- e0001417
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC13175488
- OpenAlex
- W7161139886
- Cited by
- 9
- References
- 39
- Citations by year
- 2026: 9
- FWCI
- 30.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- explainable AI, clinical decision support systems, healthcare professionals' perceptions, hospital settings, technology acceptance, explanation design, SHAP, feature importance, clinical workflow integration, overreliance on AI, professional autonomy, trust in AI, cognitive burden, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

