The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 72 of the 128 sources its claims cite were checked for support.
Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis
Open accessAbstract
While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-centered framework to guide more effective and trustworthy XAI-CDSS development. Drawing on a comprehensive literature review, we detail a three-phase framework encompassing user-centered XAI method selection, interface co-design, and iterative evaluation and refinement. We demonstrate its application through a retrospective case study analysis of a published XAI-CDSS for sepsis care. Our synthesis highlights the importance of aligning XAI with clinical workflows, supporting calibrated trust, and deploying robust evaluation methodologies that capture real-world clinician–AI interaction patterns, such as negotiation. The case analysis shows how the framework can systematically identify and address user-centric gaps, leading to better workflow integration, tailored explanations, and more usable interfaces. We conclude that achieving trustworthy and clinically useful XAI-CDSS requires a fundamentally user-centered approach; our framework offers actionable guidance for creating explainable, usable, and trusted AI systems in healthcare.
Overview
CoChat Summary
TL;DR
This literature review synthesizes XAI for clinical decision support systems and argues that technical XAI often fails to address clinician needs, workflow integration, and evaluation. It proposes a three-phase user-centered framework and illustrates it retrospectively on a published sepsis XAI-CDSS case.
Key findings
- The review identified approximately 1032 records; after removing 274 duplicates, 758 titles/abstracts were screened, 56 full texts reviewed, and 30 core studies were included (13 foundational pre-2021 and 17 recent 2021–2024).
- A persistent CDS–clinician gap arises from three factors: clinicians’ needs and abilities, interface design, and evaluation of XAI methods; only a handful of papers included formal user evaluations of XAI components, and just one study explicitly investigated whether explanations helped users trust or use the model’s output.
- The proposed three-phase framework comprises user-centered XAI method selection, user-centered interface design, and evaluation and iterative refinement.
- In the retrospective sepsis case analysis of Sivaraman et al. (24 clinicians, mixed methods), SHAP explanations increased confidence but not concordance; a “negotiation” interaction pattern was dominant; trust was driven more by external validation than interface features or explanation types; and a mismatch remained between discrete AI recommendations and continuous clinical workflow. Related work found a disconnect between low self-reported trust and high behavioral reliance (WoA) in Laxar et al., and greater clinician trust for example-based over feature-importance explanations in Gomez et al.
Methods
- Literature search from 2010 through December 2024 across PubMed, Scopus, Web of Science, Google Scholar, the ACM Digital Library, and IEEE Xplore, using combined XAI and CDSS keywords; PRISMA-style screening with inclusion criteria for peer-reviewed publications or highly cited preprints focused on XAI in healthcare CDSS.
- Thematic synthesis of the final 30 core studies, focusing primarily on the 17 recent publications (2021–2024), with foundational works integrated for background and contrast.
- The paper proposes a three-phase user-centered framework and applies it retrospectively to a published XAI-CDSS implementation for sepsis care (Sivaraman et al., CHI 2023), contextualized with Laxar et al. and Gomez et al.; the analysis is illustrative, not a formal validation.
Limitations
- The paper states that the literature review may not have captured every relevant study; the case study is a retrospective, post hoc analysis of a single published work and not a prospective validation; the framework is conceptual and Phase 3 evaluation principles remain conceptual; the framework is primarily clinician-centric and not adapted for patient-facing XAI; it relies heavily on end-user involvement, and clinician attitudes vary (e.g., early vs. late believers, automation bias, confirmation bias); the framework requires further empirical validation and ongoing refinement.
- Evident from the text: no new data were created or analyzed, and the case analysis is presented as an illustration of the framework’s analytical utility rather than a real-world prospective test.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
54 Valid · 11 Not valid · 42 Not checked · 88 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“A central challenge in this domain is navigating the well-documented trade-off between model accuracy and inherent interpretability [6].”
What the sources say
- Disputedcites [6] Rudin, 2019, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell. sourceExperimental — accuracy not yet measured
In the section titled “It is a myth that there is necessarily a trade-off between accuracy and interpretability” (page not supplied), Rudin describes the trade-off as a widespread belief, not an established constraint. She says that, particularly with structured data and meaningful features, simpler interpretable models often perform comparably to more complex ones. (LLM, conf=0.94)
Checked by a reasoning model — may be wrong.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“XAI methods, therefore, hold significant potential to enhance the implementation, utility, and adoption of AI in CDSS.”
What the sources say
- Partly supportedcites [8] Arrieta, 2020, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion sourceExperimental — accuracy not yet measured
The indexed Abstract describes explainability as crucial to the practical deployment of AI and says it can help professionals embrace AI in their fields. It does not address clinical decision support systems specifically or establish improved utility in that setting. (LLM, conf=0.83)
- Partly supportedcites [9] Tjoa, 2020, A survey on explainable artificial intelligence (XAI): Toward medical XAI, IEEE Trans. Neural Netw. Learn. Syst. sourceExperimental — accuracy not yet measured
The Abstract (page not provided) says explanations are needed to justify the reliability of medical AI, and §C, “Other Perspectives: Data-Driven” (page not provided), describes interpretable information helping users decide whether to follow a decision-support system. These passages support potential clinical utility, but the paper does not establish that XAI enhances implementation or adoption of AI in CDSS; §D, “Future Directions for Clinicians and Practitioners,” cautions that interpretability remains complementary support requiring human supervision. (LLM, conf=0.80)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Predictions from these ML models are often characterized by a lack of interpretability [5].”
What the sources say
- Supportedcites [5] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. source
The paper’s opening summary (p. 1) says complex algorithms often cannot provide insights into their behavior. The Introduction (p. 1) also describes widespread black-box models and the difficulty of understanding opaque models, supporting the claim that ML predictions often lack interpretability. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, surrogate models may only accurately approximate the original model in specific local regions [22].”
What the sources say
- Supportedcites [22] Magesh, P.R., Myloth, R.D., and Tom, R.J. (2020). An explainable machine learning model for early detection of Parkinson’s disease using LIME on DaTSCAN imagery. Comput. Biol. Med., 126. source
Section 4.2 describes LIME as a local surrogate model trained with greater weight on samples near the instance being explained. It says the learned model must plausibly estimate the original model’s predictions locally, supporting the claim’s qualified statement. (LLM, conf=0.90)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The effectiveness of such complex cognitive work depends on the characteristics of both the user and the CDSS, as well as the strength of the coupling between them.”
What the sources say
- Partly supportedcites [40] Sedig, 2017, Aligning information technologies with evidence-based health-care activities: A design and evaluation framework, Hum. Technol. Interdiscip. J. Humans Ict Environ. sourceExperimental — accuracy not yet measured
The joint cognitive-systems section says cognitive work emerges from the interaction of the stakeholder and health IT, and “The Mental Space” section links stronger coupling to better support for evidence-based health-care activities. The paper frames this as a principle about health IT generally, rather than making the claim specifically about CDSSs. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
59 more claims · 26 with their cited sources checked
Create a free account to see all 195 claims and to verify the 27 references still open.
- Probably not a substantial claim. 93
- It may state two separate claims as one. 69
- Borderline: one of the tests for a substantial claim is unclear. 1
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 3×
“Ultimately, the goal is to enable users to understand, and consequently trust, the models and their outputs [8,9].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 6×
“There are two general categories of XAI methods: ante hoc and post hoc [8,9,11,12].”
InfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · cited 2×
“Much of the literature still speaks in subjective terms of ‘good’ or ‘quality’ explanations [73], often without measuring how well those explanations integrate into clinicians’ decision-making processes.”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 1×
“These include propagation-based techniques like LRP, which backpropagate a quantity of interest to determine input contributions [30–32], and activation maximization, which identifies input patterns that maximally activate specific neurons [33].”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · cited 2×
“Ultimately, the goal is to enable users to understand, and consequently trust, the models and their outputs [8,9].”
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · cited 1×
“Adversarial examples are a related concept: these are small, often imperceptible perturbations to inputs designed to intentionally cause misclassification, thereby revealing model vulnerabilities or sensitivities [37].”
Zhang, Y., & Wallace, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03820 · Cited by 1,593
Explaining machine learning classifiers through diverse counterfactual explanations
Mothilal, R. K., Sharma, A., & Tan, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372850 · Cited by 1,233 · cited 1×
“…typical model behavior, possibly supplemented by ‘criticisms’ that represent atypical cases [34]. ounterfac-tual explanations identify minimal changes to an input instance’s features that would alter the model’s prediction outcome, helping users understand the model’s decision boundary [35,36].”
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“…which uses game theory to assign contribution values to features for local explanations or approximate global importance [26,27]; and saliency maps, often used with deep learning models on images or text, which visually highlight the input regions most influential for a given output [28,29].”
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 2×
“We adapt the categorization used by Adadi and Berrada [11] and Guidotti et al. [12] (refer to Figure 1).”
Tickle, A. B., Andrews, R., Golea, M., & Diederich, J. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.728352 · Cited by 438
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326 · cited 1×
“…evolution toward more rigorous, multifaceted trust assessments in recent work [53,63–65] stands in contrast to many earlier XAI studies [11,45,46,66,67], which often demonstrated technical feasibility and interpretability but did not empirically evaluate user trust or integration into real…”
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318
Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024) · JMIR AI · doi:10.2196/53207 · Cited by 305 · cited 4×
“Moreover, fostering trust is not an unqualified good–excessive or unfounded trust (over-trust) in AI is risky [59].”
InfluentialMagesh, P. R., Myloth, R. D., & Tom, R. J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104041 · Cited by 295 · cited 1×
“However, surrogate models may only accurately approximate the original model in specific local regions [22].”
Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294 · cited 3×
“This critique builds on limitations that were often implicit in early XAI studies [38,45,46,66,68], where evaluations prioritized technical feasibility or proof-of-concept over robust behavioral assessment.”
Böhle, M., Eitel, F., Weygandt, M., & Ritter, K. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00194 · Cited by 291 · cited 2×
“While early explanation techniques (e.g., LRP [48,49]) provided intuitive outputs, recognition of the need to validate those explanations’ utility underscores the challenge of ensuring explanations are both faithful to the model and relevant to the user’s task.”
Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach
Lamy, J.-B., Sekar, B., Guezennec, G., Bouaud, J., & Séroussi, B. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.01.001 · Cited by 282
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251
Prototype selection for interpretable classification
Bien, J., & Tibshirani, R. (2011) · The Annals of Applied Statistics · doi:10.1214/11-aoas495 · Cited by 245 · cited 1×
“…instances (‘prototypes’) from the dataset to exemplify typical model behavior, possibly supplemented by ‘criticisms’ that represent atypical cases [34]. ounterfac-tual explanations identify minimal changes to an input instance’s features that would alter the model’s prediction outcome, helping…”
Rulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243 · cited 4×
“This critique builds on limitations that were often implicit in early XAI studies [38,45,46,66,68], where evaluations prioritized technical feasibility or proof-of-concept over robust behavioral assessment.”
InfluentialMachine Learning for Clinical Outcome Prediction
Shamout, F., Zhu, T., & Clifton, D. A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3007816 · Cited by 239
Abbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236 · cited 5×
“For image-based XAI, the key challenges include managing visual confirmation bias, interpreting spatial ambiguity (as a heatmap shows *where* but not necessarily *why*), and ensuring the visual explanation is a faithful representation of the model’s logic [82].”
InfluentialFundamentals of Clinical Data Science
(2019) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 235
Sharma, S., Henderson, J., & Ghosh, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375812 · Cited by 226
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278725 · Cited by 211
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207 · cited 2×
“For instance, Sáez et al. [71] introduced a ‘Resilient AI Framework’ focused on robustness in AI for health, and Nasarian et al. [72] proposed a ‘Responsible Clinician–AI Collaboration Framework.’”
Overview of Clinical Decision Support Systems
Berner, E. S., & Lande, T. J. L. (2016) · Health informatics · doi:10.1007/978-3-319-31913-1_1 · Cited by 206
Clinical Decision Support Systems
(2016) · Health Informatics · doi:10.1007/978-3-319-31913-1 · Cited by 196
Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187 · cited 1×
“Building on earlier efforts that compared different model outputs [70] or validated the intuitiveness of visual explanations (like those from LRP) [48,49], recent studies have begun to experimentally test the impact of different explanation strategies on user performance and reliance.”
Distilling Knowledge from Deep Networks with Applications to Healthcare DomainPreprint
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.03542 · Cited by 175
Western University, Sedig, K., Parsons, P., & Western University (2013) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00055 · Cited by 145
Self-citationSivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141 · cited 8×
“We selected the study ‘Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care’ by Sivaraman et al. (CHI 2023) [62] as our primary case for the following reasons: 1.”
InfluentialArtificial Intelligence in Clinical Decision Support: a Focused Literature Survey
Stefania Montani, B., & Striani, M. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677911 · Cited by 139
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127 · cited 3×
“Related concepts such as co-design and participatory frameworks for AI development [56], new behavioral evaluation tools (e.g., Zeno [69]), and emerging ethical and governance principles for XAI [11] also enrich this evolving landscape.”
Rule Extraction Algorithm for Deep Neural Networks: A ReviewPreprint
Hailesilassie, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05267 · Cited by 122
XAI-Based Clinical Decision Support Systems: A Systematic Review
Kim, S. Y., Kim, D. H., Kim, M., Ko, H.-J., & Jeong, O. (2024) · Applied Sciences · doi:10.3390/app14156638 · Cited by 108
Explainability in medicine in an era of AI-based clinical decision support systems
Pierce, R. L., Van Biesen, W., Van Cauwenberge, D., Decruyenaere, J., & Sterckx, S. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.903600 · Cited by 95 · cited 4×
“There is now a growing call for more robust, behaviorally grounded, and human-centered evaluation methodologies [53].”
InfluentialExplaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks
Yang, Y., Volker Tresp, C., Wunderle, M., & Fasching, P. A. (2018) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2018.00025 · Cited by 88 · cited 1×
“…[43], explaining inferences in Bayesian networks [44], deriving rule-based explanations from predictive models [45], employing mimic learning (training simpler models to replicate complex model behavior) [46], and applying post hoc methods such as LIME and LRP for specific use cases [47–49].”
Explaining Machine Learning Models for Clinical Gait Analysis
Slijepcevic, D., Horst, F., Lapuschkin, S., Horsak, B., Raberger, A.-M., Kranzl, A., Samek, W., Breiteneder, C., Schöllhorn, W. I., & Zeppelzauer, M. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3474121 · Cited by 82
Active Learning-Based Pedagogical Rule Extraction
de Fortuny, E. J., & Martens, D. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2389037 · Cited by 74
XAI-based Clinical Decision Support System: A Systematic Review
Kim, S., Kim, D., Kim, M., Ko, H., & Jeong, O. (2024) · doi:10.20944/preprints202406.0721.v1 · Cited by 72
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model DistillationPreprint
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06169 · Cited by 66
Zytek, A., Liu, D., Vaithianathan, R., & Veeramachaneni, K. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114864 · Cited by 62 · cited 3×
“For structured data XAI, challenges include managing cognitive overload from long lists of features and assessing the plausibility of abstract feature importance scores, which may be statistically powerful but counter-intuitive to a clinician [83].”
InfluentialOutlining the Design Space of Explainable Intelligent Systems for Medical DiagnosisPreprint
Xie, Y., Gao, G., & Chen, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06019 · Cited by 62
Health Care Professionals’ Experience of Using AI: Systematic Review With Narrative Synthesis
Ayorinde, A., Mensah, D. O., Walsh, J., Ghosh, I., Ibrahim, S. A., Hogg, J., Peek, N., & Griffiths, F. (2024) · Journal of Medical Internet Research · doi:10.2196/55766 · Cited by 55 · cited 3×
“The ‘black-box’ nature of many AI systems remains a significant hurdle, limiting transparency and hindering trust among clinicians who require understandable justifications for AI-driven recommendations, especially in high-stakes decisions [51].”
InfluentialInterpreting the Predictions of Complex ML Models by Layer-wise Relevance PropagationPreprint
Samek, W., Montavon, G., Binder, A., Lapuschkin, S., & Müller, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08191 · Cited by 54
Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51 · cited 3×
“Related concepts such as co-design and participatory frameworks for AI development [56], new behavioral evaluation tools (e.g., Zeno [69]), and emerging ethical and governance principles for XAI [11] also enrich this evolving landscape.”
Parsons, P., & Sedig, K. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23002 · Cited by 49 · cited 1×
“These properties include appearance, complexity, dynamism, fidelity, fragmentation, interiority, scope, and type, as outlined by Parsons and Sedig [78].”
Self-citationHuman-Centered Interactivity of Visualization Tools: Micro- and Macro-level Considerations
Sedig, K., Parsons, P., Dittmer, M., & Haworth, R. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7485-2_29 · Cited by 46 · cited 1×
“Interactivity in the interface can be considered at two levels: micro and macro [79].”
Self-citationInterpreting Deep Classifier by Visual Distillation of Dark KnowledgePreprint
Xu, K., Park, D. H., Yi, C., & Sutton, C. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04042 · Cited by 43
Micocci, M., Borsci, S., Thakerar, V., Walne, S., Manshadi, Y., Edridge, F., Mullarkey, D., Buckle, P., & Hanna, G. B. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10143101 · Cited by 39 · cited 4×
“The field grapples with dual risks: on one hand, passive over-reliance on AI (taking AI suggestions without sufficient scrutiny) [58], and on the other, under-reliance or excessive skepticism of AI (dismissal of useful AI input) [60].”
InfluentialEvaluating the impact of human explanation strategies on human-AI visual decision-making
Morrison, K., Shin, D., Holstein, K., & Perer, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579481 · Cited by 37
Sáez, C., Ferri, P., & García-Gómez, J. M. (2024) · Journal of Medical Internet Research · doi:10.2196/50295 · Cited by 36
Bunn, J. (2020) · Records Management Journal · doi:10.1108/rmj-08-2019-0038 · Cited by 33 · cited 1×
“While current XAI methods show promise in making systems more accountable and transparent [10], effectively communicating explanations to users via the system interface remains a major obstacle to integrating ML into clinical practice.”
Kyrimi, E., Mossadegh, S., Tai, N. R. M., & Marsh, W. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101812 · Cited by 32 · cited 2×
“While some earlier work tried to link specific explanation types to perceived trustworthiness (e.g., rule-based vs. example-based explanations) [44], recent studies confirm that the relationship is complex [59].”
User Studies in Visualization: A Reflection on Methods
Tory, M. K. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7485-2_16 · Cited by 31 · cited 1×
“This means prioritizing cognitive elements such as perception, reasoning, and insight—each integral to effective human–AI interaction [77].”
Explainable AI decision support improves accuracy during telehealth strep throat screening
Gomez, C., Smith, B.-L., Zayas, A., Unberath, M., & Canares, T. (2024) · Communications Medicine · doi:10.1038/s43856-024-00568-x · Cited by 30 · cited 7×
“The type of explanation provided matters significantly: for instance, Gomez et al. [60] found that in their telehealth application, example-based explanations garnered more trust from clinicians than feature-importance explanations, indicating the importance of selecting explanation modalities that…”
InfluentialUnveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Aziz, N. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W. (2024) · doi:10.1101/2024.08.10.24311735 · Cited by 30
Freitas, C. M. D. S., Pimenta, M. S., & Scapin, D. L. (2014) · Handbook of Human Centric Visualization · doi:10.1007/978-1-4614-7485-2_12 · Cited by 28 · cited 1×
“Despite the existence of comprehensive evaluation guidelines (e.g., those by Freitas et al. [81] for visualization), significant challenges remain in conducting truly user-centered usability assessments for XAI-CDSS.”
Parsons, P., & Sedig, K. (2014) · Handbook of Human Centric Visualization · doi:10.1007/978-1-4614-7485-2_28 · Cited by 27 · cited 2×
“In this joint system, cognitive activities result from a coupling between the clinician’s cognitive processes and the CDSS’s external representations of information [41].”
Self-citationLamy, J.-B., Sedki, K., & Tsopra, R. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103407 · Cited by 25 · cited 1×
“…foundational studies explored a wide range of interpretability techniques, including visualizing ontology-based preferences for decision support [42], using visual case-based reasoning in oncology [38], developing inherently interpretable models like RETAIN for electronic health records [43],…”
A Rule Extraction Study from SVM on Sentiment Analysis
Bologna, G., & Hayashi, Y. (2018) · Big Data and Cognitive Computing · doi:10.3390/bdcc2010006 · Cited by 20
Laxar, D., Eitenberger, M., Maleczek, M., Kaider, A., Hammerle, F. P., & Kimberger, O. (2023) · BMC Medicine · doi:10.1186/s12916-023-03068-2 · Cited by 19 · cited 5×
“Transparency of the underlying model was also a concern (echoed in Laxar et al. [61]).”
InfluentialInternet of Things and Bigdata Analytics in Preventive Healthcare: A Synthetic Review
Šajnović, U., Vošner, H. B., Završnik, J., Žlahtič, B., & Kokol, P. (2024) · doi:10.20944/preprints202408.0445.v1 · Cited by 19
Internet of Things and Big Data Analytics in Preventive Healthcare: A Synthetic Review
(2024) · doi:10.3390/electronics13183642 · Cited by 19
Turri, V., Morrison, K., Robinson, K.-M., Abidi, C., Perer, A., Forlizzi, J., & Dzombak, R. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658985 · Cited by 10 · cited 2×
“Turri et al. [57] found that achieving meaningful transparency in a deployed AI system required addressing nuanced user needs and practical limitations far beyond implementing a basic XAI technique.”
Morrison, K., Jain, M., Hammer, J., & Perer, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610064 · Cited by 10
Sedig, K., Naimi, A., & Haggerty, N. (2017) · Human Technology · doi:10.17011/ht/urn.201711104211 · Cited by 9
Self-citationRule extraction algorithm from support vector machines and its application to credit screening
Su, C., & Chen, Y.-C. (2011) · Soft Computing · doi:10.1007/s00500-011-0762-8 · Cited by 8
Evaluating Visualization Environments: Cognitive, Social, and Cultural Perspectives
Hundhausen, C. D. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7485-2_5 · Cited by 6 · cited 2×
“The evaluation process must be contextualized to the specific environment and tasks in which the XAI system operates, as highlighted by Hundhausen [80].”
A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems
Gambetti, A., Han, Q., Shen, H., & Soares, C. (2025) · arXiv · doi:10.48550/arxiv.2502.09849 · Cited by 4
Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support
Sivaraman, V., Morrison, K., Epperson, W., & Perer, A. (2025) · arXiv · doi:10.48550/arxiv.2504.07423 · Cited by 1
Maximally Invariant Data Perturbation as Explanation
Hara, S., Ikeno, K., Soma, T., & Maehara, T. (2018) · arXiv · doi:10.48550/arxiv.1806.07004
Research works printed but not linked: 9
- Hooker, S., Erhan, D., Kindermans, P.J., and Kim, B. (2019). A benchmark for interpretability methods in deep neural networks. Adv. Neural Inf. Process. Syst., 32.
- Lundberg, S.M., and Lee, S.I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30.
- Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
- Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst., 29.
- Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., and Stewart, W. (2016). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Adv. Neural Inf. Process. Syst., 29.
- Che, Z., Purushotham, S., Khemani, R., and Liu, Y. (2017, January 4–8). Interpretable deep models for ICU outcome prediction. Proceedings of the AMIA Annual Symposium Proceedings, Washington, DC, USA.
- Katuwal, G.J., and Chen, R. (2016). Machine learning model interpretability for precision medicine. arXiv.
- Sedig, K., and Liang, H.N. (2007, January 25–29). On the design of interactive visual representations: Fitness of interaction. Proceedings of the EdMedia+ Innovate Learning, Vancouver, BC, Canada.
- Gadanidis, 2004, Designing online mathematical investigation, J. Comput. Math. Sci. Teach., 23, 275
Cited by
EvoMDT: a self-evolving multi-agent system for structured clinical decision-making in multi-cancer
Liu, Q., Hu, Z., Huang, T., Niu, Y., Zhang, X., Ma, S., Lin, C., Goh, K. H., Kwon, H. E., Gao, F., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02304-8 · Cited by 31
“…trail;futuredeploymentswillintegratefull-textnotesandimagingreports via controlled parsers with human-in-the-loop veri fi cation 25 single-blind physician ratings and low safety-violation rates complement benchmark gains, aligning with CONSORT-AI/DECIDE-AI emphases on…”
Artificial Intelligence for Predicting Difficult Airways: A Review
Alatau, M., Bauer, J., & Sazonov, V. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14238600 · Cited by 12
“A growing body of literature emphasizes the importance of explainability in clinical decision support systems (CDSSs) beyond raw predictive performance [18].”
Artificial Intelligence-Based Models for Predicting Disease Course Risk Using Patient Data
Chowdhury, R. I., Bari, W., Hasan, M. T., Hossain, Z., & Rahman, M. (2026) · Computers · doi:10.3390/computers15020113 · Cited by 2
“ML (a subfield of AI) can learn from patients’ past follow-up data and recognize useful patterns [71,72].”
Sun, T., Liu, S., Qian, K., & Li, S. (2026) · Journal of Materials Chemistry B · doi:10.1039/d6tb00696e · Cited by 4
Ramos, F., dos Santos, A. M. P., Rocha, T., Petronilho, F. A. S., & Pereira, R. P. G. (2026) · Systems · doi:10.3390/systems14070764 · Cited by 3
“Recent evidence reinforces this gap, indicating that current explain-able AI approaches in clinical decision support still face unresolved challenges related to integration, usability, and real-world applicability [4].”
Metadata
- Authors
- Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T.
- Affiliations
- Western University (CA)
- Corresponding author
- Sedig, Kamran
- Venue
- Informatics
- Publisher
- MDPI AG
- Published
- 2025-10-28
- Volume
- 12
- Issue
- 4
- Pages
- 119
- ISSN
- 2227-9709 (ISSN-L)
- OpenAlex
- W4415620636
- Cited by
- 39
- References
- 64
- Citations by year
- 2026: 38 · 2025: 1
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Informatics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- explainable AI, clinical decision support systems, model interpretability, clinician trust in AI, user-centered design, clinical workflow integration, explanation evaluation, sepsis care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.