Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in “black box” models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the “user”. Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders’ needs, ensuring better implementation and operation in a healthcare context. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Seeking Qualitative Rigor in Inductive Research: Notes on the Gioia Methodology
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013) · Organizational Research Methods · doi:10.1177/1094428112452151 · Cited by 13,227
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 4×
“Here, ‘transparency’ is mainly used in computer science literature to describe different frameworks or methods to simplify complex models into binary models, rule generators or the like [1], [5], [42], [45]–[50].”
BackgroundMethodInfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
An interactive web-based dashboard to track COVID-19 in real time
Dong, E., Du, H., & Gardner, L. (2020) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(20)30120-1 · Cited by 10,048
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Projects such as “Watson for Oncology” [19], an unnamed, broadly used algorithm to manage the health of large populations [20], Google Health [21] which can identify signs of diabetic retinopathy, or COMPAS [22] a decision support tool used in court houses to assess risk of recidivism, have all failed one way or anoth…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909 · cited 1×
“In particular, they have utilized the Gradient-weighted Class Activation Mapping (Grad-CAM) method [70], originally a debugging tool for DNN, to visualize the output of the model with a heatmap overlay.”
Method“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“xAI refers to methods and techniques that seek to provide insights into the outcome of a ML model and present it in qualitative, understandable terms or visualizations to the stakeholders of the model [4], [6], [24], [43].”
MethodExplanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338
Cited by
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Jung, J., Lee, H., Jung, H., & Kim, H. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16110 · Cited by 87
Kim, M., Kim, S., Kim, J., Song, T.-J., & Kim, Y. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103160 · Cited by 53
Examining explainable clinical decision support systems with think aloud protocols
Anjara, S. G., Janik, A., Dunford-Stenger, A., Mc Kenzie, K., Collazo-Lorduy, A., Torrente, M., Costabello, L., & Provencio, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0291443 · Cited by 28
“In response to black-box ML, the field of explainable AI emerged with the aim of “providing explanations catered to human understanding, trust, and transparency” [8].”
Background
Metadata
- Authors
- Gerlings, J., Jensen, M. S., & Shollo, A.
- Venue
- Intelligent Systems Reference Library
- Publisher
- Springer International Publishing
- Published
- 2021-11-27
- Pages
- 169-198
- ISSN
- 1868-4394 (ISSN-L)
- 1868-4408
- OpenAlex
- W3216202622
- Cited by
- 68
- References
- 39
- FWCI
- 13.68
- Type
- book chapter
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare. https://app.cochat.ai/papers/10.1007/978-3-030-83620-7_7
- The paper
- Gerlings, J., Jensen, M. S., & Shollo, A. (2022). Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare. Intelligent Systems Reference Library. https://doi.org/10.1007/978-3-030-83620-7_7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.