Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in “black box” models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the “user”. Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders’ needs, ensuring better implementation and operation in a healthcare context. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Seeking Qualitative Rigor in Inductive Research: Notes on the Gioia Methodology
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013) · Organizational Research Methods · doi:10.1177/1094428112452151 · Cited by 13,227
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 4×
“Here, ‘transparency’ is mainly used in computer science literature to describe different frameworks or methods to simplify complex models into binary models, rule generators or the like [1], [5], [42], [45]–[50].”
BackgroundMethodInfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
An interactive web-based dashboard to track COVID-19 in real time
Dong, E., Du, H., & Gardner, L. (2020) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(20)30120-1 · Cited by 10,048
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Projects such as “Watson for Oncology” [19], an unnamed, broadly used algorithm to manage the health of large populations [20], Google Health [21] which can identify signs of diabetic retinopathy, or COMPAS [22] a decision support tool used in court houses to assess risk of recidivism, have all failed one way or anoth…”
BackgroundGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909 · cited 1×
“In particular, they have utilized the Gradient-weighted Class Activation Mapping (Grad-CAM) method [70], originally a debugging tool for DNN, to visualize the output of the model with a heatmap overlay.”
Method“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“xAI refers to methods and techniques that seek to provide insights into the outcome of a ML model and present it in qualitative, understandable terms or visualizations to the stakeholders of the model [4], [6], [24], [43].”
MethodExplanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800 · cited 2×
“Hence, AI technologies create new opportunities for discovering new and improved treatment plans, facilitating early detection of diseases, and monitoring disease progression [12].”
BackgroundHe, K., Gkioxari, G., Dollar, P., & Girshick, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844175 · Cited by 3,457
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 2×
“Recently, scholars have addressed the issue of an ‘AI Chasm’ in the clinical research community [27], [36]–[38], identifying a gap in the literature between building a sound and scientifically correct model and using it in a real-world medical clinic.”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“Here, ‘transparency’ is mainly used in computer science literature to describe different frameworks or methods to simplify complex models into binary models, rule generators or the like [1], [5], [42], [45]–[50].”
MethodCan AI help in screening viral and COVID-19 pneumonia?
Chowdhury, M. E. H., Rahman, T., Khandakar, A., Mazhar, R., Kadir, M. A., Mahbub, Z. B., Islam, K. R., Khan, M. S., Iqbal, A., Emadi, N. A., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3010287 · Cited by 1,913 · cited 2×
“This has fueled the search for alternative ways of detecting the virus [18], [58], [63].”
BackgroundCausability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“2 Drivers for xAI Many scientists suggest that xAI could ease and promote the adoption of AI in the medical domain; as explainability could accommodate understanding and trust, making stakeholders more willing to adopt a given AI application [6]–[8], [11], [40].”
BackgroundGoldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“Explanation concerns expressed by and for the development team have been highly recognized in the literature [6], [47], [81] where developers use xAI frameworks to make sense of the datasets used as well as the inner workings of AI-based models in order to achieve performance results.”
BackgroundIntensive care management of coronavirus disease 2019 (COVID-19): challenges and recommendations
Phua, J., Weng, L., Ling, L., Egi, M., Lim, C.-M., Divatia, J. V., Shrestha, B. R., Arabi, Y. M., Ng, J., Gomersall, C. D., et al. (2020) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(20)30161-2 · Cited by 1,509
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data
Goldstein, M., & Uchida, S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0152173 · Cited by 1,032
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636 · cited 4×
“Other areas where xAI is emphasized as being relevant in relation to AI in healthcare include accountability issues, reliability, justification, and risk reduction [7], [8].”
BackgroundInfluentialVellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 7×
“Recent studies show that model transparency, interpretable results, and an understanding of clinical workflow are all necessary in order to ensure the correct use and adoption of these powerful models [2], [35], [36] in healthcare practices.”
BackgroundResultInfluentialMetrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“Further, it becomes apparent that the need for xAI also emerges from trying to establish discretionary boundaries, a point linked to the problem of accountability when using AI-based systems [82].”
BackgroundThe “inconvenient truth” about AI in healthcare
Panch, T., Mattie, H., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0155-4 · Cited by 473 · cited 2×
“Further, these powerful technologies promise to solve problems of sparse resources by carrying out tasks currently operated by specialists such as radiologists, whose availability and mobility are limited [13], [14].”
BackgroundLauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06870 · Cited by 406
IBM Watson, heal thyself: How IBM overpromised and underdelivered on AI health care
Strickland, E. (2019) · IEEE Spectrum · doi:10.1109/mspec.2019.8678513 · Cited by 402
Viral pneumonia screening on chest X-rays using confidence-aware anomaly detection
Zhang, J., Xie, Y., Pang, G., Liao, Z., Verjans, J., Li, W., Sun, Z., He, J., Li, Y., Shen, C., et al. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3040950 · Cited by 389 · cited 1×
“The classification and detection approaches have the same goal of identifying COVID-19, whereas the other approach is concerned with a severity measure for disease progression itself [66].”
BackgroundExplanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · cited 1×
“Finally, audience related concerns drive xAI needs in order to be able to communicate decisions to patients as well as manage and mitigate decision consequences that may severely impact patients [84].”
BackgroundTransparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260 · cited 1×
“However, due to this heightened complexity, it becomes impossible to understand how these models work, how the data is processed, and how outcomes are generated [7], [25]–[27].”
BackgroundHealth intelligence: how artificial intelligence transforms population and personalized health
Shaban-Nejad, A., Michalowski, M., & Buckeridge, D. L. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0058-9 · Cited by 252
Wehbe, R. M., Sheng, J., Dutta, S., Chai, S., Dravid, A., Barutcu, S., Wu, Y., Cantrell, D. R., Xiao, N., Allen, B. D., et al. (2021) · Radiology · doi:10.1148/radiol.2020203511 · Cited by 190 · cited 1×
“However, image quality has proven suboptimal, as original X-ray images offer much higher quality and contain more helpful metadata in the DICOM (Digital Imaging and Communications in Medicine format than the images often available in a compressed format, such as JPEG [67]–[69].”
BackgroundThe Potential for Artificial Intelligence in Healthcare
Puaschunder, J. M. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3525037 · Cited by 177
Keane, M. T., & Kenny, E. M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-29249-2_11 · Cited by 84 · cited 4×
“Recently, scholars have addressed the issue of an ‘AI Chasm’ in the clinical research community [27], [36]–[38], identifying a gap in the literature between building a sound and scientifically correct model and using it in a real-world medical clinic.”
BackgroundInfluential
Cited by
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Jung, J., Lee, H., Jung, H., & Kim, H. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16110 · Cited by 87
Kim, M., Kim, S., Kim, J., Song, T.-J., & Kim, Y. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103160 · Cited by 53
Examining explainable clinical decision support systems with think aloud protocols
Anjara, S. G., Janik, A., Dunford-Stenger, A., Mc Kenzie, K., Collazo-Lorduy, A., Torrente, M., Costabello, L., & Provencio, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0291443 · Cited by 28
“In response to black-box ML, the field of explainable AI emerged with the aim of “providing explanations catered to human understanding, trust, and transparency” [8].”
Background
Metadata
- Authors
- Gerlings, J., Jensen, M. S., & Shollo, A.
- Venue
- Intelligent Systems Reference Library
- Publisher
- Springer International Publishing
- Published
- 2021-11-27
- Pages
- 169-198
- ISSN
- 1868-4394 (ISSN-L)
- 1868-4408
- OpenAlex
- W3216202622
- Cited by
- 68
- References
- 39
- FWCI
- 13.68
- Type
- book chapter
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.