Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle
The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
The interpretability or explainability of AI systems (XAI) has been a topic gaining renewed attention in recent years across AI and HCI communities. Recent work has drawn attention to the emergent explainability requirements of in situ, applied projects, yet further exploratory work is needed to more fully understand this space. This paper investigates applied AI projects and reports on a qualitative interview study of individuals working on AI projects at a large technology and consulting company. Presenting an empirical understanding of the range of stakeholders in industrial AI projects, this paper also draws out the emergent explainability practices that arise as these projects unfold, highlighting the range of explanation audiences (who), as well as how their explainability needs evolve across the AI project lifecycle (when). We discuss the importance of adopting a sociotechnical lens in designing AI systems, noting how the “AI lifecycle” can serve as a design metaphor to further the XAI design field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Participants' questions reflected their desire for an actionable or utility-oriented understanding to support their end goal of optimizing code generation and programming productivity [18,50,52], such as asking the Input, Output, How and How-to XAI questions (described in Section 4) to facilitate strategies to get better outputs from the AI.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [18]
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Hutchins, E. (1995) · doi:10.7551/mitpress/1881.001.0001 · Cited by 10,801
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Explanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“We must also take into account individuals’ AI literacy [30].”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1×
“The contemporary AI landscape has been aptly characterized as opaque[6] with its ever-increasing interdependencies on multiple components and algorithmic sophistication.”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Context and consciousness: activity theory and human-computer interaction
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5756 · Cited by 2,511
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“According to authors in [18], explainability relates to being able to convey model outputs in ways that match human semantic concepts (interpretability) and simultaneously being complete in that description.”
Background(2006) · Cited by 2,256
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“Research communities around expert systems [48] and recommender systems [20] have grappled with this concern before.”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Plans and Situated Actions: The Problem of Human Machine Communication.
Woolgar, S., & Suchman, L. A. (1989) · Contemporary Sociology · doi:10.2307/2073874 · Cited by 1,604
The algorithmic imaginary: exploring the ordinary affects of Facebook algorithms
Bucher, T. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1154086 · Cited by 1,327
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“Core concepts in XAI have been synthesized in various reviews [10, 19, 35]; a comprehensive review is outside the scope of this paper.”
BackgroundAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“In considering how much of a model should be explained, one study cautions against the potential information overload engendered by a transparent approach in presenting model technicalities [38].”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Visualizing and Understanding Neural Models in NLP
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1082 · Cited by 625
7. Respondent-Driven Sampling: An Assessment of Current Methodology
Gile, K. J., & Handcock, M. S. (2010) · Sociological Methodology · doi:10.1111/j.1467-9531.2010.01223.x · Cited by 613
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 2×
“[7] point to the human-AI onboarding phase as the time “when users are first being introduced to an AI system, learning its capabilities, and determining how they will partner with it in practice” (p.”
BackgroundUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“A large-scale experiment study on the effects of accuracy on trust in AI noted lay users overly trust a model when its observed accuracy is higher than their own accuracy [56].”
BackgroundRight for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · cited 1×
“[42], humans need to feel confident that a model is “right for the right reasons.”
BackgroundVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“An AI system deployed in the real world has a wide range of stakeholders that extends beyond the immediate users (e.g., regulators, model developers, decision-makers, consumers) [21, 50].”
BackgroundUnderstanding User Beliefs About Algorithmic Curation in the Facebook News Feed
Rader, E., & Gray, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702174 · Cited by 386
Olson, J. S., & Kellogg, W. A. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8 · Cited by 273
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 226
Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design
Holtzblatt, K., Wendell, J. B., & Wood, S. (2005) · Ubiquity · doi:10.1145/1066348.1066325 · Cited by 222
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219 · cited 2×
“[23], noting how interpretability involves cooperation and trust-building among stakeholders.”
BackgroundAutomated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210 · cited 1×
“[14] use automatically generated rationales in natural language to present the inner workings of a black-box”
Gell, A. (1988) · Anthropology Today · doi:10.2307/3033230 · Cited by 203
User attitudes towards algorithmic opacity and transparency in online reviewing platforms
Eslami, M., Vaccaro, K., Lee, M. K., Elazari Bar On, A., Gilbert, E., & Karahalios, K. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300724 · Cited by 184 · cited 1×
“An essential challenge to confront is the problem of awareness, or the recognition that a sociotechnical system even includes AI [5, 15, 39].”
BackgroundExplainability scenarios: towards scenario-based XAI design
Wolf, C. T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302317 · Cited by 172
Self-citationText analytics in industry: Challenges, desiderata and trends
Ittoo, A., Nguyen, L.-M., & van den Bosch, A. P. J. (2015) · Computers in Industry · doi:10.1016/j.compind.2015.12.001 · Cited by 164
Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry
Benjamin, J. J., Berger, A., Merrill, N., & Pierce, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445481 · Cited by 130
Mapping Machine Learning Advances from HCI Research to Reveal Starting Places for Design Innovation
Yang, Q., Banovic, N., & Zimmerman, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173704 · Cited by 125
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · cited 1×
“, embedding the progressive disclosure principles [44]); (iii) catering to wide variations in stakeholders’ AI literacies; (iv) minimizing expectation violations during deployment (e.”
BackgroundCOGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · cited 1×
“Very recent work in this space has suggested different visualization techniques that can be customized to a person’s desired level of detail and cognitive load [2, 57].”
BackgroundVenerating the Black Box: Magic in Media Discourse on Technology
Stahl, W. A. (1995) · Science Technology & Human Values · doi:10.1177/016224399502000205 · Cited by 71
Hind, M. (2019) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/3313096 · Cited by 60
Monsters, Metaphors, and Machine Learning
Dove, G., & Fayard, A.-L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376275 · Cited by 59
Interpreting Neural Networks to Improve Politeness Comprehension
Aubakirova, M., & Bansal, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1216 · Cited by 58
Yu, B., Yuan, Y., Terveen, L., Wu, Z. S., Forlizzi, J., & Zhu, H. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395528 · Cited by 44
Designing and Prototyping from the Perspective of AI in the Wild
Malsattar, N., Kihara, T., & Giaccardi, E. (2019) · Proceedings of the 2019 on Designing Interactive Systems Conference · doi:10.1145/3322276.3322351 · Cited by 35
Wolf, C. T., Zhu, H., Bullard, J., Lee, M. K., & Brubaker, J. R. (2018) · Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3272973.3273005 · Cited by 32
Self-citationEncountering ethics through design: a workshop with nonhuman participants
Reddy, A. V., Nicenboim, I. P., Pierce, J., & Giaccardi, E. (2020) · AI & Society · doi:10.1007/s00146-020-01088-7 · Cited by 25
Artificial Intelligence and Risk in Design
Luusua, A., & Ylipulli, J. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395491 · Cited by 19
Research works printed but not linked: 7
- 10.5555/1597122.1597135 · doi:10.5555/1597122.1597135
- Shipi Dhanorkar , Christine T Wolf , Kun Qian , Anbang Xu , Lucian Popa , and Yunyao Li. 2020. Tutorial on Explainability for Natural Language Processing. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing(2020). Shipi Dhanorkar, Christine T Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2020. Tutorial on Explainability for Natural Language Processing. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing(2020).
- Upol Ehsan and Mark O. Riedl . 2020. Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach . In HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.) . Springer International Publishing , Cham , 449–466. Upol Ehsan and Mark O. Riedl. 2020. Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. In HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 449–466.
- Riccardo Guidotti , Anna Monreale , Salvatore Ruggieri , Franco Turini , Fosca Giannotti , and Dino Pedreschi . 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 ( 2018 ), 1–42. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1–42.
- Karen Holtzblatt Jessamyn Wendell and Shelley Wood. 2004. Rapid Contextual Design. Morgan Kaufmann. https://www.elsevier.com/books/rapid-contextual-design/holtzblatt/978-0-12-354051-5 Karen Holtzblatt Jessamyn Wendell and Shelley Wood. 2004. Rapid Contextual Design. Morgan Kaufmann. https://www.elsevier.com/books/rapid-contextual-design/holtzblatt/978-0-12-354051-5
- 10.5555/2891730.2891924 · doi:10.5555/2891730.2891924
- William R Swartout . 1983 . XPLAIN: A system for creating and explaining expert consulting programs. Artificial intelligence 21, 3 (1983), 285–325. William R Swartout. 1983. XPLAIN: A system for creating and explaining expert consulting programs. Artificial intelligence 21, 3 (1983), 285–325.
Cited by
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Haque, A. B., Islam, A. K. M. N., & Mikalef, P. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 240
“In this case, the user wants minimal and adequate information to be provided to avoid cognitive overload (Ehsan et al., 2019; Schmidt et al., 2020; Hudon et al., 2021; Dhanorkar et al., 2021).”
BackgroundWhat is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“…an explanation [23, 103, 213, 231], towards ensuring that people are able to interpret and comprehend those explanations [5, 7, 16, 32, 44, 65, 75, 78–81, 89, 95, 120, 127, 132, 133, 136, 142, 145, 146, 150, 151, 158, 159, 170, 174, 186, 187, 196, 197, 207, 209, 221, 224, 235, 236, 246, 257, 262,…”
Metadata
- Authors
- Dhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y.
- Affiliations
- Pennsylvania State University (US)
- Independent Researcher, United States
- Amazon (United States) (US)
- IBM Research - Almaden (US)
- IBM (United States) (US)
- Venue
- Designing Interactive Systems Conference 2021
- Publisher
- ACM
- Published
- 2021-06-28
- Pages
- 1591-1602
- OpenAlex
- W3173721282
- Cited by
- 119
- References
- 51
- Citations by year
- 2026: 12 · 2025: 28 · 2024: 35 · 2023: 20 · 2022: 20 · 2021: 4
- FWCI
- 9.13
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- explainable AI, AI lifecycle, stakeholder needs, socio-technical design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.