Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 35 of the 56 sources its claims cite were checked for support.

Investigating Explainability of Generative AI for Code through Scenario-based Design

Open access
Sun, Jiao, Liao, Q. Vera, Muller, Michael, Agarwal, Mayank, Houde, Stephanie, Talamadupula, Kartik, Weisz, Justin D. · 2022 · 27th International Conference on Intelligent User Interfaces

Abstract

What does it mean for a generative AI model to be explainable? The emergent discipline of explainable AI (XAI) has made great strides in helping people understand discriminative models. Less attention has been paid to generative models that produce artifacts, rather than decisions, as output. Meanwhile, generative AI (GenAI) technologies are maturing and being applied to application domains such as software engineering. Using scenario-based design and question-driven XAI design approaches, we explore users’ explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion. We conducted 9 workshops with 43 software engineers in which real examples from state-of-the-art generative AI models were used to elicit users’ explainability needs. Drawing from prior work, we also propose 4 types of XAI features for GenAI for code and gathered additional design ideas from participants. Our work explores explainability needs for GenAI for code and demonstrates how human-centered approaches can drive the technical development of XAI in novel domains.

Overview

CoChat Summary

TL;DR

This paper uses scenario-based design and question-driven XAI methods to investigate what software engineers need to understand about generative AI for code. Across 9 workshops with 43 software engineers, it identifies 11 categories of explainability needs and proposes four XAI feature areas for code-generation use cases.

Key findings

  • The study identified 11 categories of explainability needs for GenAI for code; the top five most frequent were Input, Output, How (global), Performance, and How to. Four categories did not appear as prominent needs in prior discriminative-ML decision-support work: Input, Control, System Requirements & Impact, and Limitations.
  • Participants’ needs centered on understanding the scope, characteristics, and limitations of input/output spaces and on getting a global view of how code is generated, rather than on model internals or representation visualizations.
  • The paper proposes four XAI feature areas: AI documentation, uncertainty indicators, attention visualization, and social transparency. For AI documentation, participants requested categories including Examples and tutorials, Software engineering capabilities, Output code quality and utility, Supported languages and frameworks, Control, and Deployment Requirements & Platform.
  • For uncertainty, participants found line-level indicators useful but insufficient; they wanted alternative outputs, uncertainty explanations, fine-grained uncertainty, human input to resolve uncertainty, and interactive testing. Attention visualizations were seen as useful for understanding and improving inputs, while social transparency was mapped across software development life-cycle stages.

Methods

  • Design: scenario-based design workshops combined with a question-driven XAI approach. The study ran 9 semi-structured remote workshops lasting 60–70 minutes with 3–6 participants each, using Mural for collaboration.
  • Data/sample: 43 software engineers were selected from 81 survey respondents; selection required 1+ year of Python experience, or 1+ year of both Python and Java for code translation. Three use cases were studied: code translation, code autocompletion, and natural language to code. Real outputs from TransCoder and Copilot were used in scenarios.
  • Analysis: content analysis of Mural content and transcribed recordings. The authors extracted 487 Mural messages and 249 relevant transcript paragraphs; around 600 minutes of interviews were recorded and transcribed. For question coding, two researchers independently coded 106 questions and reached agreement on 66 of 71 messages (93%) before finalizing the schema.

Limitations

  • Stated by the paper: the method has limitations the authors reflect on, including that less experienced participants generally asked fewer questions and may need more proactive explanations; the choice of data point can sway workshop discussions; and the open-endedness of design probes required balancing concreteness and openness. The paper also notes that some transcribed comments could not be reliably attributed to individual speakers.
  • Stated by the paper: the AI documentation categories in Table 3 are not meant to be an exhaustive template, and the paper raises a fundamental question about the readiness of GenAI for code that may not be addressable by explainability alone.
  • Evident from the text: participants came from one large international IT company, were mostly from the United States and India, and were predominantly male; the study covers only three GenAI-for-code use cases and does not report an evaluation of the proposed XAI features with users.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3490099.3511119

Cites

103 of 103 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overview of the Mural used in workshop #6 (W6-NL2Code) for the code autocompletion use case. We introduce the workshop and set ground rules in (a) , introduce the Alex persona in (b) , and show the model output for the use case in (c) . Then, we ask participants for what they want to know about the GenAI model in (d) , elicit design ideas for AI documentation in (e) , and ask participants to ideate on what information Alex would want to know about his team members' use of the model in (f) .
Figure 1 Overview of the Mural used in workshop #6 (W6-NL2Code) for the code autocompletion use case. We introduce the workshop and set ground rules in (a) , introduce the Alex persona in (b) , and show the model output for the use case in (c) . Then, we ask participants for what they want to know about the GenAI model in (d) , elicit design ideas for AI documentation in (e) , and ask participants to ideate on what information Alex would want to know about his team members' use of the model in (f) .CC-BY · From the PDF
(c) Attention Visualizer
(c) Attention VisualizerCC-BY · From the PDF
Image 3
CC-BY · From the PDF
Image 4
CC-BY · From the PDF

Verification

Claims the paper makes

20 Valid · 4 Not valid · 13 Not checked · 98 Own results and interpretations

50 more claims

Create a free account to see all 135 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 64
  • It may state two separate claims as one. 31
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 78 references

Research works printed but not linked: 6

  1. Cecilia Aragon , Shion Guha , Marina Kogan , Michael Muller , and Gina Neff . 2022. Human-Centered Data Science: An Introduction . MIT Press , Cambridge, MA . Cecilia Aragon, Shion Guha, Marina Kogan, Michael Muller, and Gina Neff. 2022. Human-Centered Data Science: An Introduction. MIT Press, Cambridge, MA.
  2. Upol Ehsan and Mark Riedl . 2021 . Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022. Upol Ehsan and Mark Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022.
  3. Ryan Louie , Any Cohen , Cheng- Zhi Anna Huang , Michael Terry , and Carrie J Cai . 2020 . Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI. Ryan Louie, Any Cohen, Cheng-Zhi Anna Huang, Michael Terry, and Carrie J Cai. 2020. Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI.
  4. Michael Muller , Plamen Angelov , Shion Guha , Marina Kogan , Gina Neff , Nuria Oliver , Manuel Gomez Rodriquez , and Adrian Weller . 2021. HCAI@NeurIPS2021 : Human Centered AI workshop at NeurIPS 2021 . https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022. Michael Muller, Plamen Angelov, Shion Guha, Marina Kogan, Gina Neff, Nuria Oliver, Manuel Gomez Rodriquez, and Adrian Weller. 2021. HCAI@NeurIPS2021: Human Centered AI workshop at NeurIPS 2021. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022.
  5. John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). · arXiv
  6. Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS. Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS.

Cited by

See all 219

Metadata

Authors
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D.
Affiliations
  • University of Southern California (US)
  • IBM (United States) (US)
Venue
27th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2022-03-22
Pages
212-228
OpenAlex
W4220962633
Cited by
219
References
78
Citations by year
2026: 29 · 2025: 58 · 2024: 91 · 2023: 37 · 2022: 4
FWCI
26.25
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
explainable AI, scenario-based design, natural language to code, program translation, human-centered AI design, software engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Investigating Explainability of Generative AI for Code through Scenario-based Design. https://app.cochat.ai/papers/10.1145/3490099.3511119
The paper
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022). Investigating Explainability of Generative AI for Code through Scenario-based Design. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511119

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.