The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 35 of the 56 sources its claims cite were checked for support.
Investigating Explainability of Generative AI for Code through Scenario-based Design
Open accessAbstract
What does it mean for a generative AI model to be explainable? The emergent discipline of explainable AI (XAI) has made great strides in helping people understand discriminative models. Less attention has been paid to generative models that produce artifacts, rather than decisions, as output. Meanwhile, generative AI (GenAI) technologies are maturing and being applied to application domains such as software engineering. Using scenario-based design and question-driven XAI design approaches, we explore users’ explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion. We conducted 9 workshops with 43 software engineers in which real examples from state-of-the-art generative AI models were used to elicit users’ explainability needs. Drawing from prior work, we also propose 4 types of XAI features for GenAI for code and gathered additional design ideas from participants. Our work explores explainability needs for GenAI for code and demonstrates how human-centered approaches can drive the technical development of XAI in novel domains.
Overview
CoChat Summary
TL;DR
This paper uses scenario-based design and question-driven XAI methods to investigate what software engineers need to understand about generative AI for code. Across 9 workshops with 43 software engineers, it identifies 11 categories of explainability needs and proposes four XAI feature areas for code-generation use cases.
Key findings
- The study identified 11 categories of explainability needs for GenAI for code; the top five most frequent were Input, Output, How (global), Performance, and How to. Four categories did not appear as prominent needs in prior discriminative-ML decision-support work: Input, Control, System Requirements & Impact, and Limitations.
- Participants’ needs centered on understanding the scope, characteristics, and limitations of input/output spaces and on getting a global view of how code is generated, rather than on model internals or representation visualizations.
- The paper proposes four XAI feature areas: AI documentation, uncertainty indicators, attention visualization, and social transparency. For AI documentation, participants requested categories including Examples and tutorials, Software engineering capabilities, Output code quality and utility, Supported languages and frameworks, Control, and Deployment Requirements & Platform.
- For uncertainty, participants found line-level indicators useful but insufficient; they wanted alternative outputs, uncertainty explanations, fine-grained uncertainty, human input to resolve uncertainty, and interactive testing. Attention visualizations were seen as useful for understanding and improving inputs, while social transparency was mapped across software development life-cycle stages.
Methods
- Design: scenario-based design workshops combined with a question-driven XAI approach. The study ran 9 semi-structured remote workshops lasting 60–70 minutes with 3–6 participants each, using Mural for collaboration.
- Data/sample: 43 software engineers were selected from 81 survey respondents; selection required 1+ year of Python experience, or 1+ year of both Python and Java for code translation. Three use cases were studied: code translation, code autocompletion, and natural language to code. Real outputs from TransCoder and Copilot were used in scenarios.
- Analysis: content analysis of Mural content and transcribed recordings. The authors extracted 487 Mural messages and 249 relevant transcript paragraphs; around 600 minutes of interviews were recorded and transcribed. For question coding, two researchers independently coded 106 questions and reached agreement on 66 of 71 messages (93%) before finalizing the schema.
Limitations
- Stated by the paper: the method has limitations the authors reflect on, including that less experienced participants generally asked fewer questions and may need more proactive explanations; the choice of data point can sway workshop discussions; and the open-endedness of design probes required balancing concreteness and openness. The paper also notes that some transcribed comments could not be reliably attributed to individual speakers.
- Stated by the paper: the AI documentation categories in Table 3 are not meant to be an exhaustive template, and the paper raises a fundamental question about the readiness of GenAI for code that may not be addressable by explainability alone.
- Evident from the text: participants came from one large international IT company, were mostly from the United States and India, and were predominantly male; the study covers only three GenAI-for-code use cases and does not report an evaluation of the proposed XAI features with users.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
20 Valid · 4 Not valid · 13 Not checked · 98 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“users' explainability needs emerge in a usage context, guided by their goals and shaped by their backgrounds, expectations, as well as socio, organizational and cultural contexts [20,48,52].”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [20] 20. Upol Ehsan and Mark Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI -paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022. source
- Partly supportedcites [48] 48. Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-15. sourceExperimental — accuracy not yet measured
In “Understanding user needs for explainability” (page unavailable), the paper identifies usage points, decision contexts, and user characteristics as factors relevant to explainability needs. It also links motivation for explanation to users’ system goals and discusses backgrounds and beliefs in “In quest for human-like explanations.” Our copy does not establish that cultural contexts shape those needs, and the paper explicitly says it does not conclude how the identified factors vary them. (LLM, conf=0.84)
- Partly supportedcites [52] 52. Q Vera Liao and Kush R Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv preprint arXiv:2110.10790 (2021). sourceExperimental — accuracy not yet measured
Section 3 (p. 5) says users’ explainability needs vary with their goals, backgrounds, and usage contexts. Section 4 (p. 12) also says those needs should be situated in socio-organizational contexts. Our copy does not clearly say that users’ expectations or cultural contexts shape those needs. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 6 DISCUSSION 6.1 Informing XAI approaches for GenAI for code
What the sources say
- Partly supportedcites [9] 9. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). sourceExperimental — accuracy not yet measured
Section 4.2 (p. 7) examines performance by the number of prompting examples, and Section 4.3 (p. 8) discusses how long prompts can hurt synthesis performance. But the study is not limited to those questions: the Introduction (p. 3) also lists model size, example selection, and sampling technique among the factors analyzed. (LLM, conf=0.85)
- Partly supportedcites [15] 15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchi sourceExperimental — accuracy not yet measured
The APPS results (section 3.5, p. 7) evaluate a one-shot prompt, and the Limitations section (p. 10) reports declining performance as synthetic docstrings grow longer. Our copy does not show an investigation of how varying the number of few-shot examples affects performance, nor establish that similar code studies are limited to these two topics. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Participants' questions reflected their desire for an actionable or utility-oriented understanding to support their end goal of optimizing code generation and programming productivity [18,50,52], such as asking the Input, Output, How and How-to XAI questions (described in Section 4) to facilitate strategies to get better outputs from the AI.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 6 DISCUSSION 6.1 Informing XAI approaches for GenAI for code
What the sources say
- Not checked — no full text of the sourcecites [18] 18. Shipi Dhanorkar, Christine T Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021. Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021. 1591-1602. source
- Partly supportedcites [50] 50. Q Vera Liao, Milena Pribić, Jaesik Han, Sarah Miller, and Daby Sow. 2021. Question-Driven Design Process for Explainable AI User Experiences. arXiv preprint arXiv:2104.03483 (2021). sourceExperimental — accuracy not yet measured
In “Question Analysis” (pp. 13–14), participants’ questions are linked to practical goals, including deciding actionable next steps for patient management. The paper’s participants were healthcare providers considering patient-risk predictions; it does not support the claim that their goal was optimizing code generation or programming productivity. (LLM, conf=0.87)
- Partly supportedcites [52] 52. Q Vera Liao and Kush R Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv preprint arXiv:2110.10790 (2021). sourceExperimental — accuracy not yet measured
Section 4 (p. 8) says users seek actionable understanding that serves their objectives, and Section 3 (p. 6) describes eliciting users’ XAI questions. The paper does not report the citing study’s participants or establish that their questions aimed to optimize code generation and programming productivity. (LLM, conf=0.87)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“These results move beyond Ehsan et al. [19]'s work that defined ST features at the point of AI decision-support.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 5.4 Social Transparency
What the sources say
- Supportedcites [19] 19. Upol Ehsan, Q Vera Liao, Michael Muller, Mark O Riedl, and Justin D Weisz. 2021. Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-19. source
Section 3 (p. 8) calls the What, Why, Who, and When elements the constitutive design features of social transparency and presents them in a scenario built around an AI-mediated price recommendation tool. This supports the description of Ehsan et al.’s work; the citing authors’ statement that their results “move beyond” it is their own comparison. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To understand what output the GenAI can produce is another frequent explainability need.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Output.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 135 claims and every source verdict.
- Probably not a substantial claim. 64
- It may state two separate claims as one. 31
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This process is typically informal and relies on trial-and-error [104, 131, 165, 190].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [165]
In the citing paper
“The recommendation to, 'Make clear what the system can do' may be difficult to implement in light of the emergent and unanticipated behaviors of generative foundation models [19], as well as the trial-and-error methods by which users iterate toward a desired outcome [104, 131, 165, 190].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [165]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(N=14; 41%), with smaller numbers from Canada”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Selection Criteria.
Reported 41 · recomputed 32.55814
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 7 CONCLUSION
“Figure 1: Overview of the Mural used in workshop #6 (W6-NL2Code) for the code autocompletion use case.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Additional information
“We show the example code as the base case for workshops about natural language to code (W6-NL2Code, W7-NL2Code, W8-NL2Code) in Figure 1.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the majority of our participants were from the United States (N=22; 65%)”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Selection Criteria.
Reported 65 · recomputed 51.162791
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > How (global).
“P61 from W1-CT saying that”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Data availability
“4 Code Autocompletion (CA) Software Engineer (P17, P39), Researcher/Scientist (P37, P55), Bioinformatician (P48), Data Scientist (P61)”
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating posthoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundProceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“To understand performance differences and limitations with regard to different types of input, solutions have been explored for natural language generation under Prompt Engineering [57, 58].”
Background
Research works printed but not linked: 6
- Cecilia Aragon , Shion Guha , Marina Kogan , Michael Muller , and Gina Neff . 2022. Human-Centered Data Science: An Introduction . MIT Press , Cambridge, MA . Cecilia Aragon, Shion Guha, Marina Kogan, Michael Muller, and Gina Neff. 2022. Human-Centered Data Science: An Introduction. MIT Press, Cambridge, MA.
- Upol Ehsan and Mark Riedl . 2021 . Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022. Upol Ehsan and Mark Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022.
- Ryan Louie , Any Cohen , Cheng- Zhi Anna Huang , Michael Terry , and Carrie J Cai . 2020 . Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI. Ryan Louie, Any Cohen, Cheng-Zhi Anna Huang, Michael Terry, and Carrie J Cai. 2020. Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI.
- Michael Muller , Plamen Angelov , Shion Guha , Marina Kogan , Gina Neff , Nuria Oliver , Manuel Gomez Rodriquez , and Adrian Weller . 2021. HCAI@NeurIPS2021 : Human Centered AI workshop at NeurIPS 2021 . https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022. Michael Muller, Plamen Angelov, Shion Guha, Marina Kogan, Gina Neff, Nuria Oliver, Manuel Gomez Rodriquez, and Adrian Weller. 2021. HCAI@NeurIPS2021: Human Centered AI workshop at NeurIPS 2021. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022.
- John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). · arXiv
- Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS. Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS.
Cited by
Generative AI and ChatGPT: applications, challenges, and AI-human collaboration
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2023.2233814 · Cited by 1,363
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Sun et al. [82] focus on users’ explainability needs for generative AI in three software engineering use cases: code generation based on natural language description (with Copilot), translation between different programming languages (with Transcoder), and code autocompletion (with Copilot).”
MethodGenerative artificial intelligence
Banh, L., & Strobel, G. (2023) · Electronic Markets · doi:10.1007/s12525-023-00680-1 · Cited by 579
“Programmers using applications such as GitHub Copilot are supported by automatically written chunks of code, ideas converted into actionable scripts, auto-completion functions, generated unit tests, duplicate code detection, and bug fixing (Sun et al., 2022).”
BackgroundMethodLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
A Comprehensive Review on Generative AI for Education
Mittal, U., Sai, S., Chamola, V., & Sangwan, D. (2024) · IEEE Access · doi:10.1109/access.2024.3468368 · Cited by 383
“This issue can be addressed by promoting the concept of explainable AI, which is based on the idea that the model should be able to explain each and everything that it produces [141].”
Background
Metadata
- Authors
- Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D.
- Affiliations
- University of Southern California (US)
- IBM (United States) (US)
- ORCID
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 212-228
- OpenAlex
- W4220962633
- Semantic Scholar
- 76ebf56d6ebb833d8c8c4124f7b2f15771a4997c
- Cited by
- 219
- References
- 78
- Citations by year
- 2026: 29 · 2025: 58 · 2024: 91 · 2023: 37 · 2022: 4
- FWCI
- 26.25
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- explainable AI, scenario-based design, natural language to code, program translation, human-centered AI design, software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating Explainability of Generative AI for Code through Scenario-based Design. https://app.cochat.ai/papers/10.1145/3490099.3511119
- The paper
- Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022). Investigating Explainability of Generative AI for Code through Scenario-based Design. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511119
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.