The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 35 of the 56 sources its claims cite were checked for support.
Investigating Explainability of Generative AI for Code through Scenario-based Design
Open accessAbstract
What does it mean for a generative AI model to be explainable? The emergent discipline of explainable AI (XAI) has made great strides in helping people understand discriminative models. Less attention has been paid to generative models that produce artifacts, rather than decisions, as output. Meanwhile, generative AI (GenAI) technologies are maturing and being applied to application domains such as software engineering. Using scenario-based design and question-driven XAI design approaches, we explore users’ explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion. We conducted 9 workshops with 43 software engineers in which real examples from state-of-the-art generative AI models were used to elicit users’ explainability needs. Drawing from prior work, we also propose 4 types of XAI features for GenAI for code and gathered additional design ideas from participants. Our work explores explainability needs for GenAI for code and demonstrates how human-centered approaches can drive the technical development of XAI in novel domains.
Overview
CoChat Summary
TL;DR
This paper uses scenario-based design and question-driven XAI methods to investigate what software engineers need to understand about generative AI for code. Across 9 workshops with 43 software engineers, it identifies 11 categories of explainability needs and proposes four XAI feature areas for code-generation use cases.
Key findings
- The study identified 11 categories of explainability needs for GenAI for code; the top five most frequent were Input, Output, How (global), Performance, and How to. Four categories did not appear as prominent needs in prior discriminative-ML decision-support work: Input, Control, System Requirements & Impact, and Limitations.
- Participants’ needs centered on understanding the scope, characteristics, and limitations of input/output spaces and on getting a global view of how code is generated, rather than on model internals or representation visualizations.
- The paper proposes four XAI feature areas: AI documentation, uncertainty indicators, attention visualization, and social transparency. For AI documentation, participants requested categories including Examples and tutorials, Software engineering capabilities, Output code quality and utility, Supported languages and frameworks, Control, and Deployment Requirements & Platform.
- For uncertainty, participants found line-level indicators useful but insufficient; they wanted alternative outputs, uncertainty explanations, fine-grained uncertainty, human input to resolve uncertainty, and interactive testing. Attention visualizations were seen as useful for understanding and improving inputs, while social transparency was mapped across software development life-cycle stages.
Methods
- Design: scenario-based design workshops combined with a question-driven XAI approach. The study ran 9 semi-structured remote workshops lasting 60–70 minutes with 3–6 participants each, using Mural for collaboration.
- Data/sample: 43 software engineers were selected from 81 survey respondents; selection required 1+ year of Python experience, or 1+ year of both Python and Java for code translation. Three use cases were studied: code translation, code autocompletion, and natural language to code. Real outputs from TransCoder and Copilot were used in scenarios.
- Analysis: content analysis of Mural content and transcribed recordings. The authors extracted 487 Mural messages and 249 relevant transcript paragraphs; around 600 minutes of interviews were recorded and transcribed. For question coding, two researchers independently coded 106 questions and reached agreement on 66 of 71 messages (93%) before finalizing the schema.
Limitations
- Stated by the paper: the method has limitations the authors reflect on, including that less experienced participants generally asked fewer questions and may need more proactive explanations; the choice of data point can sway workshop discussions; and the open-endedness of design probes required balancing concreteness and openness. The paper also notes that some transcribed comments could not be reliably attributed to individual speakers.
- Stated by the paper: the AI documentation categories in Table 3 are not meant to be an exhaustive template, and the paper raises a fundamental question about the readiness of GenAI for code that may not be addressable by explainability alone.
- Evident from the text: participants came from one large international IT company, were mostly from the United States and India, and were predominantly male; the study covers only three GenAI-for-code use cases and does not report an evaluation of the proposed XAI features with users.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
20 Valid · 4 Not valid · 13 Not checked · 98 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“users' explainability needs emerge in a usage context, guided by their goals and shaped by their backgrounds, expectations, as well as socio, organizational and cultural contexts [20,48,52].”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [20] 20. Upol Ehsan and Mark Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI -paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022. source
- Partly supportedcites [48] 48. Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-15. sourceExperimental — accuracy not yet measured
In “Understanding user needs for explainability” (page unavailable), the paper identifies usage points, decision contexts, and user characteristics as factors relevant to explainability needs. It also links motivation for explanation to users’ system goals and discusses backgrounds and beliefs in “In quest for human-like explanations.” Our copy does not establish that cultural contexts shape those needs, and the paper explicitly says it does not conclude how the identified factors vary them. (LLM, conf=0.84)
- Partly supportedcites [52] 52. Q Vera Liao and Kush R Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv preprint arXiv:2110.10790 (2021). sourceExperimental — accuracy not yet measured
Section 3 (p. 5) says users’ explainability needs vary with their goals, backgrounds, and usage contexts. Section 4 (p. 12) also says those needs should be situated in socio-organizational contexts. Our copy does not clearly say that users’ expectations or cultural contexts shape those needs. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar studies in the code space are limited to the number of few-shot examples in the prompt, and the effect of length of prompt on performance [9,15].”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 6 DISCUSSION 6.1 Informing XAI approaches for GenAI for code
What the sources say
- Partly supportedcites [9] 9. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). sourceExperimental — accuracy not yet measured
Section 4.2 (p. 7) examines performance by the number of prompting examples, and Section 4.3 (p. 8) discusses how long prompts can hurt synthesis performance. But the study is not limited to those questions: the Introduction (p. 3) also lists model size, example selection, and sampling technique among the factors analyzed. (LLM, conf=0.85)
- Partly supportedcites [15] 15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchi sourceExperimental — accuracy not yet measured
The APPS results (section 3.5, p. 7) evaluate a one-shot prompt, and the Limitations section (p. 10) reports declining performance as synthetic docstrings grow longer. Our copy does not show an investigation of how varying the number of few-shot examples affects performance, nor establish that similar code studies are limited to these two topics. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Participants' questions reflected their desire for an actionable or utility-oriented understanding to support their end goal of optimizing code generation and programming productivity [18,50,52], such as asking the Input, Output, How and How-to XAI questions (described in Section 4) to facilitate strategies to get better outputs from the AI.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 6 DISCUSSION 6.1 Informing XAI approaches for GenAI for code
What the sources say
- Not checked — no full text of the sourcecites [18] 18. Shipi Dhanorkar, Christine T Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021. Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Designing Interactive Systems Conference 2021. 1591-1602. source
- Partly supportedcites [50] 50. Q Vera Liao, Milena Pribić, Jaesik Han, Sarah Miller, and Daby Sow. 2021. Question-Driven Design Process for Explainable AI User Experiences. arXiv preprint arXiv:2104.03483 (2021). sourceExperimental — accuracy not yet measured
In “Question Analysis” (pp. 13–14), participants’ questions are linked to practical goals, including deciding actionable next steps for patient management. The paper’s participants were healthcare providers considering patient-risk predictions; it does not support the claim that their goal was optimizing code generation or programming productivity. (LLM, conf=0.87)
- Partly supportedcites [52] 52. Q Vera Liao and Kush R Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv preprint arXiv:2110.10790 (2021). sourceExperimental — accuracy not yet measured
Section 4 (p. 8) says users seek actionable understanding that serves their objectives, and Section 3 (p. 6) describes eliciting users’ XAI questions. The paper does not report the citing study’s participants or establish that their questions aimed to optimize code generation and programming productivity. (LLM, conf=0.87)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“These results move beyond Ehsan et al. [19]'s work that defined ST features at the point of AI decision-support.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 5.4 Social Transparency
What the sources say
- Supportedcites [19] 19. Upol Ehsan, Q Vera Liao, Michael Muller, Mark O Riedl, and Justin D Weisz. 2021. Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-19. source
Section 3 (p. 8) calls the What, Why, Who, and When elements the constitutive design features of social transparency and presents them in a scenario built around an AI-mediated price recommendation tool. This supports the description of Ehsan et al.’s work; the citing authors’ statement that their results “move beyond” it is their own comparison. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To understand what output the GenAI can produce is another frequent explainability need.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Output.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 135 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 64
- It may state two separate claims as one. 31
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This process is typically informal and relies on trial-and-error [104, 131, 165, 190].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [165]
In the citing paper
“The recommendation to, 'Make clear what the system can do' may be difficult to implement in light of the emergent and unanticipated behaviors of generative foundation models [19], as well as the trial-and-error methods by which users iterate toward a desired outcome [104, 131, 165, 190].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [165]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(N=14; 41%), with smaller numbers from Canada”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Selection Criteria.
Reported 41 · recomputed 32.55814
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > 7 CONCLUSION
“Figure 1: Overview of the Mural used in workshop #6 (W6-NL2Code) for the code autocompletion use case.”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Additional information
“We show the example code as the base case for workshops about natural language to code (W6-NL2Code, W7-NL2Code, W8-NL2Code) in Figure 1.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the majority of our participants were from the United States (N=22; 65%)”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Selection Criteria.
Reported 65 · recomputed 51.162791
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > How (global).
“P61 from W1-CT saying that”
§ Investigating Explainability of Generative AI for Code through Scenario-based Design > Data availability
“4 Code Autocompletion (CA) Software Engineer (P17, P39), Researcher/Scientist (P37, P55), Bioinformatician (P48), Data Scientist (P61)”
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating posthoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodLearning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundProceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“To understand performance differences and limitations with regard to different types of input, solutions have been explored for natural language generation under Prompt Engineering [57, 58].”
BackgroundA model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1×
“Their suggested improvement and follow-up interactions can be categorized as AI initiated and human initiated [38, 74].”
BackgroundExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 1×
“A full review of XAI techniques is beyond the scope of this paper and can be found in many recent survey papers [1, 29, 55, 56].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundGaver, B., Dunne, T., & Pacenti, E. (1999) · Interactions · doi:10.1145/291224.291235 · Cited by 2,303 · cited 1×
“In short, the design of a “probe” for participatory ideation and feedback is a non-trivial design decision, requiring a balance between concreteness and openness, considerations of both technical feasibility and user value, and diligent pilot testing [12, 24].”
BackgroundIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1×
“been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating posthoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“This task has been an important benchmark for technical work in GenAI for code [61] and has gained extensive attention from both industry and academia [3, 23, 30, 88, 99].”
BackgroundWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“Their suggested improvement and follow-up interactions can be categorized asAI initiated and human initiated [38, 74].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · cited 1×
“This notion of driving technical development based on insights from empirical user studies is also at the core of broad IUI research [5].”
BackgroundAttention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“a distribution over attended-to input units [101] to indicate the relative importance of input units for the output.”
BackgroundQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 9×
“We adapted the question-driven method of Liao et al. [48, 50] to elicit and comprehensively explore participants’ explainability needs by what kind questions they would ask in the scenarios.”
BackgroundMethodInfluentialSelf-citationIsolating Sources of Disentanglement in Variational AutoencodersPreprint
Chen, R. T. Q., Li, X., Grosse, R., & Duvenaud, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04942 · Cited by 866
Rationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 1×
“been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating posthoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodShneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · cited 1×
“Many XAI techniques have been developed for discriminative ML models, both through producing directly interpretable models [14, 44] and generating post-hoc explanations for a trained opaque-box model [47, 62, 80, 104].”
MethodUnified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · cited 2×
“Models trained on massive code data sets are even able to handle multiple use cases at the same time, such as PLBART [3], CodeBERT [23], GraphCodeBERT [30].”
BackgroundMethodExplainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · cited 1×
“…human-centered definition, many have argued that it is necessary to provide transparent information beyond the model internals, such as its performance, limitations, training data, and development procedure, to enable a holistic understanding of the AI and more actionable insights [11, 48, 73, 94].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · cited 2×
“Prior work has shown how to use attention weights to visually depict which words in an input sequence, plus which words in previously-generated output, were responsible for a given token’s output [95, 96].”
BackgroundWhy and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 2×
“…referred to prior work using prototypical questions to represent explainability needs by Liao et al. [48] in the context of discriminative ML systems, and “intelligibility types” [53, 54] for context-aware intelligent systems by Lim and Dey, and used these existing definitions to guide our coding.”
MethodExpanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · cited 4×
“Theseresults move beyond Ehsan et al. [19]’s work that defined ST features at the point of AI decision-support.”
BackgroundInfluentialSelf-citationCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Boehner, K., Vertesi, J., Sengers, P., & Dourish, P. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240789 · Cited by 525 · cited 1×
“In short, the design of a “probe” for participatory ideation and feedback is a non-trivial design decision, requiring a balance between concreteness and openness, considerations of both technical feasibility and user value, and diligent pilot testing [12, 24].”
BackgroundGraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
Conversational processes and causal explanation.
Hilton, D. J. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.1.65 · Cited by 456
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · cited 3×
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundSelf-citationProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Human‐centered artificial intelligence and machine learning
Riedl, M. O. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.117 · Cited by 371 · cited 1×
“The termHuman-centered AI has emerged inmany academic works and public discussions [22, 25, 45, 65, 85, 89].”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
Analyzing the Structure of Attention in a Transformer Language Model
Vig, J., & Belinkov, Y. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4808 · Cited by 301 · cited 2×
“Prior work has shown how to use attention weights to visually depict which words in an input sequence, plus which words in previously-generated output, were responsible for a given token’s output [95, 96].”
BackgroundUncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300 · cited 4×
“We built upon this work, as well as other work that explores XAI or transparency features [10, 21, 31], to select and adapt features that could apply to the context of GenAI for code.”
BackgroundMethodInfluentialSelf-citationUnit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 290
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274 · cited 8×
“This question ofwhat do users need to understand about AI systems is core to the nascent field of Human-Centered Explainable AI (HCXAI) [21, 22, 52], which is a subset of the fields of human centered AI and human centered data science [6, 7, 25, 41, 65–67].”
BackgroundMethodInfluentialNovice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 1×
“Recent HCI works also explored the approach of “explainability through interaction" for generative models, by allowing users to interact with the input or guiding output generation process [59, 60, 86, 103].”
BackgroundInternational Journal of Engineering and Computer Science
(2017) · International Journal Of Engineering And Computer Science · doi:10.18535/ijecs · Cited by 266
Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation
Oda, Y., Fudaba, H., Neubig, G., Hata, H., Sakti, S., Toda, T., & Nakamura, S. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.36 · Cited by 250 · cited 1×
“One example is how existing work on automatic, machine translation between human natural languages [71, 72] was applied to code.”
BackgroundCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
The Pragmatic Turn in Explainable Artificial Intelligence (XAI)
Páez, A. (2019) · Minds and Machines · doi:10.1007/s11023-019-09502-w · Cited by 230
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · cited 2×
“Other GenAI models have been developed that implement other use cases, such as generating documentation for code [23], auto-completing code [15, 39], generating unit tests [93], and finding duplicate code [30].”
BackgroundMethodWeBuildAI: Participatory Framework for Algorithmic Governance
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359283 · Cited by 206
Explainability scenarios: towards scenario-based XAI design
Wolf, C. T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302317 · Cited by 172 · cited 1×
“SBD also allows us to explore XAI design without the constraint of current technical feasibility, as adopted by several prior XAI works [19, 102].”
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151 · cited 1×
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundSelf-citationA Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150 · cited 1×
“The application of modern NLP techniques to programming language can be traced back to the naturalness hypothesis [4, 17, 37], that software is a form of human communication.”
What Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148
Toolkit to support intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864353 · Cited by 143 · cited 7×
“Categories that appeared in Lim and Dey [53] but not Liao et al.”
BackgroundMethodInfluentialMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 1×
“For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9, 15, 33, 88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency.”
MethodPerfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Self-citationOperationalizing Human-Centered Perspectives in Explainable AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Mara, M., Streit, M., Wachter, S., Riener, A., & Riedl, M. O. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3441342 · Cited by 123
Self-citationHuman-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
Self-citationDhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119 · cited 1×
“…desire for an actionable or utility-oriented understanding to support their end goal of optimizing code generation and programming productivity [18, 50, 52], such as asking the Input, Output, How and How-to XAI questions (described in Section 4) to facilitate strategies to get better outputs…”
BackgroundRosson, M. B., & Carroll, J. M. (2009) · Human factors and ergonomics · doi:10.1201/9781420088892.ch8 · Cited by 117
Social transparency in networked information exchange: a theoretical framework
Stuart, H. C., Dabbish, L., Kiesler, S., Kinnaird, P., & Kang, R. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145275 · Cited by 100 · cited 1×
“[19], built on the concept of social transparency in social computing[90].”
BackgroundCodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92
A Human-Centered Agenda for Intelligible Machine Learning
Vaughan, J. W., & Wallach, H. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/12186.003.0014 · Cited by 92
The Sanction of Authority: Promoting Public Trust in AI
Knowles, B., & Richards, J. T. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445890 · Cited by 86 · cited 1×
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundDesigning Ground Truth and the Social Life of Labels
Muller, M., Wolf, C. T., Andres, J., Desmond, M., Joshi, N. N., Ashktorab, Z., Sharma, A., Brimijoin, K., Pan, Q., Duesterwald, E., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445402 · Cited by 83 · cited 1×
“In short, explainability needs should be understood and addressed with the ecological factors in mind, situated in an understanding of the behavioral and social contexts of the system [41, 70].”
BackgroundSelf-citationORES: Lowering Barriers with Participatory Machine Learning in Wikipedia
Halfaker, A., & Geiger, R. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415219 · Cited by 82
Towards A Process Model for Co-Creating AI Experiences
Subramonyam, H., Seifert, C. M., & Adar, E. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462012 · Cited by 77 · cited 1×
“This approach also echos a recent trend in AI design to utilize real data points as “data probes” to aid design ideation [91].”
MethodA Survey of Inductive Biases for Factorial Representation-LearningPreprint
Ridgeway, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.05299 · Cited by 76
Visualizing Uncertainty and Alternatives in Event Sequence Predictions
Guo, S., Du, F., Malik, S., Koh, E., Kim, S., Liu, Z., Kim, Y., Hongyuan Zha, B., & Cao, N. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300803 · Cited by 74 · cited 1×
“We built upon this work, as well as other work that explores XAI or transparency features [10, 21, 31], to select and adapt features that could apply to the context of GenAI for code.”
MethodHuman-Centered Approaches to Fair and Responsible AI
Lee, M. K., Grgić-Hlača, N., Tschantz, M. C., Binns, R., Weller, A., Carney, M., & Inkpen, K. M. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375158 · Cited by 72 · cited 1×
“The termHuman-centered AI has emerged inmany academic works and public discussions [22, 25, 45, 65, 85, 89].”
BackgroundMigrating code with statistical machine translation
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591072 · Cited by 67 · cited 1×
“One example is how existing work on automatic, machine translation between human natural languages [71, 72] was applied to code.”
BackgroundHuman-Centered Study of Data Science Work Practices
Muller, M. J., Feinberg, M., George, T., Jackson, S. J., John, B. E., Kery, M. B., & Passi, S. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299018 · Cited by 58 · cited 1×
“This question of what do users need to understand about AI systems is core to the nascent field of Human-Centered Explainable AI (HCXAI) [21, 22, 52], which is a subset of the fields of human centered AI and human centered data science [6, 7, 25, 41, 65–67].”
BackgroundSelf-citationUnsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57
Developing a Research Agenda for Human-Centered Data Science
Aragon, C., Hutto, C. J., Echenique, A., Fiore-Gartland, B., Huang, Y. K., Kim, J., Neff, G., Xing, W., & Bayer, J. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818052.2855518 · Cited by 53 · cited 1×
“This question of what do users need to understand about AI systems is core to the nascent field of Human-Centered Explainable AI (HCXAI) [21, 22, 52], which is a subset of the fields of human centered AI and human centered data science [6, 7, 25, 41, 65–67].”
BackgroundKuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44 · cited 1×
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42, 43] We also noticed that the explainability needs and user needs in general may differ for users with varied levels of programming skills.”
BackgroundExperiences with Improving the Transparency of AI Models and Services
Hind, M., Houde, S., Martino, J., Mojsilovic, A., Piorkowski, D., Richards, J., & Varshney, K. R. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3383051 · Cited by 44 · cited 1×
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundSelf-citation(2018) · arXiv (Cornell University) · Cited by 41
Mapping Out Human-Centered Data Science: Methods, Approaches, and Best Practices
Kogan, M., Halfaker, A., Guha, S., Aragón, C., Muller, M. J., & Geiger, R. S. (2019) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/3323994.3369898 · Cited by 41 · cited 2×
“This question of what do users need to understand about AI systems is core to the nascent field of Human-Centered Explainable AI (HCXAI) [21, 22, 52], which is a subset of the fields of human centered AI and human centered data science [6, 7, 25, 41, 65–67].”
BackgroundSelf-citationEvaluating the Interpretability of Generative Models by Interactive Reconstruction
Ross, A., Chen, N., Hang, E. Z., Glassman, E. L., & Doshi-Velez, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445296 · Cited by 40 · cited 4×
“Recent works by Goodfellow et al. [28] and Ross et al. [86] have explored developing more interpretable GenAI models that follow more human-understandable processes.”
BackgroundInfluentialGhosh, S., Liao, Q. V., Ramamurthy, K. N., Navrátil, J., Sattigeri, P., Varshney, K. R., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01410 · Cited by 39
Self-citationRaji, I. D., & Yang, J. (2019) · arXiv · doi:10.48550/arxiv.1912.06166 · Cited by 36
Introduction to Explainable AI
Liao, Q. V., Singh, M., Zhang, Y., & Bellamy, R. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3445016 · Cited by 34 · cited 1×
“• Attention visualization is a type of local explanation to answer the why question [51]: why did the model produce a particular output for a given input? For NLP tasks using deep neural networks, a common approach to explain a prediction is to utilize the attention mechanism, which provides a distribution over attend…”
BackgroundSelf-citationZhang, E., & Banović, N. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445714 · Cited by 34 · cited 2×
“Recent HCI works also explored the approach of “explainability through interaction" for generative models, by allowing users to interact with the input or guiding output generation process [59, 60, 86, 103].”
BackgroundA Methodology for Creating AI FactSheets
Richards, J., Piorkowski, D., Hind, M., Houde, S., & Mojsilović, A. (2020) · arXiv · doi:10.48550/arxiv.2006.13796 · Cited by 28
Self-citationKuttal, S. K., Myers, J., Gurka, S., Magar, D., Piorkowski, D., & Bellamy, R. (2020) · 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127276 · Cited by 27 · cited 1×
“We envision that a conversational agent interface could be a natural fit for AI assistant for code or co-programming tools, as explored in recent work [42, 43] We also noticed that the explainability needs and user needs in general may differ for users with varied levels of programming skills.”
BackgroundKurosu, M., & Hashizume, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35596-7 · Cited by 27
Participatory Machine Learning Using Community-Based System Dynamics.
Prabhakaran, V., & Martin, D. (2020) · PubMed Central · Cited by 26
Enabling Value Sensitive AI Systems through Participatory Design FictionsPreprint
Liao, Q. V., & Muller, M. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07381 · Cited by 22
Self-citationMuller, M. J., Aragon, C., Guha, S., Kogan, M., Neff, G., Seidelin, C., Shilton, K., & Tanweer, A. (2020) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3406865.3418584 · Cited by 20
Self-citationMachine Learning Model Cards Transparency Review : Using model card toolkit
Wadhwani, A., & Jain, P. (2020) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon50868.2020.9362382 · Cited by 19 · cited 1×
“But the AI did not • AI documentation is embodied by the recently-proposed concepts of FactSheets [8, 81], Model Cards [98], and About ML pages [78].”
BackgroundQuestion-Driven Design Process for Explainable AI User Experiences
Liao, Q. V., Pribić, M., Han, J., Miller, S., & Sow, D. (2021) · arXiv · doi:10.48550/arxiv.2104.03483 · Cited by 18 · cited 9×
“Liao et al. [50] provides a suggested mapping between prototypical user questions (e.g., why, performance, data, output) and XAI methods or features that can answer the questions, primarily in the context of discriminative AI for decision-support systems.”
BackgroundMethodInfluentialSelf-citationHAI-GEN 2021: 2nd Workshop on Human-AI Co-Creation with Generative Models
Geyer, W., Chilton, L. B., Weisz, J. D., & Maher, M. L. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397482.3450707 · Cited by 11 · cited 2×
“The term Human-centered AI has emerged in many academic works and public discussions [22, 25, 45, 65, 85, 89].”
BackgroundSelf-citationTalamadupula, K. (2021) · AI Matters · doi:10.1145/3465074.3465080 · Cited by 10 · cited 1×
“This hypothesis opened the door for applying NLP techniques previously used on human natural languages to source code, and recent work in this space is summarized by Talamadupula [92] and Allamanis et al. [4].”
MethodSelf-citationNew Initiative: The Naturalness of Software
Devanbu, P. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.190 · Cited by 9 · cited 1×
“The application of modern NLP techniques to programming language can be traced back to the naturalness hypothesis [4, 17, 37], that software is a form of human communication.”
Towards evaluating and eliciting high-quality documentation for intelligent systems
Piorkowski, D., González, D., Richards, J., & Houde, S. (2020) · arXiv · doi:10.48550/arxiv.2011.08774 · Cited by 5
Self-citationQuality Estimation & Interpretability for Code Translation
Agarwal, M., Talamadupula, K., Houde, S., Martinez, F., Muller, M., Richards, J., Ross, S., & Weisz, J. D. (2020) · arXiv · doi:10.48550/arxiv.2012.07581 · Cited by 3 · cited 1×
“[2], which demonstrates how to derive line-level confidences by aggregating token probabilities.”
BackgroundSelf-citationGeyer, W., Chilton, L. B., Kumar, R., & Kalai, A. T. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3379336.3379355 · Cited by 3
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · arXiv · doi:10.48550/arxiv.1908.04626
Towards A Process Model for Co-Creating AI Experiences
Subramonyam, H., Seifert, C., & Adar, E. (2021) · arXiv · doi:10.48550/arxiv.2104.07595
Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · arXiv · doi:10.48550/arxiv.2002.01092
Research works printed but not linked: 6
- Cecilia Aragon , Shion Guha , Marina Kogan , Michael Muller , and Gina Neff . 2022. Human-Centered Data Science: An Introduction . MIT Press , Cambridge, MA . Cecilia Aragon, Shion Guha, Marina Kogan, Michael Muller, and Gina Neff. 2022. Human-Centered Data Science: An Introduction. MIT Press, Cambridge, MA.
- Upol Ehsan and Mark Riedl . 2021 . Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022. Upol Ehsan and Mark Riedl. 2021. Explainability Pitfalls: Beyond Dark Patterns in Explainable AI - paper at HCAI@NeurIPS2021 workshop on human centered AI. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 19, 2022.
- Ryan Louie , Any Cohen , Cheng- Zhi Anna Huang , Michael Terry , and Carrie J Cai . 2020 . Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI. Ryan Louie, Any Cohen, Cheng-Zhi Anna Huang, Michael Terry, and Carrie J Cai. 2020. Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models.. In HAI-GEN+ user2agent@ IUI.
- Michael Muller , Plamen Angelov , Shion Guha , Marina Kogan , Gina Neff , Nuria Oliver , Manuel Gomez Rodriquez , and Adrian Weller . 2021. HCAI@NeurIPS2021 : Human Centered AI workshop at NeurIPS 2021 . https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022. Michael Muller, Plamen Angelov, Shion Guha, Marina Kogan, Gina Neff, Nuria Oliver, Manuel Gomez Rodriquez, and Adrian Weller. 2021. HCAI@NeurIPS2021: Human Centered AI workshop at NeurIPS 2021. https://sites.google.com/view/hcai-human-centered-ai-neurips/home Accessed January 17, 2022.
- John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). John T. Richards David Piorkowski M. Hind Stephanie Houde and Aleksandra Mojsilovi’c. 2020. A Methodology for Creating AI FactSheets. ArXiv abs/2006.13796(2020). · arXiv
- Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS. Karl Ridgeway and Michael C. Mozer. 2018. Learning Deep Disentangled Embeddings with the F-Statistic Loss. In NeurIPS.
Cited by
Generative AI and ChatGPT: applications, challenges, and AI-human collaboration
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2023.2233814 · Cited by 1,363
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Sun et al. [82] focus on users’ explainability needs for generative AI in three software engineering use cases: code generation based on natural language description (with Copilot), translation between different programming languages (with Transcoder), and code autocompletion (with Copilot).”
MethodGenerative artificial intelligence
Banh, L., & Strobel, G. (2023) · Electronic Markets · doi:10.1007/s12525-023-00680-1 · Cited by 579
“Programmers using applications such as GitHub Copilot are supported by automatically written chunks of code, ideas converted into actionable scripts, auto-completion functions, generated unit tests, duplicate code detection, and bug fixing (Sun et al., 2022).”
BackgroundMethodLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
A Comprehensive Review on Generative AI for Education
Mittal, U., Sai, S., Chamola, V., & Sangwan, D. (2024) · IEEE Access · doi:10.1109/access.2024.3468368 · Cited by 383
“This issue can be addressed by promoting the concept of explainable AI, which is based on the idea that the model should be able to explain each and everything that it produces [141].”
Background
Metadata
- Authors
- Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D.
- Affiliations
- University of Southern California (US)
- IBM (United States) (US)
- ORCID
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 212-228
- OpenAlex
- W4220962633
- Semantic Scholar
- 76ebf56d6ebb833d8c8c4124f7b2f15771a4997c
- Cited by
- 219
- References
- 78
- Citations by year
- 2026: 29 · 2025: 58 · 2024: 91 · 2023: 37 · 2022: 4
- FWCI
- 26.25
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- explainable AI, scenario-based design, natural language to code, program translation, human-centered AI design, software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.