The DOI matches its Crossref record; 86 of the 88 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Using an LLM to Help With Code Understanding
Open accessAbstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Overview
CoChat Summary
TL;DR
GILT is an in-IDE VS Code plugin that uses GPT-3.5-turbo to provide prompt-less and contextualized LLM support for code understanding. In a 32-participant user study, GILT improved task progress over web search but did not significantly improve completion time or measured code understanding.
Key findings
- Participants using GILT completed statistically significantly more sub-tasks than those using search engines: 0.47 more sub-tasks, 𝑝 < 0.01, controlling for experience and AI tool familiarity.
- GILT did not show significant differences in task completion time or code understanding quiz scores compared with search.
- Professionals completed 0.57 more sub-tasks with GILT support, a significant effect, whereas students did not experience significant gains.
- Usage averaged 15.34 GILT interactions per participant (range 5–23); Overview was most frequent (4.76 activations), followed by Prompt-context (4.12) and Prompt-followup (2.88). TAM scores for perceived usefulness/ease were higher for GILT than search (𝑝 < 0.001), and TLX showed significant differences in temporal demand and performance (𝑝 < 0.05).
Methods
- The authors built GILT, a VS Code plugin querying OpenAI’s GPT-3.5-turbo with four prompt-less buttons (Overview, API, Concept, Usage) and open-ended prompts automatically contextualized with selected code.
- They conducted a within-subjects user study with 32 analyzed participants (33 recruited, 1 excluded), using Bokeh data-visualization and Open3D 3D-rendering tasks, each with 4 sub-tasks and 20 minutes per task, comparing GILT against search engines on different counterbalanced tasks.
- They estimated quasi-Poisson models for task progress and understanding and linear regression for completion time, controlling for domain experience, programming experience, and AI tool familiarity; they also analyzed logs, prompts, and TAM/TLX survey responses.
Limitations
- The paper states threats from task/library selection, visible-output tasks, less common libraries, possible in-IDE versus search confounds, lab setting, time pressure, novelty effects, a small sample of 32 skewed toward academia, rapid LLM development, and standard statistical conclusion threats; they removed top 1% outliers and checked VIF < 2.5.
- The paper also notes that different libraries or larger codebases could produce different results, and that further real-world studies with larger, more diverse samples are needed.
- Evident from the text: the study used only two Python libraries and short timed tasks, relied on GPT-3.5-turbo at the time, and did not use think-aloud during tasks, which may limit generalization to everyday professional programming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 3 Not checked · 99 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30,31,34,44,48,57].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [30] 30. Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collo- cated Software Development Teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 344-353. https://doi.org/10.1109/ICSE.2007.45 source
- Not checked — no full text of the sourcecites [31] 31. Amy J. Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (11 2006), 971 -987. https://doi.org/10.1109/tse.2006.116 source
- Not checked — no full text of the sourcecites [34] 34. Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In Proceedings of the 28th international conference on Software engineering. 492-501. source
- Partly supportedcites [44] 44. Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 31:1-31:37. https://doi.org/10.1145/2622669 sourceExperimental — accuracy not yet measured
Section 6.4.5 (page unavailable) says that students search beyond course materials for help, including on StackOverflow. It does not establish that developers spend a significant amount of time searching or foraging, or that they spend time organizing and digesting what they find. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [48] 48. Andre N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178-1193. https://doi.org/10.1109/tse.2017.2656886 source
- Partly supportedcites [57] 57. David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret M. Burnett. 2016. Foraging and navigations, fundamentally: developers' predictions of value and cost. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE 2016), Seattle, WA, USA, November 13-18, 2016. ACM, 97-108. https://doi.org/10.1145/ 2950290.2950302 sourceExperimental — accuracy not yet measured
Our copy of Rahman et al. (2018) supports the searching/foraging half of the claim: the Abstract (p. 1) states 'Search is an integral part of a software development process. Developers often use search engines to look for information during development,' and the effort results (RQ2/RQ3, pp. 6-8; Conclusion, p. 10) report that developers spend substantial and significantly greater time, web visits and query edits on code search (median code task 2 min 53 s vs 1 min 35 s for non-code). However, the paper never addresses the 'organizing and digesting the information they find' component — its logs track browsing activity only, with no measure of post-retrieval comprehension, organization or digestion. The claim's magnitude is also framed absolutely ('a significant amount of time'), whereas this source's time figures are comparative between code and non-code tasks, not a share of developers' working time. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [24] 24. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, (ICSE Companion 2022), Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, 319-321. https://doi.org/10.1145/3510454.3522684 source
- Could not verifycites [79] 79. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productiv- ity assessment of neural code completion. In 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS@PLDI 2022), San Diego, CA, USA, 13 June 2022. ACM, 21-29. https://doi.org/10.1145/3520312.3534864 sourceExperimental — accuracy not yet measured
The Introduction describes neural tools that generate multi-line code suggestions and raises a possible cost of reviewing them, but it does not say developers may receive code they do not understand. Searches for understanding and comprehension of generated suggestions found no passage addressing that claim. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, professionals completed 0.57 more sub-tasks with GILT support compared to when they used search engines, whereas students did not experience significant gains.”
§ Using an LLM to Help With Code Understanding > 5.4 Additional Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not find any significant impact of participants' information processing style on their use of GILT.”
§ Using an LLM to Help With Code Understanding > Results.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, 83% of the failed tasks were also preceded by prompt-based interactions”
§ Using an LLM to Help With Code Understanding > 6.1 Usage of Features
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 110 claims and every source verdict.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 41
- The quote only partly states this claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“we analyzed the text of the prompts they wrote, by comparing the frequencies of bi-, tri-, and quadrigrams in the prompts.”
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“('make', 'bunny', 'sit', 'upright', 'chair') 3 0”
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Technology Acceptance Model: Past, Present, and Future
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01250 · Cited by 2,572 · cited 2×
“We based our questionnaire on the Technology Acceptance Model (TAM) [36], NASA Task Load Index (TLX) [21], and pre-and post-study questionnaires that were previously used in similar studies [63, 68].”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Johnston, R., Jones, K., & Manley, D. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0584-6 · Cited by 1,181
Open3D: A Modern Library for 3D Data Processing
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,124
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundSelf-citationVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Darley, W. K., & Smith, R. E. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673467 · Cited by 640 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Self-citationProceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2006) · doi:10.1145/1124772 · Cited by 517
Information Needs in Collocated Software Development Teams
Ko, A. J., DeLine, R., & Venolia, G. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.45 · Cited by 499
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
(2009) · doi:10.1145/1518701.1518944 · Cited by 447
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0943 · Cited by 384
What Makes APIs Hard to Learn? Answers from Developers
Robillard, M. P. (2009) · IEEE Software · doi:10.1109/ms.2009.193 · Cited by 382
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
A field study of API learning obstacles
Robillard, M. P., & DeLine, R. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9150-8 · Cited by 352 · cited 1×
“Researchers cataloged problems developers face when using documentation [9, 60, 61] and identified types of knowledge developers report to be critical [46, 60, 61, 70], which were taken into account when we designed GILT for our study.”
BackgroundExample-centric programming: integrating web search into the development environment
Brandt, J., Dontcheva, M., Weskamp, M., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753402 · Cited by 324
Asking and Answering Questions during a Programming Change Task
Sillito, J., Murphy, G. C., & De Volder, K. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.26 · Cited by 319 · cited 1×
“Therefore, researchers have studied to understand how developers learn and comprehend unfamiliar code and domains [30, 38, 51, 62], especially how they search for and acquire information, as developers need a variety of kinds of knowledge [43, 66, 68, 77].”
BackgroundGenderMag: A Method for Evaluating Software's Gender Inclusiveness
Burnett, M., Stumpf, S., Macbeth, J., Makri, S., Beckwith, L., Kwan, I., Peters, A., & Jernigan, W. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv046 · Cited by 293 · cited 2×
“%) was statistically significantly lower ( 𝜒 2 (1, 66), 𝑝 < .”
BackgroundDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Conditions of Learning in Novice Programmers
Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986) · Journal of Educational Computing Research · doi:10.2190/gujt-jcbj-q6qu-q9pl · Cited by 262
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
Self-citationA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
Self-citationProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195
Patterns of Knowledge in API Reference Documentation
Maalej, W., & Robillard, M. P. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.12 · Cited by 183 · cited 1×
“Therefore, researchers have studied to understand how developers learn and comprehend unfamiliar code and domains [30, 38, 51, 62], especially how they search for and acquire information, as developers need a variety of kinds of knowledge [43, 66, 68, 77].”
BackgroundOn the Comprehension of Program Comprehension
Maalej, W., Tiarks, R., Roehm, T., & Koschke, R. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2622669 · Cited by 180 · cited 2×
“” This behavior might be attributed to developers’ inclination to focus on task completion rather than comprehending the software, as reported in the literature [44].”
BackgroundResultTinkering and gender in end-user programmers' debugging
Beckwith, L., Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A., & Cook, C. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124808 · Cited by 176
What do developers search for on the web?
Xia, X., Bao, L., Lo, D., Kochhar, P. S., Hassan, A. E., & Xing, Z. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9514-4 · Cited by 174 · cited 3×
“(P5) Participants also found it extremely useful to prompt the LLM with just code context, as it allowed them to bypass the need to write proficient queries, a well-known challenge in search-based information seeking [32, 74].”
The Work Life of Developers: Activities, Switches and Perceived Productivity
Meyer, A. N., Barton, L. E., Murphy, G. C., Zimmermann, T., & Fritz, T. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2656886 · Cited by 170 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundAre code examples on an online Q&A forum reliable?: a study of API misuse on stack overflow
Zhang, T., Upadhyaya, G., Reinhardt, A., Rajan, H., & Kim, M. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180260 · Cited by 167
Chen, J.-C., & Huang, S.-J. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2008.12.036 · Cited by 161 · cited 1×
“Researchers cataloged problems developers face when using documentation [9, 60, 61] and identified types of knowledge developers report to be critical [46, 60, 61, 70], which were taken into account when we designed GILT for our study.”
BackgroundUddin, G., & Robillard, M. P. (2015) · IEEE Software · doi:10.1109/ms.2014.80 · Cited by 158 · cited 1×
“Researchers cataloged problems developers face when using documentation [9, 60, 61] and identified types of knowledge developers report to be critical [46, 60, 61, 70], which were taken into account when we designed GILT for our study.”
BackgroundExploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Trust in Generative AI among students: An Exploratory Study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Ragavan, S. S., & Alipour, M. A. (2023) · arXiv · doi:10.48550/arxiv.2310.04631 · Cited by 143
Self-citationTrust in Generative AI among Students: An exploratory study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Srinivasa Ragavan, S., & Alipour, M. A. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630842 · Cited by 143
Self-citationHow Programmers Debug, Revisited: An Information Foraging Theory Perspective
Lawrance, J., Bogart, C., Burnett, M., Bellamy, R., Rector, K., & Fleming, S. D. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.111 · Cited by 142 · cited 1×
“Particularly, lots of research was done on the information seeking strategies of developers, mostly in general software maintenance [18, 30, 35] or web search settings [7, 59].”
BackgroundHow do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C., & Inoue, K. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 129 · cited 1×
“Studies showing developers’ heavy reliance on Stack Overflow, despite its known limitations in accuracy and currency [73, 78], further emphasize the need for caution before widely adopting LLM-based tools in code development.”
Program comprehension as fact finding
LaToza, T. D., Garlan, D., Herbsleb, J. D., & Myers, B. A. (2007) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1287624.1287675 · Cited by 126 · cited 2×
“We chose these based on API learning theory [14, 33, 46, 68], expecting this may particularly assist novice programmers or those unfamiliar with the APIs/domains or the LLM, as writing efficient search queries or prompts can be difficult for novices [13, 15, 32].”
BackgroundMethodInfluentialSelf-citationIs GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 2×
“Additionally, we explore the pros and cons participants reported, and suggestions for improving the tool.”
BackgroundIs ChatGPT the Ultimate Programming Assistant -- How far is it?
Tian, H., Lu, W., Li, T. O., Tang, X., Cheung, S.-C., Klein, J., & Bissyandé, T. F. (2023) · arXiv · doi:10.48550/arxiv.2304.11938 · Cited by 114
What help do developers seek, when and how?
Li, H., Xing, Z., Peng, X., & Zhao, W. (2013) · 2013 20th Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671289 · Cited by 81 · cited 1×
“Therefore, researchers have studied to understand how developers learn and comprehend unfamiliar code and domains [30, 38, 51, 62], especially how they search for and acquire information, as developers need a variety of kinds of knowledge [43, 66, 68, 77].”
BackgroundApplication Programming Interface Documentation: What Do Software Developers Want?
Meng, M., Steinhardt, S., & Schubert, A. (2018) · Journal of Technical Writing and Communication · doi:10.1177/0047281617721853 · Cited by 78 · cited 3×
“During the pilot studies, we observed that participants approached the tasks differently depending on their familiarity with other LLM-based tools, and styles of information processing and learning as observed in many previous studies on software documentation and debugging [4, 46, 47].”
BackgroundMethodAsking and answering questions about unfamiliar APIs: An exploratory study
Duala-Ekoko, E., & Robillard, M. P. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227187 · Cited by 77 · cited 1×
“We chose these based on API learning theory [14, 33, 46, 68], expecting this may particularly assist novice programmers or those unfamiliar with the APIs/domains or the LLM, as writing efficient search queries or prompts can be difficult for novices [13, 15, 32].”
MethodContextualizing the information‐seeking behavior of software engineers
Freund, L. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23278 · Cited by 75 · cited 1×
“Particularly, lots of research was done on the information seeking strategies of developers, mostly in general software maintenance [18, 30, 35] or web search settings [7, 59].”
BackgroundPiloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 69
Evaluating how developers use general-purpose web-search for code retrieval
Rahman, M. M., Barson, J., Paul, S., Kayani, J., Lois, F. A., Quezada, S. F., Parnin, C., Stolee, K. T., & Ray, B. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196425 · Cited by 62 · cited 1×
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundHow Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow
Wang, S., Chen, T.-H., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2874470 · Cited by 50 · cited 1×
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundWhat is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47 · cited 1×
“Fewer studies have specifically explored the usefulness of LLM-based programming tools [2, 29, 40, 49, 63, 64, 71, 75, 79, 79] with actual users or their usage data, and many of these studies have focused on code generation tools like CoPilot [26].”
BackgroundHow developers use API documentation: an observation study
Meng, M., Steinhardt, S. B., & Schubert, A. (2019) · Communication Design Quarterly · doi:10.1145/3358931.3358937 · Cited by 43
Liu, M. X., Kittur, A., & Myers, B. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501968 · Cited by 43
Self-citationForaging and navigations, fundamentally: developers' predictions of value and cost
Piorkowski, D., Henley, A. Z., Nabi, T., Fleming, S. D., Scaffidi, C., & Burnett, M. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950302 · Cited by 38
A strategy-centric approach to the design of end-user debugging tools
Grigoreanu, V., Burnett, M., & Robertson, G. G. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753431 · Cited by 33 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundThe role of conceptual knowledge in API usability
Ko, A. J., & Riche, Y. (2011) · 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2011.6070395 · Cited by 30 · cited 4×
“(P5) Participants also found it extremely useful to prompt the LLM with just code context, as it allowed them to bypass the need to write proficient queries, a well-known challenge in search-based information seeking [32, 74].”
InfluentialEnabling Data-Driven API Design with Community Usage Data: A Need-Finding Study
Zhang, T., Hartmann, B., Kim, M., & Glassman, E. L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376382 · Cited by 25
A Theory of Robust API Knowledge
Thayer, K., Chasins, S. E., & Ko, A. J. (2021) · ACM Transactions on Computing Education · doi:10.1145/3444945 · Cited by 25
Analyzing Web Search Behavior for Software Engineering Tasks
Rao, N., Bansal, C., Zimmermann, T., Awadallah, A. H., & Nagappan, N. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378083 · Cited by 24 · cited 1×
“Particularly, lots of research was done on the information seeking strategies of developers, mostly in general software maintenance [18, 30, 35] or web search settings [7, 59].”
BackgroundGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
What’s (Not) Working in Programmer User Studies?
Davis, M. C., Aghayi, E., LaToza, T. D., Wang, X., Myers, B. A., & Sunshine, J. S. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3587157 · Cited by 22
Self-citationExempla gratis (E.G.): code examples for free
Barnaby, C., Sen, K., Zhang, T., Glassman, E. L., & Chandra, S. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417052 · Cited by 16
amAssist: In-IDE ambient search of online programming resources
Li, H., Zhao, X., Xing, Z., Bao, L., Peng, X., Gao, D., & Zhao, W. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081849 · Cited by 15
Improving API Knowledge Discovery with ML: A Case Study of Comparable API Methods
Nam, D., Myers, B. A., Vasilescu, B., & Hellendoorn, V. J. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00161 · Cited by 12 · cited 1×
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundSelf-citationTwo User Perspectives in Program Comprehension: End Users and Developer Users
Roehm, T. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.22 · Cited by 11 · cited 1×
“Therefore, researchers have studied to understand how developers learn and comprehend unfamiliar code and domains [30, 38, 51, 62], especially how they search for and acquire information, as developers need a variety of kinds of knowledge [43, 66, 68, 77].”
BackgroundThe Long Tail: Understanding the Discoverability of API Functionality
Horvath, A., Myers, B., Grover, S., Dong, S., Zhou, E., Voichick, F., Kery, M. B., Shinju, S., Nam, D., & Nagy, M. (2019) · 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2019.8818681 · Cited by 9 · cited 1×
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundSelf-citationUnderstanding Documentation Use Through Log Analysis: A Case Study of Four Cloud Services
Nam, D., Macvean, A., Myers, B. A., & Vasilescu, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642721 · Cited by 9
Self-citationHow to Measure Diversity Actionably in Technology
Hamid, M. M., Chatterjee, A., Guizani, M., et al. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_27 · Cited by 8
Artifacts for Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V. J., Vasilescu, B., & Myers, B. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10461384 · Cited by 1
Self-citationNam, D., Macvean, A., Myers, B. A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10817
Self-citationDo Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv
Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14699
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.03938
Artifacts for Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., et al. (2024) · Zenodo · doi:10.5281/zenodo.10461385
Self-citation
Research works printed but not linked: 2
- Victoria Clarke and Virginia Braun. 2013. Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. The psychologist 26, 2 (2013), 120--123.
- Tabnine. 2024. Tabnine: AI assistant for software developers. https://www.tabnine.com/ Retrieved: 2024-01-11.
Cited by
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 208
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 230
“Nam et al. [54] compared the code understanding ability of LLMs when using them to help developers understand the project.”
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 160
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodPrompt-based automation of building code information transformation for compliance checking
Yang, F., & Zhang, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105817 · Cited by 59
Metadata
- Authors
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745423
- Semantic Scholar
- f5ea85f67fc5ea9cce2c5345ed92740f7377cb2e
- Cited by
- 487
- References
- 61
- Citations by year
- 2026: 184 · 2025: 233 · 2024: 66
- FWCI
- 114.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, program comprehension, conversational interfaces, GPT-3.5-turbo, API documentation, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.