The DOI matches its Crossref record; 86 of the 88 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Using an LLM to Help With Code Understanding
Open accessAbstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Overview
CoChat Summary
TL;DR
GILT is an in-IDE VS Code plugin that uses GPT-3.5-turbo to provide prompt-less and contextualized LLM support for code understanding. In a 32-participant user study, GILT improved task progress over web search but did not significantly improve completion time or measured code understanding.
Key findings
- Participants using GILT completed statistically significantly more sub-tasks than those using search engines: 0.47 more sub-tasks, 𝑝 < 0.01, controlling for experience and AI tool familiarity.
- GILT did not show significant differences in task completion time or code understanding quiz scores compared with search.
- Professionals completed 0.57 more sub-tasks with GILT support, a significant effect, whereas students did not experience significant gains.
- Usage averaged 15.34 GILT interactions per participant (range 5–23); Overview was most frequent (4.76 activations), followed by Prompt-context (4.12) and Prompt-followup (2.88). TAM scores for perceived usefulness/ease were higher for GILT than search (𝑝 < 0.001), and TLX showed significant differences in temporal demand and performance (𝑝 < 0.05).
Methods
- The authors built GILT, a VS Code plugin querying OpenAI’s GPT-3.5-turbo with four prompt-less buttons (Overview, API, Concept, Usage) and open-ended prompts automatically contextualized with selected code.
- They conducted a within-subjects user study with 32 analyzed participants (33 recruited, 1 excluded), using Bokeh data-visualization and Open3D 3D-rendering tasks, each with 4 sub-tasks and 20 minutes per task, comparing GILT against search engines on different counterbalanced tasks.
- They estimated quasi-Poisson models for task progress and understanding and linear regression for completion time, controlling for domain experience, programming experience, and AI tool familiarity; they also analyzed logs, prompts, and TAM/TLX survey responses.
Limitations
- The paper states threats from task/library selection, visible-output tasks, less common libraries, possible in-IDE versus search confounds, lab setting, time pressure, novelty effects, a small sample of 32 skewed toward academia, rapid LLM development, and standard statistical conclusion threats; they removed top 1% outliers and checked VIF < 2.5.
- The paper also notes that different libraries or larger codebases could produce different results, and that further real-world studies with larger, more diverse samples are needed.
- Evident from the text: the study used only two Python libraries and short timed tasks, relied on GPT-3.5-turbo at the time, and did not use think-aloud during tasks, which may limit generalization to everyday professional programming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 3 Not checked · 99 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30,31,34,44,48,57].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [30] 30. Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collo- cated Software Development Teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 344-353. https://doi.org/10.1109/ICSE.2007.45 source
- Not checked — no full text of the sourcecites [31] 31. Amy J. Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (11 2006), 971 -987. https://doi.org/10.1109/tse.2006.116 source
- Not checked — no full text of the sourcecites [34] 34. Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In Proceedings of the 28th international conference on Software engineering. 492-501. source
- Partly supportedcites [44] 44. Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 31:1-31:37. https://doi.org/10.1145/2622669 sourceExperimental — accuracy not yet measured
Section 6.4.5 (page unavailable) says that students search beyond course materials for help, including on StackOverflow. It does not establish that developers spend a significant amount of time searching or foraging, or that they spend time organizing and digesting what they find. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [48] 48. Andre N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178-1193. https://doi.org/10.1109/tse.2017.2656886 source
- Partly supportedcites [57] 57. David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret M. Burnett. 2016. Foraging and navigations, fundamentally: developers' predictions of value and cost. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE 2016), Seattle, WA, USA, November 13-18, 2016. ACM, 97-108. https://doi.org/10.1145/ 2950290.2950302 sourceExperimental — accuracy not yet measured
Our copy of Rahman et al. (2018) supports the searching/foraging half of the claim: the Abstract (p. 1) states 'Search is an integral part of a software development process. Developers often use search engines to look for information during development,' and the effort results (RQ2/RQ3, pp. 6-8; Conclusion, p. 10) report that developers spend substantial and significantly greater time, web visits and query edits on code search (median code task 2 min 53 s vs 1 min 35 s for non-code). However, the paper never addresses the 'organizing and digesting the information they find' component — its logs track browsing activity only, with no measure of post-retrieval comprehension, organization or digestion. The claim's magnitude is also framed absolutely ('a significant amount of time'), whereas this source's time figures are comparative between code and non-code tasks, not a share of developers' working time. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [24] 24. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, (ICSE Companion 2022), Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, 319-321. https://doi.org/10.1145/3510454.3522684 source
- Could not verifycites [79] 79. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productiv- ity assessment of neural code completion. In 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS@PLDI 2022), San Diego, CA, USA, 13 June 2022. ACM, 21-29. https://doi.org/10.1145/3520312.3534864 sourceExperimental — accuracy not yet measured
The Introduction describes neural tools that generate multi-line code suggestions and raises a possible cost of reviewing them, but it does not say developers may receive code they do not understand. Searches for understanding and comprehension of generated suggestions found no passage addressing that claim. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, professionals completed 0.57 more sub-tasks with GILT support compared to when they used search engines, whereas students did not experience significant gains.”
§ Using an LLM to Help With Code Understanding > 5.4 Additional Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not find any significant impact of participants' information processing style on their use of GILT.”
§ Using an LLM to Help With Code Understanding > Results.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, 83% of the failed tasks were also preceded by prompt-based interactions”
§ Using an LLM to Help With Code Understanding > 6.1 Usage of Features
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 110 claims and every source verdict.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 41
- The quote only partly states this claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“we analyzed the text of the prompts they wrote, by comparing the frequencies of bi-, tri-, and quadrigrams in the prompts.”
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“('make', 'bunny', 'sit', 'upright', 'chair') 3 0”
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Technology Acceptance Model: Past, Present, and Future
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01250 · Cited by 2,572 · cited 2×
“We based our questionnaire on the Technology Acceptance Model (TAM) [36], NASA Task Load Index (TLX) [21], and pre-and post-study questionnaires that were previously used in similar studies [63, 68].”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Johnston, R., Jones, K., & Manley, D. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0584-6 · Cited by 1,181
Open3D: A Modern Library for 3D Data Processing
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,124
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundSelf-citationVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Darley, W. K., & Smith, R. E. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673467 · Cited by 640 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Research works printed but not linked: 2
- Victoria Clarke and Virginia Braun. 2013. Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. The psychologist 26, 2 (2013), 120--123.
- Tabnine. 2024. Tabnine: AI assistant for software developers. https://www.tabnine.com/ Retrieved: 2024-01-11.
Cited by
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 208
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 230
“Nam et al. [54] compared the code understanding ability of LLMs when using them to help developers understand the project.”
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 160
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodPrompt-based automation of building code information transformation for compliance checking
Yang, F., & Zhang, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105817 · Cited by 59
Metadata
- Authors
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745423
- Semantic Scholar
- f5ea85f67fc5ea9cce2c5345ed92740f7377cb2e
- Cited by
- 487
- References
- 61
- Citations by year
- 2026: 184 · 2025: 233 · 2024: 66
- FWCI
- 114.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, program comprehension, conversational interfaces, GPT-3.5-turbo, API documentation, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using an LLM to Help With Code Understanding. https://app.cochat.ai/papers/10.1145/3597503.3639187
- The paper
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024). Using an LLM to Help With Code Understanding. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. https://doi.org/10.1145/3597503.3639187
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.