Using an LLM to Help With Code Understanding
Open accessThe DOI matches its Crossref record; 87 of the 89 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Abstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Overview
CoChat Summary
TL;DR
GILT is an in-IDE VS Code plugin that uses GPT-3.5-turbo to provide prompt-less and contextualized LLM support for code understanding. In a 32-participant user study, GILT improved task progress over web search but did not significantly improve completion time or measured code understanding.
Key findings
- Participants using GILT completed statistically significantly more sub-tasks than those using search engines: 0.47 more sub-tasks, 𝑝 < 0.01, controlling for experience and AI tool familiarity.
- GILT did not show significant differences in task completion time or code understanding quiz scores compared with search.
- Professionals completed 0.57 more sub-tasks with GILT support, a significant effect, whereas students did not experience significant gains.
- Usage averaged 15.34 GILT interactions per participant (range 5–23); Overview was most frequent (4.76 activations), followed by Prompt-context (4.12) and Prompt-followup (2.88). TAM scores for perceived usefulness/ease were higher for GILT than search (𝑝 < 0.001), and TLX showed significant differences in temporal demand and performance (𝑝 < 0.05).
Methods
- The authors built GILT, a VS Code plugin querying OpenAI’s GPT-3.5-turbo with four prompt-less buttons (Overview, API, Concept, Usage) and open-ended prompts automatically contextualized with selected code.
- They conducted a within-subjects user study with 32 analyzed participants (33 recruited, 1 excluded), using Bokeh data-visualization and Open3D 3D-rendering tasks, each with 4 sub-tasks and 20 minutes per task, comparing GILT against search engines on different counterbalanced tasks.
- They estimated quasi-Poisson models for task progress and understanding and linear regression for completion time, controlling for domain experience, programming experience, and AI tool familiarity; they also analyzed logs, prompts, and TAM/TLX survey responses.
Limitations
- The paper states threats from task/library selection, visible-output tasks, less common libraries, possible in-IDE versus search confounds, lab setting, time pressure, novelty effects, a small sample of 32 skewed toward academia, rapid LLM development, and standard statistical conclusion threats; they removed top 1% outliers and checked VIF < 2.5.
- The paper also notes that different libraries or larger codebases could produce different results, and that further real-world studies with larger, more diverse samples are needed.
- Evident from the text: the study used only two Python libraries and short timed tasks, relied on GPT-3.5-turbo at the time, and did not use think-aloud during tasks, which may limit generalization to everyday professional programming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 3 Not checked · 99 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30,31,34,44,48,57].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [30] 30. Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collo- cated Software Development Teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 344-353. https://doi.org/10.1109/ICSE.2007.45 source
- Not checked — no full text of the sourcecites [31] 31. Amy J. Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (11 2006), 971 -987. https://doi.org/10.1109/tse.2006.116 source
- Not checked — no full text of the sourcecites [34] 34. Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In Proceedings of the 28th international conference on Software engineering. 492-501. source
- Partly supportedcites [44] 44. Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 31:1-31:37. https://doi.org/10.1145/2622669 sourceExperimental — accuracy not yet measured
Section 6.4.5 (page unavailable) says that students search beyond course materials for help, including on StackOverflow. It does not establish that developers spend a significant amount of time searching or foraging, or that they spend time organizing and digesting what they find. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [48] 48. Andre N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178-1193. https://doi.org/10.1109/tse.2017.2656886 source
- Partly supportedcites [57] 57. David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret M. Burnett. 2016. Foraging and navigations, fundamentally: developers' predictions of value and cost. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE 2016), Seattle, WA, USA, November 13-18, 2016. ACM, 97-108. https://doi.org/10.1145/ 2950290.2950302 sourceExperimental — accuracy not yet measured
Our copy of Rahman et al. (2018) supports the searching/foraging half of the claim: the Abstract (p. 1) states 'Search is an integral part of a software development process. Developers often use search engines to look for information during development,' and the effort results (RQ2/RQ3, pp. 6-8; Conclusion, p. 10) report that developers spend substantial and significantly greater time, web visits and query edits on code search (median code task 2 min 53 s vs 1 min 35 s for non-code). However, the paper never addresses the 'organizing and digesting the information they find' component — its logs track browsing activity only, with no measure of post-retrieval comprehension, organization or digestion. The claim's magnitude is also framed absolutely ('a significant amount of time'), whereas this source's time figures are comparative between code and non-code tasks, not a share of developers' working time. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [24] 24. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, (ICSE Companion 2022), Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, 319-321. https://doi.org/10.1145/3510454.3522684 source
- Could not verifycites [79] 79. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productiv- ity assessment of neural code completion. In 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS@PLDI 2022), San Diego, CA, USA, 13 June 2022. ACM, 21-29. https://doi.org/10.1145/3520312.3534864 sourceExperimental — accuracy not yet measured
The Introduction describes neural tools that generate multi-line code suggestions and raises a possible cost of reviewing them, but it does not say developers may receive code they do not understand. Searches for understanding and comprehension of generated suggestions found no passage addressing that claim. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, professionals completed 0.57 more sub-tasks with GILT support compared to when they used search engines, whereas students did not experience significant gains.”
§ Using an LLM to Help With Code Understanding > 5.4 Additional Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not find any significant impact of participants' information processing style on their use of GILT.”
§ Using an LLM to Help With Code Understanding > Results.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, 83% of the failed tasks were also preceded by prompt-based interactions”
§ Using an LLM to Help With Code Understanding > 6.1 Usage of Features
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 110 claims and every source verdict.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 41
- The quote only partly states this claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“we analyzed the text of the prompts they wrote, by comparing the frequencies of bi-, tri-, and quadrigrams in the prompts.”
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“('make', 'bunny', 'sit', 'upright', 'chair') 3 0”
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Technology Acceptance Model: Past, Present, and Future
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01250 · Cited by 2,572 · cited 2×
“We based our questionnaire on the Technology Acceptance Model (TAM) [36], NASA Task Load Index (TLX) [21], and pre-and post-study questionnaires that were previously used in similar studies [63, 68].”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Johnston, R., Jones, K., & Manley, D. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0584-6 · Cited by 1,181
Open3D: A Modern Library for 3D Data Processing
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,124
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundSelf-citationVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Darley, W. K., & Smith, R. E. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673467 · Cited by 640 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Research works printed but not linked: 2
- 10. Victoria Clarke and Virginia Braun. 2013. Teaching thematic analysis: Overcom- ing challenges and developing strategies for effective learning. The psychologist 26, 2 (2013), 120-123.
- 67. Tabnine. 2024. Tabnine: AI assistant for software developers. https://www. tabnine.com/ Retrieved: 2024-01-11.
Cited by
Prather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 235
“Nam et al. [54] compared the code understanding ability of LLMs when using them to help developers understand the project.”
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 66
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodPrompt-Based Building Code Information Transformation Method for Automated Compliance Checking
Yang, F., & Zhang, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105817 · Cited by 90
Xie, L., Zheng, C., Xia, H., Qu, H., & Zhu-Tian, C. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676374 · Cited by 52
Formal requirements engineering and large language models: A two-way roadmap
Ferrari, A., & Spoletini, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107697 · Cited by 92
LLM-Based Agents for Tool Learning: A Survey
Xu, W., Huang, C., Gao, S., & Shang, S. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00296-9 · Cited by 74
“For code understanding, an IDE plugin [92] enhances code comprehension by integrating API interpreters directly within the IDE compiler.”
Zhao, X., Chen, B., Ji, M., Wang, X., Yan, Y., Zhang, J., Liu, S., Ye, M., & Lv, C. (2024) · Agriculture · doi:10.3390/agriculture14081359 · Cited by 44
Ivie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 67
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 90
Self-citationWhat Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 115
“Weconduct a randomized controlled user study (Section 5) with 30 prompting novices, comparing the prompts users write before and after training, and against a control group that received standard prompt engineering training.”
InfluentialExploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41
Kim, Y. W., Cha, M. C., Yoon, S. H., & Lee, S. C. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2375701 · Cited by 53
“Furthermore, AI-CAs are proficient at generating code snippets, rectifying or enhancing programs, summarizing code, and elucidating previously written code to facilitate developers’ comprehension of existing programs (Nam et al., 2024; Tian et al., 2023).”
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
“…converting natural language to code (e.g. [2, 16, 19, 56]), converting code to natural language documentation (e.g. [16, 31]) or explanations (e.g. [37]), and converting code to code, such as by translating it from one language to another (e.g. [2, 19, 46, 57]) or by creating unit tests (e.g.…”
To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36
Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
Cassano, F., Gouwar, J., Lucchetti, F., Schlesinger, C., Freeman, A., Anderson, C. J., Feldman, M. Q., Greenberg, M., Jangda, A., & Guha, A. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689735 · Cited by 149
“Moreover, Code LLMs are also emerging as a building block for research [Bareiß et al. 2022; Chen et al. 2023; First et al. 2023; Joshi et al. 2023; Lemieux et al. 2023; Murali et al. 2023; Nam et al. 2024; Phung et al. 2023; Ross et al. 2023; Schäfer et al. 2024; Xia et al. 2024].”
TOGLL: Correct and Strong Test Oracle Generation with LLMS
Hossain, S. B., & Dwyer, M. B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00098 · Cited by 110
“Large language models (LLMs) have shown promise in automating diverse software engineering tasks such as code completion [18], program comprehension [19], automated repair [20], and test generation [21]–[23].”
MEIC: Re-thinking RTL Debug Automation using LLMs
Xu, K., Sun, J., Hu, Y., Fang, X., Shan, W., Wang, X., & Jiang, Z. (2024) · doi:10.1145/3676536.3676801 · Cited by 101
“Similarly, for LLMs, this classification would aid in executing more accurate debugging processes as it would allow for a more structured reasoning [8, 26, 40, 45].”
Large language models for batteries
Zuo, W., Zheng, H., He, T., Vishwanath, V., Chan, M. K. Y., Stevens, R. L., Amine, K., & Xu, G. (2025) · Joule · doi:10.1016/j.joule.2025.102037 · Cited by 40
LearnMate: Enhancing Online Education with LLM-Powered Personalized Learning Plans and Support
Wang, X. J., Lee, C. P., & Mutlu, B. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719857 · Cited by 56
Kim, Y., Aamir, Z., Singh, M. K., Boorboor, S., Mueller, K. D., & Kaufman, A. E. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3549537 · Cited by 41
“InsightPilot [50] proposes a streamlined data exploration process that generates data insights reducing the effort needed to understand the data [13,57,89].”
Streamlining geoscience data analysis with an LLM-driven workflow
Zhang, J., Clairmont, C., Que, X., Li, W., Chen, W., Li, C.-H., & Ma, X.-G. (2025) · Applied Computing and Geosciences · doi:10.1016/j.acags.2024.100218 · Cited by 28
Haurogné, J., Basheer, N., & Islam, S. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100598 · Cited by 47
AutoIoT: Automated IoT Platform Using Large Language Models
Cheng, Y., Xu, M., Zhang, Y., Li, K., Wang, R., & Yang, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3523907 · Cited by 38
“Such capabilities have driven notable progress in natural language processing (NLP) [34], leading to diverse applications across sectors, including programming [35], code understanding [36], and 3-”
Fast State Restoration in LLM Serving with HCache
Gao, S., Chen, Y., & Shu, J. (2025) · Proceedings of the Twentieth European Conference on Computer Systems · doi:10.1145/3689031.3696072 · Cited by 24
Need Help? Designing Proactive AI Assistants for Programming
Chen, V., Zhu, A., Zhao, S., Mozannar, H., Sontag, D. A., & Talwalkar, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714002 · Cited by 105
“Concretely, we propose the following two design considerations: (1) Support efficient evaluation.”
LiSSA: Toward Generic Traceability Link Recovery Through Retrieval-Augmented Generation
Fuchß, D., Hey, T., Keim, J., Liu, H., Ewald, N., Thirolf, T., & Koziolek, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00186 · Cited by 52
“Furthermore, LLMs have already proven to be effective in a variety of software engineering tasks, including code generation [27]–[30], code summarization [31]–[33], or API recommendation [34].”
Harnessing code domain insights: Enhancing programming Knowledge Tracing with Large Language Models
Sun, X., Liu, Q., Zhang, K., Shen, S., Yang, L., & Li, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113396 · Cited by 31
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Self-citationAI and the advent of the cyborg behavioral scientist
Tomaino, G., Cooke, A. D. J., & Hoover, J. (2025) · Journal of Consumer Psychology · doi:10.1002/jcpy.1452 · Cited by 27
Automatic High-Level Test Case Generation using Large Language Models
Hasan, N. B., Islam, M. A., Khan, J. Y., Senjik, S., & Iqbal, A. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00105 · Cited by 28
“1) Motivation: Pre-trained LLMs have achieved significant success across various software engineering tasks such as code generation [24], [40], [50], code repair [41], [48], and code documentation [33], [35].”
Do LLMs consider security? an empirical study on responses to programming questions
Sajadi, A., Le, B., Nguyen, A., Damevski, K., & Chatterjee, P. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10658-6 · Cited by 50
“However, several studies show that developers struggle to create effective LLM prompts (Chopra et al., 2023; Nam et al., 2024).”
Zheng, W., Yang, J., Chen, J., He, J., Li, P., Sun, D., Chen, C., & Meng, X. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3534583 · Cited by 20
“Prior to the rise of deep learning, particularly before it was widely adopted in the remote sensing field [7], [8], [9], [10], [11], [12], [13], [14], [15], traditional methods like image differencing and change vector analysis were used for remote sensing change detection [16], [17].”
López‐Pernas, S., Misiejuk, K., Kaliisa, R., & Saqr, M. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3568599 · Cited by 21
“The importance of providing context to LLMs has been highlighted by previous literature as a crucial step for increasing the accuracy of the responses, e.g., [54].”
Detecting Misuse of Security APIs: A Systematic Review
Mousavi, Z., Islam, C., Babar, M. A., Abuadbba, A., & Moore, K. (2025) · ACM Computing Surveys · doi:10.1145/3735968 · Cited by 43
“This makes the analysis and identification of misuse significantly more complex.”
COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledge
Li, Y., Wu, Y., Liu, J., Jiang, Z., Chen, Z., Yu, G., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00234 · Cited by 54
“If docs are unavailable, we can generate standard docs using LLM-based methods [39], [56], [57].”
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 21
“Various domain-specific models have been also proposed to improve code understanding [16], code review [17], repair and refactoring [18], and parallel programming [19], as examples.”
Difficulty aware programming knowledge tracing via large language models
Yang, L., Sun, X., Li, H., Xu, R., & Wei, X. (2025) · Scientific Reports · doi:10.1038/s41598-025-96540-3 · Cited by 36
“Leveraging LLMs can help students understand code, thereby enhancing the learning experience 33 .”
From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis
Li, Z., Zhu, H., Lu, Z., Xiao, Z., & Yin, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713133 · Cited by 38
“…their exceptional capabilities across various applications to assist humans, including creative writing [47, 95, 99], software engineering [64, 65], and generative design [28], which has sparked significant interest within the HCI community to investigate the interaction between humans…”
An AI-driven workflow for the accelerated optimization of cell-free protein synthesis
Khalil, M. M., Elsawah, A., Hoang, A. N., Faulon, J., Panthu, B., & Hérisson, J. (2025) · iScience · doi:10.1016/j.isci.2025.113599 · Cited by 16
“Beyond the code itself, LLMs can be also used to understand already written code 34 or generate high-quality test sets.”
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“While our study hinted to the positive impact of AI tools on more senior developers, other research has found contradictory effects [19], [52].”
InfluentialSelf-citationA Review of Prompt Engineering Techniques for Large Language Models
Naser, M. Z. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2607553 · Cited by 25
“These structural nuances may appear trivial, but they have effectively reduced logical confusion and guided the model’s internal generative process (Nam et al., 2024).”
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Abdullin, A., Derakhshanfar, P., & Panichella, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989033 · Cited by 42
“Large Language Models (LLMs) offer new possibilities in software engineering, showing effectiveness in tasks like code completion [22], code understanding [37], and code generation [59].”
Large language model-enhanced reinforcement learning for generic bus holding control strategies
Yu, J., Wang, Y., & Ma, W. (2025) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2025.104142 · Cited by 21
AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code
Fathallah, N., Hernández, D., & Staab, S. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746360 · Cited by 32
On the Effectiveness of Large Language Models for GitHub Workflows
Zhang, X., Muralee, S., Cherupattamoolayil, S., & Machiry, A. (2024) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3664476.3664497 · Cited by 37
“LLMs [26, 32] are igniting a revolution in the heart of the software development realm, automating various software engineering tasks such as coding [40], crafting test cases [51], and enriching code with documentation [28].”
BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
Cheng, R., Barik, T., Leung, A., Hohman, F., & Nichols, J. W. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00012 · Cited by 51
“Therefore, systems such as Tutorons [23] and Mallard [33] provide in-situ explanations and demonstrations within the webpage a user is working on, and recent advancements in programming support tools often feature inline assistance within the IDEs [25], [24].”
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
“Students, by contrast, did not experience significant benefits from using the LLM-based tool (Nam et al. 2024).”
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
“In response to this, a research team proposed and scoped an in-IDE tool for generating code explanations [68].”
InfluentialGenerating SysML Behavior Models via Large Language Models: an Empirical Study
Wang, Y., Ge, N., Liu, J., Cao, Z., Chen, Z., & Hu, C. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755926 · Cited by 22
Carbon Footprint Evaluation of Code Generation through LLM as a Service
Vartziotis, T., Schmidt, M., Dasoulas, G., Dellatolas, I., Attademo, S., Le, V. D., Wiechmann, A., Hoffmann, T., Keckeisen, M., & Kotsopoulos, S. D. (2024) · Proceedings · doi:10.1007/978-3-658-45010-6_15 · Cited by 14
Large Language Models for In-File Vulnerability Localization Can Be “Lost in the End”
Sovrano, F., Bauer, A., & Bacchelli, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715758 · Cited by 21
“…and University of Zurich, Department of Informatics, Zurich, Switzerland, sovrano@collegium.ethz.ch; Adam Bauer, University of Zurich, Zurich, Switzerland, adam.bauer@uzh.ch; Alberto Bacchelli, University of Zurich, Department of Informatics, Zurich, Switzerland, bacchelli@ifi.uzh.ch.…”
Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695000 · Cited by 10
“Error Message”
STRUT: Structured Seed Case Guided Unit Test Generation for C Programs using LLMs
Liu, J., Chao, L., Chen, R., Li, S., Gu, B., & Yang, M. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728970 · Cited by 15
“With the rise of LLMs, their excellent performance in tasks such as code comprehension[29] and code generation[21] has been widely recognized.”
Balancing performance and cost of LLMs in a multi-agent framework for BIM data retrieval
Liu, D., Zhou, X., & Li, Y. (2025) · Architectural Engineering and Design Management · doi:10.1080/17452007.2025.2456768 · Cited by 10
“This signi fi cantly improves retrieval accuracy (Nam, Macvean, Hellendoorn, Vasilescu, & Myers, 2024), which is the degree to which the retrieved BIM data matches the user ’ s query intent correctly and relevantly.”
A Critical Approach to ChatGPT: An Experience in SQL Learning
Farinetti, L., & Cagliero, L. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701932 · Cited by 23
“In Computer Science LLMs are known to be capable of solving complex tasks such as coding, software design, and technical document understanding [22].”
NumScout: Unveiling Numerical Defects in Smart Contracts Using LLM-Pruning Symbolic Execution
Chen, J., Shao, Z., Yang, S., Shen, Y., Wang, Y., Chen, T., Shan, Z., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3555622 · Cited by 29
“For example, they are used for analyzing inconsistencies in code comments [33].”
Large Language Model in Materials Science: Roles, Challenges, and Strategic Outlook
Zhang, J., Chen, X., Ye, X., Yang, Y., & Ai, B. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500085 · Cited by 23
Najjar, A. A., Ashqar, H. I., Darwish, O., & Hammad, E. M. (2025) · Information · doi:10.3390/info16090767 · Cited by 33
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
Xu, C., Chen, S., Wu, J., Cheung, S.-C., Terragni, V., Zhu, H., & Cao, J. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3696020 · Cited by 9
“Firstly , we observe that directly providing LLMs with contextual information only results in around 50% generalizable transformations (Section 4.5).”
InfluentialHe, C., Tian, Y., & Lu, Z. (2024) · Journal of Membrane Computing · doi:10.1007/s41965-024-00172-x · Cited by 15
“For example, an insightful understanding of the EAs can be realized by comparingthesourcecodes, as LLMs aregoodat codeunder-standing [157].”
Matzakos, N., & Moundridou, M. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70049 · Cited by 18
“For instance, in programming, students often turn to an LLM for getting help with command syntax, generating complete code, or debugging, while confirming the code's functionality using a specialized IDE [43, 44].”
Leveraging LLMs for Legacy Code Modernization: Evaluation of LLM-Generated Documentation
Diggs, C., Doyle, M., Madan, A., Scott, E. O., Escamilla, E., Zimmer, J., Nekoo, N., Ursino, P., Bartholf, M., Robin, Z., et al. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00027 · Cited by 43
“However, it is notable that performance varies across languages [18] and existing studies assess LLM performance on simple functions and modern languages (e.g. JavaScript, Java, Python) [18]–[21].”
CodingGenie: A Proactive LLM-Powered Programming Assistant
Zhao, S., Zhu, A., Mozannar, H., Sontag, D. A., Talwalkar, A., & Chen, V. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728603 · Cited by 32
“LLMs are increasingly integrated within various tools in the software development life cycle; these tools can provide code improvements [9, 38], code explanations [29, 40], brainstorming ideas [12], unit testing [2, 19], bug fixes and solving project issues [16, 18, 41], and syntax hints [28].”
Fault Localization from the Semantic Code Search Perspective
Qin, Y., Wang, S., Lei, Y., Zhang, Z., Lin, B., Peng, X., Ma, J., Chen, L., & Mao, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3757915 · Cited by 15
“In particular, we first construct a software knowledge base for the target project by amalgamating static analysis details and dynamic method call graphs.”
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
“Some focused on finding interaction patterns [32, 148, 176] while others focused more on how productivity is impacted by the use of models [101, 131], whether code produced when using AI code generators is less secure than when not [162], and how students use code explanations generated by”
BackgroundFalcão, F. D. S., & Canedo, E. D. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701675 · Cited by 13
Tang, X., Wong, S., Huynh, M., He, Z., Yang, Y., & Chen, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720203 · Cited by 22
“Recent AI-based approaches facilitate feed-back creation for code understanding and auto generation [9, 19], yet studies show these methods can be only about 50% accurate in assessing the correct mistakes [8], and students frequently find the feedback confusing or in need of manual validation [21].”
Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects
Nunes, H., Figueiredo, E., Rocha, L., Nadi, S., Ferreira, F., & Esteves, G. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00069 · Cited by 31
“Despite these initial results, it is essential to understand the effectiveness of LLMs in addressing issues within real-world software projects [17, 18].”
Enhancing Vulnerability Detection via Inter-procedural Semantic Completion
Wu, B., Liu, C., Li, Z., Cao, Y., Sun, J., & Lin, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728912 · Cited by 12
“Since LLMs have demonstrated inspiring performance in code understanding and summarization [13, 42], we utilize LLMs to summarize the called functions by designing appropriate prompts with various prompting strategies to capture essential behaviors related to vulnerabilities within the called…”
Steenhoek, B. J., Sivaraman, K., Gonzalez, R. S., Mohylevskyy, Y., Moghaddam, R. Z., & Le, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00126 · Cited by 28
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2026) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 31
“The complete search strings used for ACM and IEEE are shown in Table 1, while the search strings for ScienceDirect are shown in Table 2.”
InfluentialIntegrating Generative AI in Software Engineering Education: Practical Strategies
Li, Y., Keung, J., & Ma, X. (2024) · 2024 International Symposium on Educational Technology (ISET) · doi:10.1109/iset61814.2024.00019 · Cited by 8
“AI models have the capability to analyze code syntax, logic, and patterns, enabling them to identify potential bugs and propose relevant debugging solutions [2, 17].”
BackgroundCan Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8
“Therefore, similar to the work of Nam et al. [47], we propose that vendors of IDE-integrated code comprehension tools should offer multiple code-related aspects that tool users can explore during the comprehension process.”
What You Need is what You Get: Theory of Mind for an LLM-Based Code Understanding Assistant
Richards, J., & Wessel, M. S. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00070 · Cited by 29
“However, the extent to which users benefit from these tools depends on their background knowledge and experience levels [3], [4].”
InfluentialPrompt engineering and its implications on the energy consumption of Large Language Models
Rubei, R., Moussaid, A., Sipio, C. D., & Ruscio, D. D. (2025) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1109/greens66463.2025.00014 · Cited by 33
Rag and Rau: A Survey on Retrieval-Augmented Language Model in Natural Language ProcessingPreprint
Hu, Y., & Lu, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5015182 · Cited by 18
Software System Testing Assisted by Large Language Models: An Exploratory Study
Augusto, C., Morán, J., Bertolino, A., de la Riva, C., & Tuya, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80889-0_17 · Cited by 15
“Code explainability using LLMs has been addressed through the explainability of the code itself [33], or the explainability of failures, debugging [34], generating documentation [35], and reports [9], [10].”
LLM-enhanced evolutionary test generation for untyped languages
Yang, R., Xu, X., & Wang, R. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00496-7 · Cited by 18
“Nam et al. (2024) developed an LLM-based conversational interface within an IDE, using four high-level queries to access the OpenAI GPT-3.”
TypeForge: Synthesizing and Selecting Best-Fit Composite Data Types for Stripped Binaries
Wang, Y., Liang, R., Li, Y., Hu, P., Chen, K., & Zhang, B. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00193 · Cited by 19
“Large Language Models (LLMs), such as GPTs [29] and LLaMA [36], have demonstrated impressive results in various code-related tasks, including code understanding [37], [38] and code generation [39], [40].”
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 18
Can LLMs Generate User Stories and Assess Their Quality?
Quattrocchi, G., Pasquale, L., Spoletini, P., & Baresi, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3670612 · Cited by 20
“LLMs are pre-trained on an extensive number of natural language and source code documents and have been applied successfully to automate various software engineering tasks [10], such as code generation [11], [12], program repair [13]–[15], and comprehension [16].”
Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning
Liu, M. X., Petridis, S., Tsai, V., Fiannaca, A. J., Olwal, A., Terry, M. A., & Cai, C. J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712085 · Cited by 11
“However, articulating clear and complete requirements is a known challenge [19, 42, 43, 46, 52, 53], even for experts who need multiple iterations to refine them [68].”
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 50
Deception at Scale: Deceptive Designs in 1K LLM-Generated E-Commerce Components
Chen, Z., Shen, J., Luna, Zhang, H., & Vaccaro, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791063 · Cited by 7
“The membership sign-up component collects customer names and email without explaining how it will be used or protected ( interface interference in Gray’s taxonomy [29]).”
AutoPAC: Exploring LLMs for Automating Policy to Code Conversion in Business Organizations
Chowdhary, N., Dutta, T., Chattopadhyay, S., & Chakraborty, S. (2025) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets63942.2025.10885751 · Cited by 13
“The difficulty in understanding the generated code led to the use of LLMs for code explanation [11], [12] and code understanding [13], [14].”
Gau, M., Kretzer, F., Maedche, A., & Brocke, J. V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93976-1_2 · Cited by 10
Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 11
“Recent advancements in software engineering show that large language models (LLMs) perform tasks such as code generation, summarization, understanding, and review effectively [13]–[16].”
Using Generative AI to Support UX Design Students in Web Development Courses
Buendía, F., & Piris-Ruano, J. (2025) · Applied Sciences · doi:10.3390/app15137389 · Cited by 12
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 24
“Beyond code generation, GenAI tools have been explored in various other software engineering domains, including program repair [7], code comprehension [19], non-coding activities [20], prompt engineering [21], productivity [22], SE education [23] [24], and the software development process (SDP) [2].”
An LLM-Enabled Multi-Agent Autonomous Mechatronics Design Framework
Wang, Z., Lo, F. P.-W., Chen, Q., Zhang, Y., Chen, L., Chen, X., Yu, Z., Thompson, A. J., Yeatman, E. M., & Lo, B. P. L. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00404 · Cited by 11
Zhao, X., Zhang, C., Guo, P., Liu, W., Chen, L., Zhao, C., & Huang, S. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00382 · Cited by 16
“Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. anomaly predictions and generate rationales, improving the transparency and trustworthiness of VAD [32, 35].”
Decentralized Multi-Agent System with Trust-Aware Communication
Ding, Y., Twabi, A., Yu, J., Zhang, L., Kondo, T., & Satō, H. (2025) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa67752.2025.00198 · Cited by 17
“LLM-based multi-agent systems have demonstrated effectiveness across diverse domains, including software engineering [20], [21] and medical science [22], [23], showcasing their potential for tackling complex, interdisciplinary tasks that require long-term planning and specialized knowledge [7],…”
ALMAS: an Autonomous LLM-based Multi-Agent Software Engineering Framework
Tawosi, V., Ramani, K., Alamir, S., & Liu, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00059 · Cited by 17
“In software engineering, Large Language Models (LLMs) are applied to a diverse array of tasks, including code summarization and comprehension [7]–[10], program synthesis [11], [12], code translation [13], [14], automated program repair [15]–[17], and test generation [18], [19].”
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 25
“Many users appreciate not having to rely on external searches [58], though some criticize interface layout, history management, and dialog interaction.”
Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild
Kumar, A., Bajpai, Y., Gulwani, S., Soares, G., & Murphy-Hill, E. R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00043 · Cited by 30
“We asked questions on prior experience to explore how this may be related to participants’ use of the tool [26].”
Aguiar, L., Paixão, M. H. E., do Carmo, R. A. F., Soares, E. L. C., Leal, A., Freitas, M., & Gama, E. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690755 · Cited by 10
Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results?
Arora, C., Sayeed, A. I., Licorish, S. A., Wang, F., & Treude, C. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00039 · Cited by 62
“However, code-related research on LLMs has long been a focus for software engineering researchers [23], [24].”
Metadata
- Authors
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745423
- Semantic Scholar
- f5ea85f67fc5ea9cce2c5345ed92740f7377cb2e
- Cited by
- 487
- References
- 61
- Citations by year
- 2026: 184 · 2025: 233 · 2024: 66
- FWCI
- 114.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, program comprehension, conversational interfaces, GPT-3.5-turbo, API documentation, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.