Using an LLM to Help With Code Understanding
Open accessThe DOI matches its Crossref record; 87 of the 89 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Abstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Overview
CoChat Summary
TL;DR
GILT is an in-IDE VS Code plugin that uses GPT-3.5-turbo to provide prompt-less and contextualized LLM support for code understanding. In a 32-participant user study, GILT improved task progress over web search but did not significantly improve completion time or measured code understanding.
Key findings
- Participants using GILT completed statistically significantly more sub-tasks than those using search engines: 0.47 more sub-tasks, 𝑝 < 0.01, controlling for experience and AI tool familiarity.
- GILT did not show significant differences in task completion time or code understanding quiz scores compared with search.
- Professionals completed 0.57 more sub-tasks with GILT support, a significant effect, whereas students did not experience significant gains.
- Usage averaged 15.34 GILT interactions per participant (range 5–23); Overview was most frequent (4.76 activations), followed by Prompt-context (4.12) and Prompt-followup (2.88). TAM scores for perceived usefulness/ease were higher for GILT than search (𝑝 < 0.001), and TLX showed significant differences in temporal demand and performance (𝑝 < 0.05).
Methods
- The authors built GILT, a VS Code plugin querying OpenAI’s GPT-3.5-turbo with four prompt-less buttons (Overview, API, Concept, Usage) and open-ended prompts automatically contextualized with selected code.
- They conducted a within-subjects user study with 32 analyzed participants (33 recruited, 1 excluded), using Bokeh data-visualization and Open3D 3D-rendering tasks, each with 4 sub-tasks and 20 minutes per task, comparing GILT against search engines on different counterbalanced tasks.
- They estimated quasi-Poisson models for task progress and understanding and linear regression for completion time, controlling for domain experience, programming experience, and AI tool familiarity; they also analyzed logs, prompts, and TAM/TLX survey responses.
Limitations
- The paper states threats from task/library selection, visible-output tasks, less common libraries, possible in-IDE versus search confounds, lab setting, time pressure, novelty effects, a small sample of 32 skewed toward academia, rapid LLM development, and standard statistical conclusion threats; they removed top 1% outliers and checked VIF < 2.5.
- The paper also notes that different libraries or larger codebases could produce different results, and that further real-world studies with larger, more diverse samples are needed.
- Evident from the text: the study used only two Python libraries and short timed tasks, relied on GPT-3.5-turbo at the time, and did not use think-aloud during tasks, which may limit generalization to everyday professional programming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 3 Not checked · 99 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30,31,34,44,48,57].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [30] 30. Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collo- cated Software Development Teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 344-353. https://doi.org/10.1109/ICSE.2007.45 source
- Not checked — no full text of the sourcecites [31] 31. Amy J. Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (11 2006), 971 -987. https://doi.org/10.1109/tse.2006.116 source
- Not checked — no full text of the sourcecites [34] 34. Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In Proceedings of the 28th international conference on Software engineering. 492-501. source
- Partly supportedcites [44] 44. Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 31:1-31:37. https://doi.org/10.1145/2622669 sourceExperimental — accuracy not yet measured
Section 6.4.5 (page unavailable) says that students search beyond course materials for help, including on StackOverflow. It does not establish that developers spend a significant amount of time searching or foraging, or that they spend time organizing and digesting what they find. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [48] 48. Andre N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178-1193. https://doi.org/10.1109/tse.2017.2656886 source
- Partly supportedcites [57] 57. David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret M. Burnett. 2016. Foraging and navigations, fundamentally: developers' predictions of value and cost. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE 2016), Seattle, WA, USA, November 13-18, 2016. ACM, 97-108. https://doi.org/10.1145/ 2950290.2950302 sourceExperimental — accuracy not yet measured
Our copy of Rahman et al. (2018) supports the searching/foraging half of the claim: the Abstract (p. 1) states 'Search is an integral part of a software development process. Developers often use search engines to look for information during development,' and the effort results (RQ2/RQ3, pp. 6-8; Conclusion, p. 10) report that developers spend substantial and significantly greater time, web visits and query edits on code search (median code task 2 min 53 s vs 1 min 35 s for non-code). However, the paper never addresses the 'organizing and digesting the information they find' component — its logs track browsing activity only, with no measure of post-retrieval comprehension, organization or digestion. The claim's magnitude is also framed absolutely ('a significant amount of time'), whereas this source's time figures are comparative between code and non-code tasks, not a share of developers' working time. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [24] 24. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, (ICSE Companion 2022), Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, 319-321. https://doi.org/10.1145/3510454.3522684 source
- Could not verifycites [79] 79. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productiv- ity assessment of neural code completion. In 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS@PLDI 2022), San Diego, CA, USA, 13 June 2022. ACM, 21-29. https://doi.org/10.1145/3520312.3534864 sourceExperimental — accuracy not yet measured
The Introduction describes neural tools that generate multi-line code suggestions and raises a possible cost of reviewing them, but it does not say developers may receive code they do not understand. Searches for understanding and comprehension of generated suggestions found no passage addressing that claim. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, professionals completed 0.57 more sub-tasks with GILT support compared to when they used search engines, whereas students did not experience significant gains.”
§ Using an LLM to Help With Code Understanding > 5.4 Additional Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not find any significant impact of participants' information processing style on their use of GILT.”
§ Using an LLM to Help With Code Understanding > Results.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, 83% of the failed tasks were also preceded by prompt-based interactions”
§ Using an LLM to Help With Code Understanding > 6.1 Usage of Features
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 110 claims and every source verdict.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 41
- The quote only partly states this claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“we analyzed the text of the prompts they wrote, by comparing the frequencies of bi-, tri-, and quadrigrams in the prompts.”
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“('make', 'bunny', 'sit', 'upright', 'chair') 3 0”
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Technology Acceptance Model: Past, Present, and Future
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01250 · Cited by 2,572 · cited 2×
“We based our questionnaire on the Technology Acceptance Model (TAM) [36], NASA Task Load Index (TLX) [21], and pre-and post-study questionnaires that were previously used in similar studies [63, 68].”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Johnston, R., Jones, K., & Manley, D. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0584-6 · Cited by 1,181
Open3D: A Modern Library for 3D Data Processing
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,124
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundSelf-citationVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Darley, W. K., & Smith, R. E. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673467 · Cited by 640 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Research works printed but not linked: 2
- 10. Victoria Clarke and Virginia Braun. 2013. Teaching thematic analysis: Overcom- ing challenges and developing strategies for effective learning. The psychologist 26, 2 (2013), 120-123.
- 67. Tabnine. 2024. Tabnine: AI assistant for software developers. https://www. tabnine.com/ Retrieved: 2024-01-11.
Cited by
Items 101–200 of 353
That's Not the Feedback I Need! - Student Engagement with GenAI Feedback in the Tutor Kai
Jacobs, S., Kempf, M., & Kiesler, N. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754512 · Cited by 15
“Similarly, Nam et al. [30] conclude that the benefit from their GenAI system “may vary depending on participants’ backgrounds or skills”.”
Brown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 15
Fuzzy-Assisted Contrastive Decoding Improving Code Generation of Large Language Models
Wang, S., Ding, L., Zhan, Y., Luo, Y., Liu, S., & Ding, W. (2025) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3575060 · Cited by 5
“Dueto thewideapplicability,researchonLLMsforcodegeneration hasgainedwidespreadattention[17],[18].”
Harnessing the Power of LLMs for Code Smell Detection in Terraform Infrastructure as Code
Vo, Q.-H., Dao, H., & Fukuda, K. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00075 · Cited by 8
“5-turbo), GPT-4 (gpt-4-0613) and GPT-4o (gpt-4o-2024-11-20), selected for their advanced capabilities in code comprehension, natural language processing, code understanding, and explanation [40]– [43].”
Connecting Large Language Models with Blockchain: Making Smart Contracts Smarter
Zeng, X., Xian, Y., Xuan, D., Yang, D., Li, C., Chen, J., Fan, P., & Liu, P. (2026) · ACM Transactions on the Web · doi:10.1145/3767296 · Cited by 16
“Since the advent of GPT-3, LLMs have achieved significant leaps in contextual understanding, complex reasoning, and multimodal generation, finding widespread applications in automated question-answering, intelligent assistants [46], code comprehension [21], and data analysis [39].”
LLM4Load: An LLM Prompt-Driven Framework for Multi-Scale Workload Prediction
Ding, Z., Ding, D., Zhang, H., Cao, J., & Xue, G. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00044 · Cited by 6
“Recent studies have successfully applied LLMs in computer vision [24], [25], recommendation systems [26], [27], and code understanding [28].”
PV-GPT: query the PV data using natural language
Zhang, D., Li, M., & Wang, Z. (2025) · Journal of Building Engineering · doi:10.1016/j.jobe.2025.114535 · Cited by 7
Schlesener, E. A., Ziolkowski, M., Wong, S., Westmoreland, B., & Babu, S. V. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3774779 · Cited by 16
“For instance, LLMs have demonstrated significant potential across various human-AI interactions (HAIIs), such as tools for code understanding [74], collaborative data annotation frameworks [113], and educational applications like LLM-based Physics tutors [66].”
Large Language Model-based power dispatch agent: Framework, application and challenges
Zhao, H. Q., Cheng, Y., Xiang, D., Zhou, X., Zhao, J., Cai, X., & Dong, Z. Y. (2026) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2026.111653 · Cited by 13
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Chen, V., Talwalkar, A., Brennan, R., & Neubig, G. (2025) · International Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790850 · Cited by 55
“We also see this in the user messages, where one participant asked “Did you delete most of the functions in [filename]?”
10 Years Later: Revisiting How Developers Search for Code
Stolee, K. T., Welp, T., Sadowski, C., & Elbaum, S. G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715774 · Cited by 7
“Some tasks that were previously solved with web search can now be solved with AI, such as finding examples to reuse [37] or understanding what code does [28], which may reduce the need for code search support.”
A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering
Martino, V. D., Castaño, J., Palomba, F., Franch, X., & Martínez‐Fernández, S. (2025) · IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) · doi:10.1109/wsese66602.2025.00008 · Cited by 21
“The use of Large Language Models (LLMs) has brought about a breakthrough in software engineering (SE), enabling more efficient and straightforward approaches to complex tasks [1]–[3].”
Benchmarking Large Language Models on Homework Assessment in Circuit Analysis
Chen, L., Qin, Z., Guo, Y., Rohde, J. A., & Zhang, Y. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00501-w · Cited by 19
The Digital Cybersecurity Expert: How Far Have We Come?
Wang, D., Zhou, G., Li, X., Bai, Y., Chen, L., Qin, T., Sun, J. Q., & Li, D. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00198 · Cited by 17
Using LLMs to enhance code quality: A systematic literature review
Alomari, N., Redah, M., Ashraf, A., & Alshayeb, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107960 · Cited by 12
Paradigm shift on Coding Productivity Using GenAI
Yu, L. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757081 · Cited by 18
Exploring Testing Methods for Large Language Models
Elvira, T., Procko, T., Vonderhaar, L., & Ochoa, O. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00177 · Cited by 6
“One use case for LLMs, especially for ChatGPT, is their ability to generate code in various programming languages, e.g., Python and Java [23].”
GLITCHES: GPU-FPGA LLM Inference Through a Collaborative Heterogeneous System
Yang, F., Yang, X., Wang, H., Wang, Z., Zhu, Z., Zeng, S., & Wang, Y. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938498 · Cited by 11
VerLog: Enhancing Release Note Generation for Android Apps using Large Language Models
Guo, J., Yang, H., & Cai, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728961 · Cited by 8
“In recent years, Large Language Models (LLMs) exhibit remarkable potential in both of these tasks [11, 32].”
Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?
Tufano, R., Martin-Lopez, A., Tayeb, A. J., Dabić, O., Haiduc, S., & Bavota, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00060 · Cited by 25
“Given the tremendous potential shown by LLMs in helping developers with various software engineering tasks [19]–[22], it seems natural to involve them also in the code review process, with the goal of reducing developer effort and software cost.”
ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs
Ge, H., Feng, J., Huang, Q., Fu, F., Nie, X., Zuo, L., Lin, H., Cui, B., & Liu, X. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3754352 · Cited by 27
Large Language Model for Financial Insights: Building a Digest to Simplify Research Activities
Sedov, D., & Lazarev, A. (2024) · Telecommunications Forum (TELFOR) · doi:10.1109/telfor63250.2024.10819154 · Cited by 4
JavaLLM: A Fine-Tuned LLM for Java Programming Education
Zhang, J., & Liu, K. (2024) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic64297.2024.00064 · Cited by 6
“In [1], Daye et al. presented the results of a user study that aimed to investigate the effectiveness of generation-based information support using LLMs to aid developers in code understanding, and showed that their approach significantly enhances developers’ ability to complete tasks compared to…”
Christakis, N., & Drikakis, D. (2025) · Applied Sciences · doi:10.3390/app15073784 · Cited by 9
“Unlike natural language, code necessitates strict adherence to syntax, logic, and context, making it difficult for LLMs to consistently produce correct and functional code [17].”
Less Is More: DocString Compression in Code Generation
Yang, G., Zhou, Y., Cheng, W., Zhang, X., Chen, X., Zhuo, T. Y., Liu, K., Zhou, X., Lo, D., & Chen, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735636 · Cited by 23
Automated Learning of Fine-Grained Citation Patterns in Open Source Large Language ModelsPreprint
Harcourt, E., Loxley, J., & Stanson, B. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/xnesr · Cited by 4
LLM-Powered Low-Code/No-Code Data Analytics in Education and Workforce Development
Zhong, C., Oruongo, J., & Kim, J. B. J. B. (2025) · Computer · doi:10.1109/mc.2024.3516614 · Cited by 7
A Unit Proofing Framework for Code-level Verification: A Research Agenda
Amusuo, P. C., Patil, P. V., Cochell, O., et al. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00013 · Cited by 3
“Hence, using the developed taxonomy, the repairer will detect faults by combining the fast but imprecise program analysis techniques the code reasoning abilities of large language models (LLMs) [39], [40].”
Build Code Needs Maintenance Too: A Study on Refactoring and Technical Debt in Build Systems
Ghammam, A., Rzig, D. E., Almukhtar, M., Khalsi, R., Hassan, F., & Kessentini, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00098 · Cited by 13
“It utilizes LLMs due to their adaptability and proclivity with source-code analysis [45]–[48].”
Measuring the Runtime Performance of C++ Code Written by Humans Using Github Copilot
Erhabor, D., Udayashankar, S., Nagappan, M., & Al-Kiswany, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00059 · Cited by 13
“A recent paper using LLMs for code understanding also has 32 participants and 2 tasks [60].”
Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants
Richards, J., & Wessel, M. S. (2025) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse67031.2025.00009 · Cited by 8
Ren, H., Wu, M., Croisdale, G. T., Guo, A., & Wang, X. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735788 · Cited by 9
“We also recorded the task completion time on the post-test.”
LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System
Li, F., Li, Y., Liu, Y., Zhou, C., Wang, Y., Deng, X., Xue, W., Liu, D., Xiao, L., Gu, H., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3750601.3750602 · Cited by 14
“LEADRE has been successfully deployed on both platforms, handling tens of billions of requests daily.”
Guriță, A.-E., & Vatavu, R.-D. (2025) · International Web for All Conference (W4A) · doi:10.1145/3744257.3744266 · Cited by 13
“Although extensive research has been conducted to examine LLMs’ capabilities in generating functionally correct, secure, and maintainable code [22,28,34], a gap still exists in our understanding of their abilities to produce code for accessible UIs, despite recent efforts to integrate them into…”
Integration of LLMs and VLMs in plant stress phenotyping: From trait recognition to decision support
Musazade, E., Mrisho, I. I., Gao, J., & Feng, X. (2025) · Plant Phenomics · doi:10.1016/j.plaphe.2025.100161 · Cited by 10
“By leveraging advanced NLP capabilities, LLMs such as GPT-4 and BERT offer robust tools for analyzing large volumes of unstructured agricultural data, including research reports, field records, and expert guidelines [95,96].”
Santos, Í., de Magalhães, C. V. C., & Santos, R. D. S. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00019 · Cited by 11
“Therefore, to better understand how these tools are being adopted and experienced in real-world settings, this study presents our preliminary findings on how practitioners utilize LLMs in coding activities, focusing on the experiences of software professionals from around the world and the types of…”
InfluentialSoftware Testing with Large Language Models: An Interview Study with Practitioners
Santana, D., de Magalhães, C. V. C., & Santos, R. D. S. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00018 · Cited by 8
“ACM ISBN 978-x-xxxx-xxxx-x/YY/MM... $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn development workflows, often via IDE-integrated tools or conversational assistants such as ChatGPT and GitHub Copilot [11, 16].”
InfluentialZhu, S., Han, G., Lin, C., et al. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3664900 · Cited by 3
“Large Language Models (LLMs), as a prominent research frontier in artificial intelligence, have demonstrated remarkable capabilities in learning, reasoning, and generative tasks across various domains, including text understanding and image synthesis [20], [21].”
Developing an Interactive OpenMP Programming Book with Large Language Models
Yi, X., Wang, A., Yan, Y., & Liao, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72567-8_12 · Cited by 3
“In a similar vein, Nam et al. (2024) explore the potential of using LLMs to enhance code understanding and development within an Integrated Development Environment (IDE) through their prototype tool, GILT [12].”
BackgroundTowards Personalized and Contextualized Code Explanations
Brachman, M., Goldberg, A., Anderson, A. W., Houde, S., Muller, M. J., & Weisz, J. D. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733681 · Cited by 9
Large Language Models-Aided Program Debloating
Lin, B., Wang, S., Qin, Y., Chen, L., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3594673 · Cited by 5
“For instance, LLMs excel at understanding complex program documentation by effectively summarizing the content and identifying the purposes and constraints of functions [51], [86].”
From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
Alfageeh, A., Zarkouei, S. A. K., Nam, D., Prol, D., Amoozadeh, M., Chattopadhyay, S., Prather, J. E., Denny, P. C., Leinonen, J., Hilton, M. L., et al. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770004 · Cited by 9
Self-citationSi, J. J., Li, D., Wang, Q., Liu, A. W., Lin, X., Zhu, Y., Zhou, X., Wang, A. Y., & Wang, K. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790366 · Cited by 7
The Wildcard XAI: from a Necessity, to a Resource, to a Dangerous Decoy
Carli, R., & Calvaresi, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70074-3_13 · Cited by 3
Zhao, Z., Bangash, A. A., Cogo, F. R., Adams, B., & Hassan, A. E. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3533972 · Cited by 10
“On the other hand, SE teams, as prominent FM users [27], [54], can leverage our findings to make more informed decisions when selecting the most appropriate leaderboards for their needs.”
A Novel LLM enabled Code Snippet Generation Framework
Sarkar, S., Kushwaha, S. P., Sharma, V., Mishra, N., & Alkhayyat, A. H. (2024) · International Conference on Intelligent & Innovative Practices in Engineering & Management (IIPEM) · doi:10.1109/iipem62726.2024.10925748 · Cited by 4
Giabbanelli, P. J., Beverley, J., Dávid, I., & Tolk, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5365069 · Cited by 4
Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling
Winston, C., Wang, R. Y., Mirhoseini, A., & Kozyrakis, C. (2026) · arXiv · doi:10.48550/arxiv.2605.21470 · Cited by 3
Exploring LLM Tools Through the Eyes of Industry Experts and Novice Programmers
Tona, C., Juárez‐Ramírez, R., Jiménez, S., & Durán, M. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00048 · Cited by 11
You Still have to Study On the Security of LLM Generated Code
Schaad, A., Götz, S., & Binder, D. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-92886-4_8 · Cited by 7
On the Usability of Reactive Programming APIs: A Mixed Evaluation
Zimmerle, C., & da Gama, K. S. (2025) · Software Practice and Experience · doi:10.1002/spe.3435 · Cited by 3
“For instance, large language models (LLM) integrated with IDEs like Visual Studio (VS) Code seems to be a prominent venue [80].”
Katzy, J., Huang, Y., Panchu, G.-R., Ziemlewski, M., Loizides, P., Vermeulen, S., van Deursen, A., & Izadi, M. (2025) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3727582.3728683 · Cited by 9
Alter, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94569-4_2 · Cited by 2
Towards an Adoption Framework to Foster Trust in AI-Assisted Software Engineering
Barón, M. M. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00038 · Cited by 2
“Recent research has placed significant emphasis on the role of large language model (LLM)-based tools in tasks such as detecting bugs [4], eliciting requirements [5], and comprehending source code [6].”
Optimizing Code Runtime Performance Through Context-Aware Retrieval-Augmented Generation
Acharya, M., Zhang, Y., Leach, K., & Huang, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00028 · Cited by 17
“Program Analysis: Program analysis methods address correctness, resource use, and speed [7], [33]–[35], but ML-based techniques often miss context-specific nuances [6], [8], [9], [36]–[38].”
LEGO: Synthesizing IoT Device Components Based on Static Analysis and Large Language Models
Liu, L., Wang, T., Chen, W., Wei, J., Wang, W., & Wu, G. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729482 · Cited by 2
“Nam et al. [53] investigate the effectiveness of LLMs for generative information support to help developers understand code.”
SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works
Daneshyan, F., He, R.-Z., Wu, J.-Y., & Zhou, M.-H. (2025) · Proc. ACM Softw. Eng. · doi:10.1145/3729345 · Cited by 8
“LLMs have significant benefits in code comprehension tasks that allow them to capture semantic meaning and connections between code and natural language text [Feng et al. 2020; Nam et al. 2024; Wang et al. 2021], allowing for intricate summaries of changes.”
Walls Have Ears: Demystifying Notification Listener Usage in Android Apps
Deng, J., Liu, T., Zhao, Y., Wang, C., Zhang, L., & Wang, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728898 · Cited by 3
“Recent advancements in LLM, particularly in their capabilities for semantic understanding and logical reasoning [49, 55], have led to their widespread application in various software engineering and security tasks [38, 61].”
An LLM Assistant for Software Project Onboarding
Larsen, K. R., Edvall, M., Ho-Quang, T., Dobslaw, F., & Jolak, R. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728719 · Cited by 3
Intent-Based Kubernetes Configuration via LLMs: Current Trends and Open Challenges
Sacco, A., Zilli, C., & Marchetto, G. (2025) · Conference on Local Computer Networks · doi:10.1109/lcn65610.2025.11146359 · Cited by 6
“…LLMs see a wide range of applications, such as natural language generation, understanding, text summarization, classification, and code generation [9], their adoption also looks very promising for creating autonomous self-healing clouds, where AI-driven approaches can detect, localize, and…”
Evaluating LLMs and Prompting Strategies for Automated Hardware Diagnosis from Textual User-Reports
Caminha, C., Silva, M. D. L. M., Chaves, I. C., Brito, F. T., Farias, V. A. E., & Machado, J. C. (2025) · Anais do Seminário Integrado de Software e Hardware (SEMISH) · doi:10.5753/semish.2025.8473 · Cited by 3
User Experience in Sustainable Tourism: Enhancing Transparency in Tour Guide App Recommendations
Chasanidou, D., Boletsis, C., & Krogstie, J. (2025) · International Conference of the ACM Greek SIGCHI Chapter (CHIGreece) · doi:10.1145/3749012.3749064 · Cited by 3
“Šturm et al. [28] used them to explain cognitive digital twins, while Nam et al. [21] applied them in IDEs to clarify code.”
Park, E. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10730-1 · Cited by 6
“…advanced tools for code generation, debugging, and learning new programming languages, providing contextual code suggestions, clarifying complex programming concepts, and automating routine coding tasks (Acher et al. 2023; Foundjem et al. 2022 ; Godliauskas & Šmite 2025 ; Nam et al. 2024).”
LLM-Driven Auto Configuration for Transient IoT Device Collaboration
Shastri, H., Hanafy, W. A., Li, W., Irwin, D., Srivastava, M. B., & Shenoy, P. J. (2025) · ACM/IEEE Symposium on Edge Computing (SEC) · doi:10.1145/3769102.3770619 · Cited by 8
Wang, S., Bai, Z., & Gai, Y. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00416-9 · Cited by 2
“The advent of large language models (LLMs) represents a watershed moment in natural language processing, demonstrating the transformative impact of neural-network techniques on nearly every NLP task [11].”
Reasoning About Human Values in GitHub Issues: What Can a Large Language Model Reveal?
Mougouei, D., Haque, A., Nurwidyantoro, A., Mougouei, E., Fahmideh, M., Shahin, M., Dharmawan, A., Widodo, T. W., & Dam, H. K. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2594137 · Cited by 3
“…(LLMs), such as ChatGPT, across various software engineering tasks, including automated comment generation (Geng et al., 2024), code comprehension and refinement (Guo et al., 2024; Nam et al., 2024), software testing (Tang et al., 2024; Wang et al., 2024), and program repair (Jin et al., 2023).”
Zhao, H., Zhang, L., Lian, K., Sun, F., Chen, B., Liu, Y., Wu, Z., Zhang, Y., & Yang, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00082 · Cited by 18
“For example, in code-related tasks, researchers have explored LLMs’ effectiveness in vulnerability detection [7], [8], code understanding [40], [35], [42], code generation [43], [44], code repair [45], [46], and program state inference [47].”
Hallucination detection in LLM code generation: A sampling-based consensus verification approach
Huang, T., Ren, Z., Huang, Y., Chen, X., Liu, Y., & Zheng, Z. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00605-0 · Cited by 7
“Compared to static similarity metrics, LLMs exhibit better semantic understand - ing capabilities, enabling them to evaluate the functional consistency between two code samples and identify whether they are consistent or contradictory (Nam et al. 2024).”
Developers’ Dilemma: Opportunities and Pitfalls of Generative AI for Software Development
Zacharias, J., Popova, A., von Zahn, M., Chen, J., & Hinz, O. F. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00998-y · Cited by 12
Brodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., et al. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2524747123 · Cited by 16
Raihauti, T., Midingoyi, C. A., Adam, M., Berg‐Mohnicke, M., Brown, H., Cichota, R., Enders, A., Ferreira, T. B., Fumagalli, D., Holzworth, D. P., et al. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111996 · Cited by 4
HSG-RAG: Hierarchical Knowledge Base Construction for Embedded System Development
Lu, Z., Xu, H., Chen, A. Q., Tang, S., Zhang, J., Feng, Y., Pan, W., & Huang, J. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3731680 · Cited by 7
“We deine short-distance knowledge as knowledge that exists within a single text unit or spans across adjacent text units.”
GenieWizard: Multimodal App Feature Discovery with Large Language Models
Yang, J., Shi, Y., Gu, C., Zheng, Z., Jain, A., Li, T., Lam, M. S., & Landay, J. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714327 · Cited by 8
A Big Step Forward? A User-Centric Examination of iOS App Privacy Report and Enhancements
Wang, L., Wang, D., Pan, S., Jiang, Z., Wang, H., & Wang, Y. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00223 · Cited by 2
“<“developer”, “camera”, “scan code to add friend”> 2… (One-shot) to perform program analysis, such as understanding the code constructs and data flows [39,40].”
Coni: Detecting Database Connector Bugs via State-Aware Test Case Generation
Deng, W., Liang, J., Wu, Z., et al. (2025) · IEEE/ACM International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00003 · Cited by 2
“To model more efficiently, the latest techniques such as Large Language Models (LLMs) can be used as auxiliary tools to help analyze interface documentation, code comments, and method names to predict the pre and post-states, thereby automating the process [30].”
Zhang, H., Shen, Y., Wang, L., Shi, C., Du, S., Tao, Y., & Shen, Y. (2025) · doi:10.1145/3730436.3730529 · Cited by 2
“[ 12 ] Therefore, more advanced options are required and large language models (LLMs) has been the perfect candidate so far.”
Attributing ChatGPT-Transformed Synthetic Code
Choi, S., Alkinoon, A., Alghuried, A., Alghamdi, A., & Mohaisen, A. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00018 · Cited by 4
“LLMs have significantly advanced their understanding and processing of programming languages [17], [31], [38].”
When Help Hurts: Verification Load and Fatigue with AI Coding Assistants
Fan, G. F., Liu, D., Pan, L., & Zhang, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791176 · Cited by 11
Jang, Y., Sakashita, M., Niinuma, K., & Gupta, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791260 · Cited by 3
“LLM-based assistance has been shown to improve sensemaking tasks and reduce completion time when compared with traditional search-based information retrieval [37].”
Kobiella, C., Breidenstein, D., & Schmidt, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790757 · Cited by 8
“These tools can explain code, debug, and generate solutions from natural-language descriptions directly in the user’s IDE [6, 50].”
Cloud-Based System for Source Code Analysis of Microservices with LLM Agents
Chaplia, O., & Klym, H. (2024) · International Conference on Computer Science and Information Technologies · doi:10.1109/csit65290.2024.10982613 · Cited by 2
“Recent advancements in artificial intelligence have led to the development of AI-driven code analysis tools [16].”
Experiences on Using Large Language Models to Re-Engineer a Legacy System at Volvo Group
Singh, V., Korlu, C., & Assunção, W. K. G. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00018 · Cited by 9
“Large Language Models (LLMs) have been explored in various aspects of software engineering, such as code understanding, code generation, review, requirements engineering, and optimization [18]–[20].”
Can an LLM Find Its Way Around a Spreadsheet?
Lee, C.-T., Neeser, A., Xu, S., Katyan, J., Cross, P., Pathakota, S., Norman, M., Simeone, J., Chandrasekaran, J., & Ramakrishnan, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00101 · Cited by 4
“By engaging in English conversations, LLMs can function as coding assistants, enabling users with limited proficiency to produce accurate and executable code to accomplish their tasks [5, 6].”
Reversed Model Verification by Inferring Conceptual Models from Simulation Code
Neykova, R., & Groen, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97635-3_43 · Cited by 1
Leveraging LLMs for Interaction Point Detection and Payload Alignment in CTF challenges
Chen, J., Li, R., Qin, F., & Tang, C. (2025) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss65394.2025.11085643 · Cited by 1
“Trained on vast corpora of programming languages, technical documentation, and security-related materials, LLMs have demonstrated an impressive ability to understand code semantics[7], generate exploit scripts[8], and reason about complex vulnerabilities[3].”
Generative AI Efficiency and Effectiveness in Software Project Documentation Review Process
Demir, K. A., Muppalla, T. N. V., & Liu, B. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11165833 · Cited by 1
“…Large Language Model (LLM)-based Generative Artificial Intelligence (GenAI) has gained significant attention in software engineering [6], [7], [9], [8], [10], [11], demonstrating the huge potential to revolutionize software development from initial planning [12], [13] to final CI/CD pipeline [14]…”
Salomon, M., Chin, K. D., Holmes, R., Fritz, T., & Murphy, G. C. (2025) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3758317.3759680 · Cited by 6
“To date, studies about how GenAI impacts software engineering and software engineering education largely align with the views that GenAI can assist individual developers (e.g., [3, 4, 24, 26, 40]).”
Towards Leveraging Large Language Model Summaries for Topic Modeling in Source Code
Carissimi, M., Saletta, M., & Ferretti, C. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757026 · Cited by 5
“Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in several domains, and their potential is also evident in the program comprehension field [15].”
Large Language Models for API Classification: An Explorative Study
Morais, G., Lemelin, E., Adda, M., & Bork, D. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756997 · Cited by 2
TraceMate: Collaborating with AI in Test-Driven Programming
Wu, J., Selig, T., & Purwanto, E. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00035 · Cited by 5
“], [9], and an IDE-based ChatGPT assistant improved code comprehension tasks in one evaluation [17].”
Revisiting data analysis with Pre-trained foundation models
Liang, C., Yang, D., Liang, Z., Liang, Z., Zhang, T., Xiao, B., Yang, Y., Wang, W., & Wang, H. (2026) · The VLDB Journal · doi:10.1007/s00778-025-00953-5 · Cited by 5
“[148] provide an IDE plugin that interprets complex repositories with scarce comments and documentation.”
Explaining Code Risk in OSS: Towards LLM-Generated Fault Prediction Interpretations
Adejumo, E. K., & Johnson, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00046 · Cited by 4
“Recent studies indicate that LLM-generated explanations and fault location suggestions can improve developer understanding while enabling faster task completion [15], [16].”
From Disruptions to Discussions: How GenAI Impacts Human Interactions in Software Development
Salomon, M., Koshchenko, E., Sergeyuk, A., Holmes, R., Murphy, G. C., & Fritz, T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655626 · Cited by 1
“In particular, our data analysis indicated that developers in our study experienced: • GenAI providing explanations and code examples that enabled developers to learn and work autonomously ( T HEME 9 and T HEME 10 ) (e.g. [8], [45], [46]).”
Faultseeker: LLM-Empowered Framework for Blockchain Transaction Fault Localization
Sun, K., Xu, Z., Li, K., Zhang, L., Sun, Y., Tan, L., & Liu, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00017 · Cited by 1
“Recent breakthroughs in large language models (LLMs) have demonstrated remarkable capabilities in code comprehension [22], [28] and vulnerability detection [12], [27].”
Large Language Models in Software Documentation and Modeling: A Literature Review and Findings
Radoský, L., & Polášek, I. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420586 · Cited by 3
“H. Software security Intelligence and Informatics • January 29–31, 2026 • Stará Lesná, Slovakia”
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 3
Exploring the architectures of large language models and impact across multiple domains: a review
Waqas, M., Rajab, R., Laghari, A. A., Kryvinska, N., Hejaili, A. A., Khan, N., Khuhro, M. A., & Zhang, F. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11609-3 · Cited by 2
Acoda: Adversarial Code Obfuscation for Defending against LLM-based Analysis
Rao, H., Dong, Z., Zhao, Y., Li, H., & Wang, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3793624 · Cited by 3
CGTest: A hybrid coverage-guided framework for cost-effective unit test generation
Yang, Z., Wang, H., Hu, Y., Gu, B., Xiong, Q., Huo, H., Xiang, C., & Liu, Z. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113055 · Cited by 1
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Al-Kaswan, A., Spieß, C. K., Dévanbu, P., van Deursen, A., & Izadi, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793341 · Cited by 5
“Their popularity stems from their ability to understand and produce code in multiple programming languages, often outperforming traditional methods in terms of speed and accessibility [11, 37].”
Metadata
- Authors
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745423
- Semantic Scholar
- f5ea85f67fc5ea9cce2c5345ed92740f7377cb2e
- Cited by
- 487
- References
- 61
- Citations by year
- 2026: 184 · 2025: 233 · 2024: 66
- FWCI
- 114.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, program comprehension, conversational interfaces, GPT-3.5-turbo, API documentation, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.