Using an LLM to Help With Code Understanding
Open accessThe DOI matches its Crossref record; 87 of the 89 references it cites were found in a registry or an index; 9 of the 20 sources its claims cite were checked for support.
Abstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Overview
CoChat Summary
TL;DR
GILT is an in-IDE VS Code plugin that uses GPT-3.5-turbo to provide prompt-less and contextualized LLM support for code understanding. In a 32-participant user study, GILT improved task progress over web search but did not significantly improve completion time or measured code understanding.
Key findings
- Participants using GILT completed statistically significantly more sub-tasks than those using search engines: 0.47 more sub-tasks, 𝑝 < 0.01, controlling for experience and AI tool familiarity.
- GILT did not show significant differences in task completion time or code understanding quiz scores compared with search.
- Professionals completed 0.57 more sub-tasks with GILT support, a significant effect, whereas students did not experience significant gains.
- Usage averaged 15.34 GILT interactions per participant (range 5–23); Overview was most frequent (4.76 activations), followed by Prompt-context (4.12) and Prompt-followup (2.88). TAM scores for perceived usefulness/ease were higher for GILT than search (𝑝 < 0.001), and TLX showed significant differences in temporal demand and performance (𝑝 < 0.05).
Methods
- The authors built GILT, a VS Code plugin querying OpenAI’s GPT-3.5-turbo with four prompt-less buttons (Overview, API, Concept, Usage) and open-ended prompts automatically contextualized with selected code.
- They conducted a within-subjects user study with 32 analyzed participants (33 recruited, 1 excluded), using Bokeh data-visualization and Open3D 3D-rendering tasks, each with 4 sub-tasks and 20 minutes per task, comparing GILT against search engines on different counterbalanced tasks.
- They estimated quasi-Poisson models for task progress and understanding and linear regression for completion time, controlling for domain experience, programming experience, and AI tool familiarity; they also analyzed logs, prompts, and TAM/TLX survey responses.
Limitations
- The paper states threats from task/library selection, visible-output tasks, less common libraries, possible in-IDE versus search confounds, lab setting, time pressure, novelty effects, a small sample of 32 skewed toward academia, rapid LLM development, and standard statistical conclusion threats; they removed top 1% outliers and checked VIF < 2.5.
- The paper also notes that different libraries or larger codebases could produce different results, and that further real-world studies with larger, more diverse samples are needed.
- Evident from the text: the study used only two Python libraries and short timed tasks, relied on GPT-3.5-turbo at the time, and did not use think-aloud during tasks, which may limit generalization to everyday professional programming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 4 Not valid · 3 Not checked · 99 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30,31,34,44,48,57].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [30] 30. Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collo- cated Software Development Teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 344-353. https://doi.org/10.1109/ICSE.2007.45 source
- Not checked — no full text of the sourcecites [31] 31. Amy J. Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (11 2006), 971 -987. https://doi.org/10.1109/tse.2006.116 source
- Not checked — no full text of the sourcecites [34] 34. Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In Proceedings of the 28th international conference on Software engineering. 492-501. source
- Partly supportedcites [44] 44. Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 31:1-31:37. https://doi.org/10.1145/2622669 sourceExperimental — accuracy not yet measured
Section 6.4.5 (page unavailable) says that students search beyond course materials for help, including on StackOverflow. It does not establish that developers spend a significant amount of time searching or foraging, or that they spend time organizing and digesting what they find. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [48] 48. Andre N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178-1193. https://doi.org/10.1109/tse.2017.2656886 source
- Partly supportedcites [57] 57. David Piorkowski, Austin Z. Henley, Tahmid Nabi, Scott D. Fleming, Christopher Scaffidi, and Margaret M. Burnett. 2016. Foraging and navigations, fundamentally: developers' predictions of value and cost. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE 2016), Seattle, WA, USA, November 13-18, 2016. ACM, 97-108. https://doi.org/10.1145/ 2950290.2950302 sourceExperimental — accuracy not yet measured
Our copy of Rahman et al. (2018) supports the searching/foraging half of the claim: the Abstract (p. 1) states 'Search is an integral part of a software development process. Developers often use search engines to look for information during development,' and the effort results (RQ2/RQ3, pp. 6-8; Conclusion, p. 10) report that developers spend substantial and significantly greater time, web visits and query edits on code search (median code task 2 min 53 s vs 1 min 35 s for non-code). However, the paper never addresses the 'organizing and digesting the information they find' component — its logs track browsing activity only, with no measure of post-retrieval comprehension, organization or digestion. The claim's magnitude is also framed absolutely ('a significant amount of time'), whereas this source's time figures are comparative between code and non-code tasks, not a share of developers' working time. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
§ Using an LLM to Help With Code Understanding > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [24] 24. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, (ICSE Companion 2022), Pittsburgh, PA, USA, May 22-24, 2022. ACM/IEEE, 319-321. https://doi.org/10.1145/3510454.3522684 source
- Could not verifycites [79] 79. Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productiv- ity assessment of neural code completion. In 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS@PLDI 2022), San Diego, CA, USA, 13 June 2022. ACM, 21-29. https://doi.org/10.1145/3520312.3534864 sourceExperimental — accuracy not yet measured
The Introduction describes neural tools that generate multi-line code suggestions and raises a possible cost of reviewing them, but it does not say developers may receive code they do not understand. Searches for understanding and comprehension of generated suggestions found no passage addressing that claim. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, professionals completed 0.57 more sub-tasks with GILT support compared to when they used search engines, whereas students did not experience significant gains.”
§ Using an LLM to Help With Code Understanding > 5.4 Additional Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not find any significant impact of participants' information processing style on their use of GILT.”
§ Using an LLM to Help With Code Understanding > Results.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, 83% of the failed tasks were also preceded by prompt-based interactions”
§ Using an LLM to Help With Code Understanding > 6.1 Usage of Features
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 110 claims and every source verdict.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 41
- The quote only partly states this claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“we analyzed the text of the prompts they wrote, by comparing the frequencies of bi-, tri-, and quadrigrams in the prompts.”
§ Using an LLM to Help With Code Understanding > 6.2 Professionals vs. Students
“('make', 'bunny', 'sit', 'upright', 'chair') 3 0”
Citations
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Technology Acceptance Model: Past, Present, and Future
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003) · Communications of the Association for Information Systems · doi:10.17705/1cais.01250 · Cited by 2,572 · cited 2×
“We based our questionnaire on the Technology Acceptance Model (TAM) [36], NASA Task Load Index (TLX) [21], and pre-and post-study questionnaires that were previously used in similar studies [63, 68].”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Johnston, R., Jones, K., & Manley, D. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0584-6 · Cited by 1,181
Open3D: A Modern Library for 3D Data Processing
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,124
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Consequently, developers spend a significant amount of time searching and foraging for the information they need and organizing and digesting the information they find [30, 31, 34, 44, 48, 57].”
BackgroundSelf-citationVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Darley, W. K., & Smith, R. E. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673467 · Cited by 640 · cited 1×
“Out of two information processing styles [8, 11, 19], people who exhibit a “selective information processor” tendency focus on the first promising option, pursuing it deeply before seeking additional information.”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Research works printed but not linked: 2
- 10. Victoria Clarke and Virginia Braun. 2013. Teaching thematic analysis: Overcom- ing challenges and developing strategies for effective learning. The psychologist 26, 2 (2013), 120-123.
- 67. Tabnine. 2024. Tabnine: AI assistant for software developers. https://www. tabnine.com/ Retrieved: 2024-01-11.
Cited by
Items 201–300 of 353
DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain
Huang, E., Sun, P., Wang, S., Lin, Z., Chen, A., Hu, K., Ouyang, J., Li, F., Zhang, Z., Wang, H., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817512
“We maintain (i) a curated list of major Web3 social platforms with verified official accounts for selected projects, and (ii) a link list of project websites and whitepapers.”
He, J., Wang, S., Zhao, Y., Hou, X., Liu, Z., Zou, Q., & Wang, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773199 · Cited by 13
“Large Language Models (LLMs) have revolutionized numerous tasks, demonstrating remarkable capabilities in areas such as code generation [11], natural language understanding [25], and logical reasoning [12].”
Rethinking the Evaluation of Secure Code Generation
Dai, S.-C., Xu, J. L., & Tao, G. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773217 · Cited by 45
“Other applications of LLMs include program repair [59, 60, 67], code analysis [19, 40, 66], and unit test generation [11, 16, 63].”
Beyond TAP: Piggybacking on IFTTT to Connect Triggers and Actions with JavaScript
Manesh, D., Wang, M., Hu, R., & Lee, S. W. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00036 · Cited by 1
“To mitigate these challenges, AI could provide explanations of the code [63], [64] or even translate the code into an intermediate representation that is more intuitive for the end-user [65].”
BackgroundGender-Related Preferences for Learning by Tinkering: Updated Research is Needed
Newendorp, A. K., & Gilbert, S. B. (2024) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813241272118 · Cited by 1
“In one study, participants who prefer to learn by process were more likely to use an LLM-based assistant to debug their code than participants who prefer to learn by tinkering, who instead were more likely to tinker with their code than use the assistant (Nam et al., 2024).”
Can ChatGPT Pass Modern Control Theory Exam?
Ogo, I., & Koga, M. (2024) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas63016.2024.10773183 · Cited by 2
“In particular, GPT models, have contributed to solving problems in a wide variety of fields, including mathematics[3], [4], coding[5], [6], etc.”
Embedding process knowledge in cyber-social systems on the example of cognitive tutor to teach scope
Александрович, С. О., Sidor, A., & Karpenko, P. (2023) · Journal of Integrated Design and Process Science · doi:10.1177/10920617241289701 · Cited by 1
“That can help develop abilities of reading, debugging and writing program code to those who already have them (Kazemitabaar et al., 2024; Nam et al., 2024; Shen et al., 2024).”
Integrating LLMs into Mathematics Laboratories Alongside CAS
Matzakos, N., & Moundridou, M. (2025) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v15i3.53151 · Cited by 2
Generative AI Driven Clinical Drug Development
Tsuchiwata, S., TANABE, Y., HOSOYA, Y., Yonekura, T., Someya, M., ITAMI, K., KITAMURA, A., ISHII, M., Yamei, H. A. N., OHASHI, H., et al. (2025) · Rinsho yakuri/Japanese Journal of Clinical Pharmacology and Therapeutics · doi:10.3999/jscpt.56.2_109 · Cited by 1
HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoning
Jiang, Z., Wu, P., Liang, Z., Chen, P. Q., Xu, Y., Jia, Y., Tu, J., Li, C., Ng, P. H. F., & Li, Q. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737378 · Cited by 27
“Recently, Large Language Models (LLMs) have shown remarkable performance across a variety of tasks, such as conversational Artificial Intelligence (AI) [4, 7, 40], text summarization [27, 68], language translation [23, 29, 36], programming assistance [35, 45].”
Kohita, R., & Kasuga, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.247 · Cited by 3
Toward Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT
Tao, Z., Chen, Y., Xi, D., li, Z., & Xu, W. X. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3779427 · Cited by 22
Jufriansah, A., Pramudya, Y., Khusnani, A., & Malahina, E. A. U. (2025) · Journal of Novel Engineering Science and Technology · doi:10.56741/jnest.v4i02.952 · Cited by 2
“The purpose of the output specification is to guide the LLM in populating the output structure elements with the steps necessary to complete the task [46].”
Giabbanelli, P. J., Beverley, J. R., Dávid, I., & Tolk, A. (2025) · Winter Simulation Conference · doi:10.1109/wsc68292.2025.11338852 · Cited by 2
“Similar avenues have been explored, e.g., by Nam et al. (2024) to generate comprehensive documentation for complex simulations.”
RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust
Ou, G., Liu, M., Chen, Y., Wang, Y., Peng, X., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00057 · Cited by 43
“LLMs like GPT-4 excel in code understanding [43].”
Prompt-Based Building Code Information Transformation Method for Automated Compliance CheckingPreprint
Yang, F., & Zhang, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4819834 · Cited by 1
Assessing Open Source Software Developer's Opportunity Costs in a Time of Generative AIPreprint
Li, X., Costello, F. J., Kim, K., & Zhang, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5100609 · Cited by 6
The fools are certain; the wise are doubtful: Exploring LLM confidence in code completion
Kotti, Z., Dritsa, K., Spinellis, D. D., & Λουρίδας, Π. (2026) · Automated Software Engineering · doi:10.21203/rs.3.rs-7302683/v1 · Cited by 3
“Code LLMs find applications in diverse software engineering tasks including code generation (Zan et al, 2023), code understanding (Nam et al, 2024), and code translation (Pan et al, 2024).”
A structured evaluation and benchmarking of Boolean logical modelling toolsPreprint
Saalfeld, B., Lange, E., Krüger, J., Lutter, P., & Heyer, R. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.28.684998 · Cited by 1
“These and further developments can be used to add documentation to the software without much labour; in the best case, developers only need to proofread the generated documentation [42–44].”
CodeVoyager: Integrating Interactive Visual Aids with LLMs for Code Comprehension
Kim, Y., Jang, K., Song, H., Shin, M., & Suh, B. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789057 · Cited by 3
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
Beason, J.-T., Cheng, R., Schoop, E., & Nichols, J. W. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789133 · Cited by 6
Wrap Your Mind Around Education: Applying Hugging Face to a Chatbot with AI
González, D. I. R., Rivero, J. A. S., & Hernandez, J. H. (2024) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-66635-3_40
Revisiting the Conflict-Resolving Problem from a Semantic Perspective
Dong, J., Sun, J., Lin, Y., Zhang, Y., Ma, M., Dong, J. S., & Hao, D. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694993 · Cited by 3
Preliminary Study of LLM-Based Wordlist Generation for Validating Broken Web Access Control
Ng, K. K. S., Yan, F., & Hung, K. (2024) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon61640.2024.10902771 · Cited by 2
“Large Language Models (LLMs) are designed to understand text-based content [6][7].”
CiGA: A Cross-Layer Fine-Grained Attention Correction Method for Large Language Model
Li, D., & Zhao, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889880 · Cited by 1
“Several methods have been proposed to address the limitations of LLMs in retaining fine-grained information [4], [5].”
CATER: A Cluster-Based Alternative-Term Recommendation Framework for Large-Scale Web Search at NAVER
Son, J., Kim, J., Kim, T., Lee, Y.-C., & Kim, S. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709426 · Cited by 1
“We leverage the capabilities of LLMs, which have demonstrated remarkable performance in various tasks ( e .g., sentiment analysis [11, 52], named entity recognition [15, 17], and summarization [19, 45]) due to their extensive model sizes and pre-training on vast amounts of data [13, 14, 35, 50, 54].”
Tool-supported Development of ML Prototypes
Aydın, S., & Lichter, H. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00063
“We use LLMs to automatically assign meaningful names to activities and to add explanations to activities, as LLMs have been shown to be effective in improving code understandability [6].”
Research on Hybrid Collaborative Development Model Based on Multi-Dimensional Behavioral Information
Gao, S., Liao, W., Shu, T., Zhao, Z.-N., & Wang, Y. (2025) · Applied Sciences · doi:10.3390/app15094907
Learning a First Programming Language with LLM Tutor
Vukojičić, M., & Korica, I. (2025) · SINTEZA · doi:10.15308/sinteza-2025-331-336 · Cited by 1
“Some papers [1] agree that standard methods of learning how to code can be demanding because of the way that instructions are presented, often with robust textual explanations.”
COmmitSHield: Tracking Vulnerability Introduction and Fix in Version Control Systems
Wu, Z., Zhao, Y., Chen, W., Wan, Z., Liu, Y., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00082 · Cited by 5
“Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing and code understanding [5], [13], making their application to vulnerability introduction and fix detection promising.”
Centaurs and Cyborgs: Models of Collaboration
(2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-2518-7.ch003
Large Language Model Based Agents for Generating Industrial Dashboards
Kimmel, R., Pfeiffer, J., Zhang, J., & Wortmann, A. (2025) · Lecture notes in production engineering · doi:10.1007/978-3-031-86893-1_57
Key-Augmented Neural Triggers for Knowledge SharingPreprint
Wolf, A., Palma, M. A., Rani, P., & Gall, H. C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5376750
Equity Research Analysis Using Generative AI
Lachure, O., Kshirsagar, S., Karale, T., & Londhe, T. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-4151-2_39
Do LLMs Suggest Consistent Identifiers? An Empirical Study on GitHub Pull Requests
Silva, J. R., Maia, M. D. A., & Dantas, C. E. C. (2025) · doi:10.5753/ise.2025.14870
How Close Is ChatGPT to Developer Judgment? A Study on Stack Overflow Java Questions
Reis, F. A. G., Maia, M. D. A., & Dantas, C. E. C. (2025) · doi:10.5753/cbsoft_estendido.2025.14164
“ChatGPT could also help with code understanding [Nam et al. 2024] and produce high-quality educational code snippets that are often easier to read than those written by humans on SO [Dantas et al. 2023].”
LLM-based program analysis for source codes, abstract syntax trees and webassembly instructions
Deng, L., Zhong, Q., Qiu, Y., Chen, J., Lei, H., Yang, S., Zhou, L., & Cheng, H. (2025) · Cluster Computing · doi:10.1007/s10586-025-05557-w · Cited by 1
“Therefore, we consider generative pre-trained transformer (GPT) [ 26–28] models and causal large-scale language models (CLLMs) [ 29, 30] to complete our experiments.”
冯, 丙. (2025) · Advances in Education · doi:10.12677/ae.2025.15101822
A Scalable Vulnerability Detection System with Multi-View Graph Representations
Dou, S., Zheng, H., Shan, J., Wu, Y., Zou, D., Huang, X., & Liu, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770075 · Cited by 4
“As shown in Figure 9, it takes only about 31.03 milliseconds on average to complete the text embedding of three graph representations.”
InfluentialExploring the Potential of Large Language Models in Fine-Grained Review Comment Classification
Nguyen, L., Liu, C., Lin, H., & Thongtanunam, P. (2025) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam67354.2025.00012 · Cited by 3
“Despite the promising capability, LLMs are sensitive to query framing and context length, leading to varied responses [20, 21].”
Condor: A Code Discriminator Integrating General Semantics With Code Details
Liang, Q., Zhang, Z., Liu, C., Sun, Z., Zhang, W., Chen, Y., Zhao, Z., Luo, Q., Wang, W., Jiang, Y., et al. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3620145 · Cited by 6
“W ITH the widespread application of large language models (LLMs), their potential for tasks in software engineering (e.g., code generation and code repair) receives signifi-cant attention [1], [2], [3], [4], [5], [6].”
Wu, S., Xiao, J., & Sang, J. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3620146 · Cited by 3
“Previous studies have demonstrated significant di ff erences in the quality of prompts generated by users with varying skill levels when utilizing large models for code tasks [37], [38].”
Chen, Z., Nava-Camal, V., Roy, T., Li, Z., Tang, Y., Zhang, X., & Yang, H. (2025) · IEEE Software · doi:10.1109/ms.2025.3618099
Determining Code Proficiency Levels from Python Textbooks
Rojpaisarnkit, R., Robles, G., Gonzalez-Barahona, J. M., Matsumoto, K., & Kula, R. G. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769864 · Cited by 1
“Nam et al. [29] developed a tool to evaluate large language models (LLMs) in aiding developers’ understanding of code, comparing their performance with traditional web search tools like StackOverflow and Medium.”
Evaluating and Improving GPT-Based Expansion of Abbreviations
Jiang, Y., Li, C., Zhao, Z., Fan, F., Zhang, L., & Liu, H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3623625
“…in the development of large language models (LLMs), the enhanced capabilities of machine learning techniques facilitate breakthroughs in fields such as text summarization [13], translation [14], sentiment analysis [15], code search [16], code generation [17], and code summarization [18].”
Towards Proficiency Assessment Through Code
Rojpaisarnkit, R., & Matsumoto, K. (2025) · IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme64153.2025.00093
“Research indicates that for a developer to effectively leverage, integrate, and validate AI-generated code, they must first comprehend its context and logic; working with code they understand leads to better and more reliable outcomes [10].”
Supporting Agile Roles and Practices with GAI
Ciancarini, P. (2025) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito66076.2025.11241628
Reconstruction-Based Prompt Generation Algorithm for Prompt Injection Attacks
Yu, Z., & Fan, C. (2025) · International Conference on Advanced Algorithms and Neural Networks (AANN) · doi:10.1109/aann66429.2025.11257661 · Cited by 1
“With the widespread adoption of large language models (LLMs) [1] in natural language processing, their exceptional language comprehension and generation capabilities have made them fundamental to various tasks such as dialogue systems [2] , content creation [3] , and code assistance [4] .”
Fan, Z., Liu, Y., Gagnon, G., Liu, L., Hou, Y., Benmeziane, H., Maghraoui, K. E., & Liu, L. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790226 · Cited by 8
Pikal, K., Ordenes, F. J. V., Herhausen, D., & Tamagnini, P. (2025) · Journal of Interactive Marketing · doi:10.1177/10949968251404927 · Cited by 1
“Step 5: Building the Front End However, LLMs can offer an easy way to adapt code to different frameworks and can be used to understand the code (Nam et al. 2024).”
BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning
Santos, J. G. A., Bonás, G. K., & Almeida, T. S. (2025) · doi:10.5753/eniac.2025.14256 · Cited by 4
LLM Vs Rule-Based - The COBRAIN Tool and An Empirical Study on Extracting Business Rules from COBOL
S, C. B., & Chimalakonda, S. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756982 · Cited by 2
“Largelanguagemodels(LLMs)havebeenextensively usedwithin software engineering for tasks such as code summarization [22, 24], code comprehension [25, 31], and program analysis [31, 35].”
HBS-KGLLM: A General Framework for Generating Knowledge Graphs for Jailbreaking
Zhao, X., Li, B., Zhuo, J., Wu, W., Huang, R., Liu, Y., Wang, H., Dai, H., & Nguyen, Q. V. H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3906-2_31
Developer Prompts in Practice: An Empirical Study of Bias, Security, and Optimization
Rzig, D. E., Paul, D. J., Pister, K. S., Henkel, J. M., & Hassan, F. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00067 · Cited by 2
“Nam et al. [75] employed LLMs for code understanding, utilizing pre-generated prompts to inquire about APIs, provide conceptual explanations, and offer code examples.”
WIP: Literature Review on LLMs to Improve Code
Nadendla, S., Karanam, S., & Gehringer, E. F. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328320
“In-IDE assistants, such as the one introduced by Nam et al. [1], show how conversational interfaces built directly into code editors can significantly enhance code understanding, often outperforming traditional web searches in terms of user efficiency.”
Shreeya, S., Kumar, K. B., & Krishna, C. V. (2025) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic64647.2025.11331205
“…that primarily focus on model-level evaluations or theoretical improvements in code generation and repair, CodeGuide emphasizes practical implementation within the development environment by integrating a fine-tuned model which enables offline code generation, correction, and explanation [9][10].”
Detecting and Fixing API Misuses of Data Science Libraries Using Large Language Models
Galappaththi, A., Ribeiro, F., & Nadi, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00083 · Cited by 1
“LLMs have demonstrated natural language understanding capabilities [33] as well as code understanding capabilities [34].”
Interpretable Vulnerability Detection Reports
Mamede, C., Campos, J., Goues, C. L., & Abreu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00168
“Participants were given access to a controlled web-based development environment through GitHub Codespaces to ensure a consistent and realistic setup without the need for local installations [32].”
Chen, L., Deng, J., & Ren, F. (2025) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip68645.2025.11352645
An Empirical Study of Python Library Migration Using Large Language Models
Islam, M., Jha, A. K., Mahmoud, M., Akhmetov, I., & Nadi, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00077 · Cited by 14
“Large Language Models (LLMs) have shown that they can be effective in various software engineering tasks [12–14], including the necessary upstream tasks for library migration, such as code generation [15–17], code comprehension [18], and code transformation [19–21].”
The Productivity Paradox of AI-Powered Development
Fortes, L., Pereira, G. V., Coelho, A., Prikladnicki, R., & Kohl, K. (2026) · IEEE Software · doi:10.1109/ms.2026.3658651 · Cited by 1
Teaching People to Spot Deepfakes: Investigating User Acceptance of LLM-Powered Chatbots
Chen, C., Goh, D. H., & Tang, Z. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2622583 · Cited by 2
“To boost ease of use, designers can implement features such as guided prompts and shortcut buttons to help users use the systems more smoothly and reduce cognitive load (Nam et al., 2024).”
A Systematic Literature Review on Software Visualization Tools for Program Comprehension
Custura, S.-O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15538-2_41
Rayarao, S. R., Bhandarwar, N. D., Dattangire, R., & Vuriti, S. (2025) · Modern Electronics Devices and Intelligent Communication Systems (MEDCOM) · doi:10.1109/medcom67532.2025.11404910 · Cited by 2
CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants
Atif, M., Chopra, K., Tsai, F., Kilic, O., Wang, T., Dong, Z.-H., Benjamin, D., Leggett, C., Lin, M.-F., Calafiura, P., et al. (2026) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw71298.2026.00112 · Cited by 5
Agentic LLM-Assisted Edge AI for CPS/IoT Applications
Li, J., Wu, H., Zhao, H., Kimura, T., Kara, D., Wang, T., Chen, Y., Wang, T., Hu, Y., Misra, A., et al. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00049
“During development, the agent of a subsystem can assist with configuration and validation by leveraging its knowledge of APIs, intended usage patterns and domain assumptions [32].”
Towards Automatic Triage and Taxonomy of Adversary Shell-Scripts
Chandler, J. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00053
“LLMs have shown strong potential for aiding with general source code-understanding [7]–[9].”
Peng, W., Lim, J. Z., Liu, Q., & Wang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7072-0_22 · Cited by 3
Reactive to Agentic:Next Gen AI Agent Transition Framework
Singh, R., Sachdeva, T., Sabita, Arora, S., & Garg, D. (2025) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit68829.2025.11434242
“The emerging generation of smart systems can be described by the feature of Large Language Models (LLM) and multimodal abilities [4].”
Ding, Z., Ding, D., Cao, J., Zhang, Y., & Xue, G. (2026) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2026.3675166
“RecentstudieshavesuccessfullyappliedLLMsin computervision[24],[25],recommendationsystems[26],[27], andcodeunderstanding[28].”
Huang, H., Guo, J., Meng, T., Yang, W., Wang, T., & Weijia Jia, B. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3676410
“L ARGE language models (LLMs) have gained widespread adoption across multiple fields, such as instruction understanding [2], information retrieval [3], software engineering [4], translation [5], education [6] and summarization [7].”
Research on Automated Software Documentation Generation Based on Modular Design
Qiao, M., Li, L., Hou, B., SHANG, Z., & Luo, Z. (2025) · International Conference on Computer Science and Management Technology (ICCSMT) · doi:10.1145/3795154.3795348 · Cited by 1
Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
Tang, N., Chen, C.-R., Fang, Z.-H., Xu, G.-L., Dhakal, M., Shi, Y.-Y., McMillan, C., Huang, Y., & Li, T. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3834377 · Cited by 25
“[7, 34, 42, 51–53] reflect these limitations, examining interactions that were largely single-turn and narrowly scoped, in which the assistant primarily served as a reactive suggestion engine for short code spans within the developer’s active cursor context.”
SLMFL: Structural and semantic fusion for fault localization based on Small Language Models
Fan, X., Hu, J., Zheng, W., Liu, Y., & Gao, S. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108143 · Cited by 1
Integrating AI Literacy, Secure Coding, and Critical Evaluation
Atul Kumar Dnyaan Prasad Global University, Namazov, G., Nadira, A., Rizamat, S., Bobur, M., Mohichekhra, K., & Ozoda, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8450-4.ch008 · Cited by 1
Improving Gemini's Ability to Generate Syntactically Correct Mermaid Code
Sarma, P., Taylor, D. S., Moh, M., & Moh, T.-S. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476269
“Other efforts have focused on enhancing developer workflows by embedding context-rich prompts and explanations directly into development tools [4].”
Li, M., & Krishnamachari, B. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476431
Ahasanuzzaman, M., Oliva, G. A., Hassan, A. E., & Jiang, Z. M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10829-z
“One promising avenue is leveraging Large Language Models (LLMs) to auto-mate the elicitation and summarization of KUs [41].”
LLM-Guided Fuzzing for Pathological Input Generation
Ishimwe, D., & Nguyen, T. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-24839-8_3
Lou, Y., Luo, Y., Song, Y., Li, J., Li, X., & Zhou, X. Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0372-7_24
LLM-MRD: LLM-Guided Multi-view Reasoning Distillation for Fake News Detection
Zhou, W., Tan, S., Gu, E., & Qian, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0375-8_38 · Cited by 1
“The paradigm is shifting from merely imitating outputs to distilling the semantic knowledge of the reasoning process [20], enabling student models to capture nuanced cues [21].”
Onboarding Software Professionals in a Hybrid World
Stol, K.-J., Stray, V. G., Moe, N. B., & Sporsem, T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3694572 · Cited by 1
“Recent research on using large language models to assist in this task may be a promising mechanism for this [97], [98].”
How Helpful Is LLM Assistance in Network Operations? a Case Study at a Large Demonstration Network
Nakamura, R., & Eguchi, K. (2026) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms69089.2026.11668182 · Cited by 1
“Recent significant advancements have enabled Large Language Models (LLMs) to perform various tasks [1]; software engineering is a popular one [2]–[4].”
LLM-SGCF: A Robust Malware Detection Framework with Spatially Guided Convolution
Zhao, L., Huang, H., Li, N., et al. (2026) · Computers · doi:10.3390/computers15060329
Müller, M. (2026) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3797246.3806659
“Intelligent tutoring systems (ITS) in programming education have evolved from rule-based approaches [Jell et al. 2023; Price et al. 2017] to LLM-based assistants embedded in development environments [Kazemitabaar et al. 2024; Nam et al. 2024].”
Concept Labeling and Code Similarity Aware Programming Knowledge TracingPreprint
Li, X., Sun, Q., Yang, H., Wu, J. H., Ouyang, Y., Li, J., & Rong, W. (2026) · Research Square · doi:10.21203/rs.3.rs-9701000/v1
SmartDoc: A Context-Aware Agentic Method Comment Generation Plugin
Etemadi, V., & Robles, G. (2026) · doi:10.1145/3786151.3788601 · Cited by 1
“There are similar tools that offer code understanding using LLMs. Authors in [10] study the procedure that developers take to send their prompt to LLM.”
An Empirical Study of Automated Machine Learning Python Libraries Using Source Code Analysis
O’Leary, C., Lynch, C., & Toosi, F. G. (2026) · Applied Computer Systems · doi:10.2478/acss-2026-0009
“As Machine Learning (ML) systems are data-driven and capable of accurate predictions, their adoption has become increasingly prevalent across a variety of use cases such as forecasting [1]–[2], anomaly detection [3]–[4], classification [5]–[6], text processing [7]–[8] and computer vision [9]–[10].”
de Carvalho, L. H. S., Campos, J. N., Zibaeirad, A., Vieira, M., & Nacif, J. A. M. (2026) · doi:10.5753/wblockchain.2026.23193
Enhancing Java Grading Through Large Language Models
Yao, L., Alsghaier, H., & Zhao, T. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3780023
“5-turbo, enables developers to request context-aware explanations without explicit prompting [6].”
LLM-Enhanced RAG System for Multilingual Fake News Detection
Mim, A. S., Shihab, H. R., & Sattar, A. (2026) · IEEE International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain69676.2026.11546194
Inoue, S., Morisaki, S., Wada, T., & Hattori, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3702933
“Previous studies have highlighted the importance of leveraging generative AI in software development activities [7], [8], [9], [10], [11], and many studies have applied generative AI to source code comprehension activities [12], [13], [14], [15].”
InfluentialFVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification
Wan, L. J., Ho, C.-T., Bai, Y., Yu, C., Chen, D., & Ren, H. M. (2026) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts69484.2026.11563220 · Cited by 2
“Despite their proficiency in generating code across various languages [11], [12], their performance on formal verification tasks remains limited.”
A Diffractive Analysis of GenAI through Perspectives on Productivity
Swift, I. P., & Chattopadhyay, D. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3808045.3808074 · Cited by 1
“Among the topics of production are writing [10, 11, 45, 61], scene creation [20, 38], creating design artifacts [7, 9], music composing [35], professional content creation [29, 30, 40], working with technology [1, 3, 8, 23, 36, 41], and others.”
Natural Language-Focused Software Engineering via Code-Documentation Equivalence
Eghbali, A., Liu, Z., & Pradel, M. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808211 · Cited by 1
How Do Developers Interact with AI? An Exploratory Study on Modeling Developer Programming Behavior
Wu, Y., Li, Z. S., Stolee, K. T., & Xu, B. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808120 · Cited by 3
Peng, C., Jiang, M., Zhou, Y., & Wu, L. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3808141
“LLMs have shown proficiency in analyzing program semantics [44].”
Niemann, S., & Schmid, U. (2026) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet69978.2026.11585137
“…different tasks, for example, making learning materials more engaging or personalized [6; 7], to deliver support and feedback if students need it [8; 9], or even support educators This research was funded by the Bavarian Research Institute for Digital Transformation - bidt with grading…”
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 7
“Studies have demonstrated their ability to automate repetitive coding tasks [10], [11], [12], [13], potentially reducing implementation time [14], augmenting programmer efficacy [15], and expanding access to complex programming techniques for developers with varied skill levels [16].”
Neves, J. P., & Barros, J.-P. (2026) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet69978.2026.11585076
“Feng et al. [7] CodeBERT Guo et al. [9] GraphCodeBERT Alon et al. [8] code2vec Sridhara et al. [5] Code Summarisation Hellendoorn et al. [12] Global Relational Models Nam et al. [11] LLM IDE Tool (GILT)”
Influential
Metadata
- Authors
- Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394745423
- Semantic Scholar
- f5ea85f67fc5ea9cce2c5345ed92740f7377cb2e
- Cited by
- 487
- References
- 61
- Citations by year
- 2026: 184 · 2025: 233 · 2024: 66
- FWCI
- 114.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, program comprehension, conversational interfaces, GPT-3.5-turbo, API documentation, developer productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.