Large Language Models for Software Engineering: A Systematic Literature Review
The DOI matches its Crossref record; 321 of the 321 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To identify relevant benchmarks, we leverage a recent and comprehensive survey on LLMs for SE tasks [33], which analyzed 395 research papers from January 2017 to January 2024.”
From LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (2025) · cites this paper as [33]
In the citing paper
“SWE-Bench is a benchmark designed to assess the ability of LLMs to solve real-world software engineering tasks collected from 12 real-world GitHub repositories.”
From Context as a Tool: Context Management for Long-Horizon SWE-Agents (2025) · cites this paper as Hou et al., 2024
In the citing paper
“The Verified split is a high-quality subset of SWE-Bench consisting of 500 instances, which are manually curated to provide clearer problem descriptions and more reliable evaluation criteria.”
From Context as a Tool: Context Management for Long-Horizon SWE-Agents (2025) · cites this paper as Hou et al., 2024
In the citing paper
“It is increasingly acting as a community contributor-submitting issues, pull requests, code reviews, and security reports [14, 24, 73].”
From To Ban or Not to Ban? How Open Source Projects Govern GenAI Contributions (2026) · cites this paper as [24]
In the citing paper
“However, LLMs struggle with code-related tasks, particularly in terms of correctness, code quality, and other context-specific limitations [11].”
From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [11]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“Simultaneously, researchers have embarked on a series of research activities regarding LLMrelated works, where a number of literature reviews or survey papers have been produced [29, 58, 59, 338].”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 271
- Mayank Agarwal Yikang Shen Bailin Wang Yoon Kim and Jie Chen. 2024. Structured code representations enable data-efficient adaptation of code language models (2024). arXiv:2401.10716. Retrieved from https://arxiv.org/abs/2401.10716 · arXiv
- Lakshya Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram Rajamani. 2023. Monitor-guided decoding of code lms with static analysis of repository context. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 32270–32298. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/662b1774ba8845fc1fa3d1fc0177ceeb-Paper-Conference.pdf
- Baleegh Ahmad Shailja Thakur Benjamin Tan Ramesh Karri and Hammond Pearce. 2023. Fixing hardware security bugs with large language models. arXiv:2302.01215. · arXiv
- Toufique Ahmed Kunal Suresh Pai Premkumar Devanbu and Earl T. Barr. 2023. Improving few-shot prompts with relevant static analysis products. arXiv:2304.06815. · arXiv
- Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen. 2023. Extending source code pre-trained language models to summarise decompiled binarie. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 260–271.
- Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. 2016. A study of visual studio usage in practice. In Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER ’16), Vol. 1. IEEE, 124–134.
- Patrick Bareiß Beatriz Souza Marcelo d’Amorim and Michael Pradel. 2022. Code generation tools (almost) for free? A study of few-shot pre-trained language models on code. arXiv:2206.01335. · arXiv
- Nghi D. Q. Bui Hung Le Yue Wang Junnan Li Akhilesh Deepak Gotmare and Steven C. H. Hoi. 2023. Codetf: One-stop transformer library for state-of-the-art code LLM. arXiv:2306.00029. · arXiv
- Jialun Cao Meiziniu Li Ming Wen and Shing-chi Cheung. 2023. A study on prompt design advantages and limitations of chatgpt for deep learning program repair. arXiv:2304.08191. · arXiv
- Aaron Chan Anant Kharkar Roshanak Zilouchian Moghaddam Yevhen Mohylevskyy Alec Helyar Eslam Kamal Mohamed Elkamhawy and Neel Sundaresan. 2023. Transformer-based vulnerability detection in code at edittime: Zero-shot few-shot or fine-tuning? arXiv:2306.01754. · arXiv
- Yiannis Charalambous Norbert Tihanyi Ridhi Jain Youcheng Sun Mohamed Amine Ferrag and Lucas C. Cordeiro. 2023. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv:2305.14752. · arXiv
- Angelica Chen Jérémy Scheurer Tomasz Korbak Jon Ander Campos Jun Shern Chan Samuel R. Bowman Kyunghyun Cho and Ethan Perez. 2023. Improving code generation by training with natural language feedback. arXiv:2303.16749. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. · arXiv
- Meng Chen Hongyu Zhang Chengcheng Wan Zhao Wei Yong Xu Juhong Wang and Xiaodong Gu. 2023. On the effectiveness of large language models in domain-specific code generation. arXiv:2312.01639. · arXiv
- Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv:2304.05128. · arXiv
- Xinyun Chen Chang Liu and Dawn Song. 2017. Towards synthesizing complex programs from input-output examples. arXiv:1706.01284. · arXiv
- Agnieszka Ciborowska and Kostadin Damevski. 2023. Too few bug reports? Exploring data augmentation for improved changeset-based bug localization. arXiv:2305.16430. · DOI
- Arghavan Moradi Dakhel Amin Nikanjam Vahid Majdinasab Foutse Khomh and Michel C. Desmarais. 2023. Effective test generation using pre-trained large language models and mutation testing. arXiv:2308.16557. · arXiv
- Pantazis Deligiannis Akash Lal Nikita Mehrotra and Aseem Rastogi. 2023. Fixing rust compilation errors using llms. arXiv:2308.05177. · arXiv
- Yinlin Deng Chunqiu Steven Xia Chenyuan Yang Shizhuo Dylan Zhang Shujing Yang and Lingming Zhang. 2023. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. arXiv:2304.02014. · DOI · Paper page
- Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. 2024. Large language models of code fail at completing code with potential bugs. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 41386–41412. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/819cebb05f993840e8a52d7564c5c282-Paper-Conference.pdf
- Guanting Dong Hongyi Yuan Keming Lu Chengpeng Li Mingfeng Xue Dayiheng Liu Wei Wang Zheng Yuan Chang Zhou and Jingren Zhou. 2023. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv:2310.05492. · arXiv
- Yihong Dong Jiazheng Ding Xue Jiang Ge Li Zhuo Li and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. arXiv:2301.09043. · arXiv
- Xueying Du Mingwei Liu Juntao Li Hanlin Wang Xin Peng and Yiling Lou. 2023. Resolving crash bugs via large language models: An empirical study. arXiv:2312.10448. · arXiv
- Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv:2308.01861. Retrieved from https://arxiv.org/abs/2308.01861 · arXiv
- Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for llm-based code completion. arXiv:2401.01701. · arXiv
- Abdelkarim El-Hajjami Nicolas Fafin and Camille Salinesi. 2023. Which ai technique is better to classify requirements? An experiment with SVM LSTM and ChatGPT. arXiv:2311.11547. · arXiv
- Madeline Endres Sarah Fakhoury Saikat Chakraborty and Shuvendu K. Lahiri. 2023. Formalizing natural language intent into program specifications via large language models. arXiv:2310.01831. Retrieved from https://arxiv.org/abs/2310.01831 · arXiv
- Sarah Fakhoury Saikat Chakraborty Madan Musuvathi and Shuvendu K. Lahiri. 2023. Towards generating functionally correct code edits from natural language issue descriptions. arXiv:2304.03816. Retrieved from https://arxiv.org.abs/2304.03816 · arXiv
- Zhiyu Fan Xiang Gao Abhik Roychoudhury and Shin Hwei Tan. 2022. Automated repair of programs from large language models. arXiv:2205.10583. Retrieved from https://arxiv.org/abs/2205.10583 · arXiv
- Mohamed Amine Ferrag Ammar Battah Norbert Tihanyi Merouane Debbah Thierry Lestable and Lucas C. Cordeiro. 2023. Securefalcon: The next cyber reasoning system for cyber security. arXiv:2307.06616. Retrieved from https://arxiv.org/abs/2307.06616 · arXiv
- Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv:2204.05999. Retrieved from https://arxiv.org/abs/2204.05999 · arXiv
- Apurva Gandhi Thong Q. Nguyen Huitian Jiao Robert Steen and Ameya Bhatawdekar. 2023. Natural language commanding via program synthesis. arXiv:2306.03460. Retrieved from https://arxiv.org/abs/2306.03460 · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite Noa Nabeshima Shawn Presser and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https://arxiv.org/abs/2101.00027 · arXiv
- Shuzheng Gao Wenxin Mao Cuiyun Gao Li Li Xing Hu Xin Xia and Michael R. Lyu. 2024. Learning in the wild: Towards leveraging unlabeled data for effectively tuning pre-trained code models. arXiv:2401.01060. Retrieved from https://arxiv.org/abs/2401.01060 · arXiv
- Shuzheng Gao Xin-Cheng Wen Cuiyun Gao Wenxuan Wang and Michael R. Lyu. 2023. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. arXiv:2304.07575. Retrieved from https://arxiv.org/abs/2304.07575 · arXiv
- Zeyu Gao Hao Wang Yuchen Zhou Wenyu Zhu and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv:2311.12420. Retrieved from https://arxiv.org/abs/2311.12420 · arXiv
- Jian Gu, Pasquale Salza, and Harald C. Gall. 2022. Assemble foundation models for automatic code summarization. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 935–946.
- Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu Michele Tufano Shao Kun Deng Colin Clement Dawn Drain Neel Sundaresan Jian Yin Daxin Jiang and Ming Zhou. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv:2009.08366. Retrieved from https://arxiv.org/abs/2009.08366 · arXiv
- Daya Guo Canwen Xu Nan Duan Jian Yin and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. arXiv:2306.14893. Retrieved from https://arxiv.org/abs/2306.14893 · arXiv
- Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li Fuli Luo Yingfei Xiong and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv:2401.14196. Retrieved from https://arxiv.org/abs/2401.14196 · arXiv
- Priyanshu Gupta Avishree Khare Yasharth Bajpai Saikat Chakraborty Sumit Gulwani Aditya Kanade Arjun Radhakrishna Gustavo Soares and Ashish Tiwari. 2023. Grace: Generation using associated code edits. arXiv:2305.14129. Retrieved from https://arxiv.org/abs/2305.14129 · arXiv
- Patrick Hajali and Ignas Budvytis. 2023. Function-constrained program synthesis. arXiv:2311.15500. Retrieved from https://arxiv.org/abs/2311.15500 · arXiv
- Yu Hao Weiteng Chen Ziqiao Zhou and Weidong Cui. 2023. E & v: Prompting large language models to perform static analysis by pseudo-code execution and verification. arXiv:2312.08477. Retrieved from https://arxiv.org/abs/2312.08477 · arXiv
- Junda He Zhou Xin Bowen Xu Ting Zhang Kisub Kim Zhou Yang Ferdian Thung Ivana Irsan and David Lo. 2023. Representation learning for stack overflow posts: How far are we? arXiv:2303.06853. Retrieved from https://arxiv.org/abs/2303.06853 · arXiv
- Jordan Henkel, Denini Silva, Leopoldo Teixeira, Marcelo d’Amorim, and Thomas Reps. 2021. Shipwright: A human-in-the-loop system for dockerfile repair. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 1148–1160.
- Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. Norbert: Transfer learning for requirements classification. In Proceedings of the 2020 IEEE 28th International Requirements Engineering Conference (RE ’20). IEEE, 169–179.
- Sirui Hong Xiawu Zheng Jonathan Chen Yuheng Cheng Jinlin Wang Ceyao Zhang Zili Wang Steven Ka Shing Yau Zijuan Lin Liyang Zhou Chenyu Ran Lingfeng Xiao Chenglin Wu and Jürgen Schmidhuber. 2023. METAGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https://arxiv.org/abs/2308.00352 · arXiv
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790–2799.
- Jie Hu Qian Zhang and Heng Yin. 2023. Augmenting greybox fuzzing with generative AI. arXiv:2306.06782. Retrieved from https://arxiv.org/abs/2306.06782 · arXiv
- Xueyu Hu Kun Kuang Jiankai Sun Hongxia Yang and Fei Wu. 2024. Leveraging print debugging to improve code generation in large language models. arXiv:2401.05319. Retrieved from https://arxiv.org/abs/2401.05319 · arXiv
- Dong Huang Qingwen Bu and Heming Cui. 2023. Codecot and beyond: Learning to program and test like a developer. arXiv: 2308.08784. Retrieved from https://arxiv.org/abs/2308.08784 · arXiv
- Dong Huang Qingwen Bu Jie M. Zhang Michael Luck and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv: 2312.13010. Retrieved from https://arxiv.org/abs/2312.13010 · arXiv
- Di Huang Ziyuan Nan Xing Hu Pengwei Jin Shaohui Peng Yuanbo Wen Rui Zhang Zidong Du Qi Guo Yewen Pu and Yunji Chen. 2023. anpl: Compiling natural programs with interactive decomposition. arXiv:2305.18498. Retrieved from https://arxiv.org/abs/2305.18498 · arXiv
- Qing Huang Yishun Wu Zhenchang Xing He Jiang Yu Cheng and Huan Jin. 2023. Adaptive intellect unleashed: The feasibility of knowledge transfer in large language models. arXiv:2308.04788. Retrieved from https://arxiv.org/abs/2308.04788 · arXiv
- Qing Huang Jiahui Zhu Zhenchang Xing Huan Jin Changjing Wang and Xiwei Xu. 2023. A chain of ai-based solutions for resolving fqns and fixing syntax errors in partial code. arXiv:2306.11981. Retrieved from https://arxiv.org/abs/2306.11981 · arXiv
- Qing Huang Zhou Zou Zhenchang Xing Zhenkang Zuo Xiwei Xu and Qinghua Lu. 2023. Ai chain on large language model for unsupervised control flow graph generation for statically-typed partial code. arXiv: 2306.00757. Retrieved from https://arxiv.org/abs/2306.00757 · arXiv
- Ali Reza Ibrahimzada Yang Chen Ryan Rong and Reyhaneh Jabbarvand. 2023. Automated bug generation in the era of large language models. arXiv: 2310.02407. Retrieved from https://arxiv.org/abs/2310.02407 · arXiv
- Nafis Tanveer Islam and Peyman Najafirad. 2024. Code security vulnerability repair using reinforcement learning with large language models. arXiv:2401.07031. Retrieved from https://arxiv.org/abs/2401.07031 · arXiv
- Abhinav Jain Chima Adiole Thomas Reps Swarat Chaudhuri and Chris Jermaine. 2024. Coarse-tuning models of code with reinforcement learning feedback. Retrieved from https://openreview.net/forum?id=vLqkCvjHRD
- Naman Jain Tianjun Zhang Wei-Lin Chiang Joseph E Gonzalez Koushik Sen and Ion Stoica. 2023. LLM-assisted code cleaning for training accurate code generators. arXiv:2311.14904. Retrieved from https://arxiv.org/abs/2311.14904 · arXiv
- Prithwish Jana Piyush Jha Haoyang Ju Gautham Kishore Aryan Mahajan and Vijay Ganesh. 2023. Attention compilation and solver-based symbolic analysis are all you need. arXiv:2306.06755. Retrieved from https://arxiv.org/abs/2306.06755 · arXiv
- Zhenlan Ji Pingchuan Ma Zongjie Li and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach. arXiv:2310.06680. Retrieved from https://arxiv.org/abs/2310.06680 · arXiv
- Nan Jiang Kevin Liu Thibaud Lutellier and Lin Tan. 2023. Impact of code language models on automated program repair. arXiv:2302.05020. Retrieved from https://arxiv.org/abs/2302.05020 · arXiv
- Nan Jiang Chengxiao Wang Kevin Liu Xiangzhe Xu Lin Tan and Xiangyu Zhang. 2023. Nova \({}^{+}\) : Generative language models for binaries. arXiv:2311.13721. Retrieved from https://arxiv.org/abs/2311.13721 · arXiv
- Shuyang Jiang Yuhao Wang and Yu Wang. 2023. Selfevolve: A code evolution framework via large language models. arXiv:2306.02907. Retrieved from https://arxiv.org/abs/2306.02907 · arXiv
- Xue Jiang Yihong Dong Lecheng Wang Qiwei Shang and Ge Li. 2023. Self-planning code generation with large language model. arXiv:2303.06689. Retrieved from https://arxiv.org/abs/2303.06689 · arXiv
- Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770. Retrieved from https://arxiv.org/abs/2310.06770 · arXiv
- Matthew Jin Syed Shahriar Michele Tufano Xin Shi Shuai Lu Neel Sundaresan and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with LLMS. arXiv:2303.07263. Retrieved from https://arxiv.org/abs/2303.07263 · arXiv
- Xin Jin Jonathan Larson Weiwei Yang and Zhiqiang Lin. 2023. Binary code summarization: Benchmarking CHATGPT/GPT-4 and other large language models. arXiv:2312.09601. Retrieved from https://arxiv.org/abs/2312.09601 · arXiv
- Md Mahir Asef Kabir Sk Adnan Hassan Xiaoyin Wang Ying Wang Hai Yu and Na Meng. 2023. An empirical study of ChatGPT-3.5 on question answering and code maintenance. arXiv:2310.02104. Retrieved from https://arxiv.org/abs/2310.02104 · arXiv
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the International Conference on Machine Learning. PMLR, 5110–5121.
- Sungmin Kang Gabin An and Shin Yoo. 2023. A preliminary evaluation of LLM-based fault localization. arXiv:2308.05487. Retrieved from https://arxiv.org/abs/2308.05487 · arXiv
- Sungmin Kang Juyeon Yoon Nargiz Askarbekkyzy and Shin Yoo. 2023. Evaluating diverse large language models for automatic and general bug reproduction. arXiv:2311.04532. Retrieved from https://arxiv.org/abs/2311.04532 · arXiv
- Sungmin Kang Juyeon Yoon and Shin Yoo. 2022. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. arXiv:2209.11515. Retrieved from https://arxiv.org/abs/2209.11515 · arXiv
- Jai Kannan. 2023. Can llms configure software tools. arXiv:2312.06121. Retrieved from https://arxiv.org/abs/2312.06121 · arXiv
- Rafael-Michael Karampatsis and Charles Sutton. 2020. Scelmo: Source code embeddings from language models. arXiv:2004.13214. Retrieved from https://arxiv.org/abs/2004.13214 · arXiv
- Adam Khakhar Stephen Mell and Osbert Bastani. 2023. PAC prediction sets for large language models of code. arXiv:2302.08703. Retrieved from https://arxiv.org/abs/2302.08703 · arXiv
- Junaed Younus Khan, Md Tawkat Islam Khondaker, Gias Uddin, and Anindya Iqbal. 2021. Automatic detection of five API documentation smells: Practitioners’ perspectives. In Proceedings of the 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’21). IEEE, 318–329.
- Junaed Younus Khan and Gias Uddin. 2022. Automatic detection and analysis of technical debts in peer-review documentation of r packages. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 765–776.
- Mohammad Abdullah Matin Khan M. Saiful Bari Xuan Long Do Weishi Wang Md Rizwan Parvez and Shafiq Joty. 2023. Xcodeeval: A large scale multilingual multitask benchmark for code understanding generation translation and retrieval. arXiv:2303.03004. Retrieved from https://arxiv.org/abs/2303.03004 · arXiv
- Muhammad Fawad Akbar Khan Max Ramsdell Erik Falor and Hamid Karimi. 2023. Assessing the promise and pitfalls of chatgpt for automated code generation. arXiv:2311.02640. Retrieved from https://arxiv.org/abs/2311.02640 · arXiv
- Hiroyuki Kirinuki and Haruto Tanno. 2024. Chatgpt and human synergy in black-box testing: A comparative analysis. arXiv:2401.13924. Retrieved from https://arxiv.org/abs/2401.13924 · arXiv
- Barbara Ann Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical Report EBSE 2007-001. Keele University and Durham University Joint Report. Retrieved from https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf
- 10.5555/1987360.1987364 · doi:10.5555/1987360.1987364
- Takashi Koide Naoki Fukushi Hiroki Nakano and Daiki Chiba. 2023. Detecting phishing sites using chatgpt. arXiv:2306.05816. Retrieved from https://arxiv.org/abs/2306.05816 · arXiv
- Bonan Kou Muhao Chen and Tianyi Zhang. 2023. Automated summarization of stack overflow posts. arXiv:2305.16680. Retrieved from https://arxiv.org/abs/2305.16680 · arXiv
- Bonan Kou Shengmai Chen Zhijie Wang Lei Ma and Tianyi Zhang. 2023. Is model attention aligned with human attention? An empirical study on large language models for code generation. arXiv:2306.01220. Retrieved from https://arxiv.org/abs/2306.01220 · arXiv
- Shuvendu K Lahiri Aaditya Naik Georgios Sakkas Piali Choudhury Curtis von Veh Madanlal Musuvathi Jeevana Priya Inala Chenglong Wang and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. arXiv:2208.05950. Retrieved from https://arxiv.org/abs/2208.05950 · arXiv
- Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning. PMLR, 18319–18345.
- Md Tahmid Rahman Laskar M. Saiful Bari Mizanur Rahman Md Amran Hossen Bhuiyan Shafiq Joty and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. arXiv:2305.18486. Retrieved from https://arxiv.org/abs/2305.18486 · arXiv
- Hung Le Hailin Chen Amrita Saha Akash Gokul Doyen Sahoo and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv:2310.08992. Retrieved from https://arxiv.org/abs/2310.08992 · arXiv
- Dong Li Yelong Shen Ruoming Jin Yi Mao Kuan Wang and Weizhu Chen. 2022. Generation-augmented query expansion for code retrieval. arXiv:2212.10692. Retrieved from https://arxiv.org/abs/2212.10692 · arXiv
- Feng-Lin Li, Jennifer Horkoff, John Mylopoulos, Renata S. S. Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014. Non-functional requirements as qualities, with a spice of ontology. In Proceedings of the 2014 IEEE 22nd International Requirements Engineering Conference (RE ’14). IEEE, 293–302.
- Haochen Li Xin Zhou and Zhiqi Shen. 2024. Rewriting the code: A simple method for large language model augmented code search. arXiv:2401.04514. Retrieved from https://arxiv.org/abs/2401.04514 · arXiv
- Jingyao Li Pengguang Chen and Jiaya Jia. 2023. Motcoder: Elevating large language models with modular of thought for challenging programming tasks. arXiv:2312.15960. Retrieved from https://arxiv.org/abs/2312.15960 · arXiv
- Jingxuan Li, Rui Huang, Wei Li, Kai Yao, and Weiguo Tan. 2021. Toward less hidden cost of code completion with acceptance and ranking models. In Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME ’21). IEEE, 195–205.
- Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Enabling programming thinking in large language models toward code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
- Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Structured chain-of-thought prompting for code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
- Jia Li Ge Li Yunfei Zhao Yongmin Li Zhi Jin Hao Zhu Huanyu Liu Kaibo Liu Lecheng Wang Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yihong Dong Yuqi Zhu Bin Gu and Mengfei Yang. 2024. Deveval: Evaluating code generation in practical software projects. arXiv:2401.06401. Retrieved from https://arxiv.org/abs/2401.06401 · arXiv
- Peng Li Tianxiang Sun Qiong Tang Hang Yan Yuanbin Wu Xuanjing Huang and Xipeng Qiu. 2023. Codeie: Large code generation models are better few-shot information extractors. arXiv:2305.05711. Retrieved from https://arxiv.org/abs/2305.05711 · arXiv
- Tsz-On Li Wenxi Zong Yibo Wang Haoye Tian Ying Wang and Shing-Chi Cheung. 2023. Finding failure-inducing test cases with ChatGPT. arXiv:2304.11686. Retrieved from https://arxiv.org/abs/2304.11686 · arXiv
- Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 14–26.
- Xin-Ye Li Jiang-Tian Xue Zheng Xie and Ming Li. 2023. Think outside the code: Brainstorming boosts large language models in code generation. arXiv:2305.10679. Retrieved from https://arxiv.org/abs/2305.10679 · arXiv
- Yichen Li Yintong Huo Zhihan Jiang Renyi Zhong Pinjia He Yuxin Su and Michael R. Lyu. 2023. Exploring the effectiveness of LLMS in automated logging generation: An empirical study. arXiv:2307.05950. Retrieved from https://arxiv.org/abs/2307.05950 · arXiv
- Youjia Li Jianjun Shi and Zheng Zhang. 2023. A novel approach for rapid development based on chatgpt and prompt engineering. arXiv:2312.13115. Retrieved from https://arxiv.org/abs/2312.13115 · arXiv
- Yao Li Tao Zhang Xiapu Luo Haipeng Cai Sen Fang and Dawei Yuan. 2022. Do pre-trained language models indeed understand software engineering tasks? arXiv:2211.10623. Retrieved from https://arxiv.org/abs/2211.10623 · arXiv
- Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. Cctest: Testing and repairing code completion systems. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1238–1250.
- Zongjie Li Chaozheng Wang Zhibo Liu Haoxuan Wang Shuai Wang and Cuiyun Gao. 2022. Cctest: Testing and repairing code completion systems. arXiv:2208.08289. Retrieved from https://arxiv.org/abs/2208.08289 · arXiv
- Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland-Huang. 2021. Traceability transformed: Generating more accurate links with pre-trained bert models. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 324–335.
- Yu-Chen Lin Akhilesh Kumar Wen-Liang Zhang Norman Chang Muhammad Zakir Rucha Apte Chao Wang and Jyh-Shing Roger Jang. 2023. Applications of large language models in data processing: Innovative approaches to segmenting and renewing information. arXiv:2311.16267. Retrieved from https://arxiv.org/abs/2311.16267 · arXiv
- Chao Liu Xuanlin Bao Hongyu Zhang Neng Zhang Haibo Hu Xiaohong Zhang and Meng Yan. 2023. Improving ChatGPT prompt for code generation. arXiv:2305.08360. Retrieved from https://arxiv.org/abs/2305.08360 · arXiv
- Jinrun Liu Xinyu Tang Linlin Li Panpan Chen and Yepang Liu. 2023. Which is a better programming assistant? A comparative study between chatgpt and stack overflow. arXiv:2308.13851. Retrieved from https://arxiv.org/abs/2308.13851 · arXiv
- Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. arXiv:2305.01210. Retrieved from https://arxiv.org/abs/2305.01210 · arXiv
- Puzhuo Liu Chengnian Sun Yaowen Zheng Xuan Feng Chuan Qin Yuncheng Wang Zhi Li and Limin Sun. 2023. Harnessing the power of llm to support binary taint analysis. arXiv:2310.08275. Retrieved from https://arxiv.org/abs/2310.08275 · arXiv
- Shangqing Liu Bozhi Wu Xiaofei Xie Guozhu Meng and Yang Liu. 2023. Contrabert: Enhancing code pre-trained models via contrastive learning. arXiv:2301.09072. Retrieved from https://arxiv.org/abs/2301.09072 · arXiv
- Tianyang Liu Canwen Xu and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv:2306.03091. Retrieved from https://arxiv.org/abs/2306.03091 · arXiv
- Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2023. Prompt injection attack against LLM-integrated applications. arXiv:2306.05499. Retrieved from https://arxiv.org/abs/2306.05499 · arXiv
- Yue Liu Thanh Le-Cong Ratnadira Widyasari Chakkrit Tantithamthavorn Li Li Xuan-Bach D. Le and David Lo. 2023. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. arXiv:2307.12596. Retrieved from https://arxiv.org/abs/2307.12596 · arXiv
- Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https://arxiv.org/abs/1907.11692 · arXiv
- Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the blank: Context-aware automated text input generation for mobile GUI testing. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE ’23). IEEE, 1355–1367.
- Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2023. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. arXiv:2310.15657. Retrieved from https://arxiv.org/abs/2310.15657 · arXiv
- Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE ’23). IEEE, 647–658.
- Dipeeka Luitel Shabnam Hassani and Mehrdad Sabetzadeh. 2023. Improving requirements completeness: Automated assistance through large language models. arXiv:2308.03784. Retrieved from https://arxiv.org/abs/2308.03784 · arXiv
- Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv:2306.08568. Retrieved from https://arxiv.org/abs/2306.08568 · arXiv
- Lezhi Ma Shangqing Liu Yi Li Xiaofei Xie and Lei Bu. 2024. SpecGen: Automated generation of formal program specifications via large language models. arXiv:2401.08807. Retrieved from https://arxiv.org/abs/2401.08807 · arXiv
- Wei Ma Shangqing Liu Wenhan Wang Qiang Hu Ye Liu Cen Zhang Liming Nie and Yang Liu. 2023. The scope of chatgpt in software engineering: A thorough investigation. arXiv:2305.12138. Retrieved from https://arxiv.org/abs/2305.12138 · arXiv
- Shantanu Mandal Adhrik Chethan Vahid Janfaza S. M. Mahmud Todd A Anderson Javier Turek Jesmin Jahan Tithi and Abdullah Muzahid. 2023. Large language models based automatic synthesis of software specifications. arXiv:2304.09181. Retrieved from https://arxiv.org/abs/2304.09181 · arXiv
- Antonio Mastropaolo, Massimiliano Di Penta, and Gabriele Bavota. 2023. Towards automatically addressing self-admitted technical debt: How far are we?. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 585–597.
- Mohammad Mahdi Mohajer Reem Aleithan Nima Shiri Harzevili Moshi Wei Alvine Boaye Belle Hung Viet Pham and Song Wang. 2023. Skipanalyzer: An embodied agent for code analysis with large language models. arXiv:2310.18532. Retrieved from https://arxiv.org/abs/2310.18532 · arXiv
- Seungjun Moon Yongho Song Hyungjoo Chae Dongjin Kang Taeyoon Kwon Kai Tzu-iunn Ong Seung-won Hwang and Jinyoung Yeo. 2023. Coffee: Boost your code LLMS by fixing bugs with feedback. arXiv:2311.07215. Retrieved from https://arxiv.org/abs/2311.07215 · arXiv
- 10.5555/1858842.1858883 · doi:10.5555/1858842.1858883
- Fangwen Mu Lin Shi Song Wang Zhuohao Yu Binquan Zhang Chenxue Wang Shichao Liu and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based code generation with intention clarification. arXiv:2310.10996. Retrieved from https://arxiv.org/abs/2310.10996 · arXiv
- Manisha Mukherjee and Vincent J Hellendoorn. 2023. Stack over-flowing with results: The case for domain-specific pre-training over one-size-fits-all models. arXiv:2306.03268. Retrieved from https://arxiv.org/abs/2306.03268 · arXiv
- Daye Nam Andrew Macvean Vincent Hellendoorn Bogdan Vasilescu and Brad Myers. 2023. In-IDE generation-based information support with a large language model. arXiv:2307.08177. Retrieved from https://arxiv.org/abs/2307.08177 · arXiv
- Nathalia Nascimento Paulo Alencar and Donald Cowan. 2023. Comparing software developers with ChatGPT: An empirical investigation. arXiv:2305.11837. Retrieved from https://arxiv.org/abs/2305.11837 · arXiv
- Phuong T. Nguyen Juri Di Rocco Claudio Di Sipio Riccardo Rubei Davide Di Ruscio and Massimiliano Di Penta. 2023. Is this snippet written by chatgpt? An empirical study with a codebert-based classifier. arXiv:2307.09381. Retrieved from https://arxiv.org/abs/2307.09381 · arXiv
- Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the International Conference on Machine Learning. PMLR, 26106–26128.
- Ansong Ni Pengcheng Yin Yilun Zhao Martin Riddell Troy Feng Rui Shen Stephen Yin Ye Liu Semih Yavuz Caiming Xiong Shafiq Joty Yingbo Zhou Dragomir Radev and Arman Cohan. 2023. L2ceval: Evaluating language-to-code generation capabilities of large language models. arXiv:2309.17446. Retrieved from https://arxiv.org/abs/2309.17446 · arXiv
- Erik Nijkamp Hiroaki Hayashi Caiming Xiong Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for training llms on programming and natural languages. arXiv:2305.02309. Retrieved from https://arxiv.org/abs/2305.02309 · arXiv
- Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https://arxiv.org/abs/2203.13474 · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:252668917
- David Noever. 2023. Can large language models find and fix vulnerable software? arXiv:2308.10345. Retrieved from https://arxiv.org/abs/2308.10345 · arXiv
- Marcel Ochs, Krishna Narasimhan, and Mira Mezini. 2023. Evaluating and improving transformers pre-trained on ASTS for code completion. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’16). IEEE, 834–844.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
- Shuyin Ouyang Jie M. Zhang Mark Harman and Meng Wang. 2023. Llm is like a box of chocolates: The non-determinism of chatgpt in code generation. arXiv:2308.02828. Retrieved from https://arxiv.org/abs/2308.02828 · arXiv
- Jialing Pan Adrien Sadé Jin Kim Eric Soriano Guillem Sole and Sylvain Flamant. 2023. Stelocoder: A decoder-only LLM for multi-language to python code translation. arXiv:2310.15539. Retrieved from https://arxiv.org/abs/2310.15539 · arXiv
- Rangeet Pan Ali Reza Ibrahimzada Rahul Krishna Divya Sankar Lambert Pouguem Wassi Michele Merler Boris Sobolev Raju Pavuluri Saurabh Sinha and Reyhaneh Jabbarvand. 2023. Understanding the effectiveness of large language models in code translation. arXiv:2308.03109. Retrieved from https://arxiv.org/abs/2308.03109 · arXiv
- Bhargavi Paranjape Scott Lundberg Sameer Singh Hannaneh Hajishirzi Luke Zettlemoyer and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014. Retrieved from https://arxiv.org/abs/2303.09014 · arXiv
- Emilio Parisotto Abdel-rahman Mohamed Rishabh Singh Lihong Li Dengyong Zhou and Pushmeet Kohli. 2016. Neuro-symbolic program synthesis. arXiv:1611.01855. Retrieved from https://arxiv.org/abs/1611.01855 · arXiv
- Arkil Patel Siva Reddy Dzmitry Bahdanau and Pradeep Dasigi. 2023. Evaluating in-context learning of libraries for code generation. arXiv:2311.09635. Retrieved from https://arxiv.org/abs/2311.09635 · arXiv
- Rishov Paul Md Mohib Hossain Masum Hasan and Anindya Iqbal. 2023. Automated program repair based on code review: How do pre-trained transformer models perform? arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
- Rishov Paul Md. Mohib Hossain Mohammed Latif Siddiq Masum Hasan Anindya Iqbal and Joanna C. S. Santos. 2023. Enhancing automated program repair through fine-tuning and prompt engineering. arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
- Tommaso Pegolotti Elias Frantar Dan Alistarh and Markus Püschel. 2023. QIGen: Generating efficient kernels for quantized inference on large language models. arXiv:2307.03738. Retrieved from https://arxiv.org/abs/2307.03738 · arXiv
- Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2023. Can large language models reason about program invariants? In Proceedings of the 40th International Conference on Machine Learning (ICML’23). Vol. 202, 27496–27520.
- Sanyogita Piya and Allison Sullivan. 2023. LLM4TDD: Best practices for test driven development using large language models. arXiv:2312.04687. Retrieved from https://arxiv.org/abs/2312.04687 · arXiv
- Amrit Poudel Jinfeng Lin and Jane Cleland-Huang. 2023. Leveraging transformer-based language models to automate requirements satisfaction assessment. arXiv:2312.04463. Retrieved from https://arxiv.org/abs/2312.04463 · arXiv
- Rohith Pudari and Neil A. Ernst. 2023. From copilot to pilot: Towards AI supported software development. arXiv:2303.04142. Retrieved from https://arxiv.org/abs/2303.04142 · arXiv
- Mengnan Qi Yufan Huang Maoquan Wang Yongqiang Yao Zihan Liu Bin Gu Colin Clement and Neel Sundaresan. 2023. Sut: Active defects probing for transcompiler models. arXiv:2310.14209. Retrieved from https://arxiv.org/abs/2310.14209 · arXiv
- Vu Le Anh Quan Chau Thuan Phat Kiet Van Nguyen Phan The Duy and Van-Hau Pham. 2023. XGV-BERT: Leveraging contextualized language model and graph neural network for efficient software vulnerability detection. arXiv:2309.14677. Retrieved from https://arxiv.org/abs/2309.14677 · arXiv
- Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. Retrieved from https://api.semanticscholar.org/CorpusID:49313245
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
- Sajjad Rahmani AmirHossein Naghshzan and Latifa Guerrouj. 2023. Improving code example recommendations on informal documentation using bert and query-aware LSH: A comparative study. arXiv:2305.03017. Retrieved from https://arxiv.org/abs/2305.03017 · arXiv
- Abhinav Rao Sachin Vashistha Atharva Naik Somak Aditya and Monojit Choudhury. 2023. Tricking llms into disobedience: Understanding analyzing and preventing jailbreaks. arXiv:2305.14965. Retrieved from https://arxiv.org/abs/2305.14965 · arXiv
- Nikitha Rao Jason Tsay Kiran Kate Vincent J. Hellendoorn and Martin Hirzel. 2023. AI for low-code for AI. arXiv:2305.20015. Retrieved from https://arxiv.org/abs/2305.20015 · arXiv
- Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, and Xiaohu Yang. 2023. From misuse to mastery: Enhancing code generation with knowledge-driven ai chaining. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 976–987.
- Tal Ridnik Dedy Kredo and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv:2401.08500. Retrieved from https://arxiv.org/abs/2401.08500 · arXiv
- Baptiste Roziere Marie-Anne Lachaux Marc Szafraniec and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. arXiv:2102.07492. Retrieved from https://arxiv.org/abs/2102.07492 · arXiv
- Fernando Vallecillos Ruiz Anastasiia Grishina Max Hort and Leon Moonen. 2024. A novel approach for automatic program repair using round-trip translation with large language models. arXiv:2401.07994. Retrieved from https://arxiv.org/abs/2401.07994 · arXiv
- Iman Saberi Fatemeh Fard and Fuxiang Chen. 2023. Multilingual adapter-based knowledge aggregation on code summarization for low-resource languages. arXiv:2307.07854. Retrieved from https://arxiv.org/abs/2307.07854 · arXiv
- Ahmed Sadik Antonello Ceravola Frank Joublin and Jibesh Patra. 2023. Analysis of ChatGPT on source code. arXiv:2306.00597. Retrieved from https://arxiv.org/abs/2306.00597 · arXiv
- Anthony Saieva Saikat Chakraborty and Gail Kaiser. 2023. On contrastive learning of semantic similarity forcode to code search. arXiv:2305.03843. Retrieved from https://arxiv.org/abs/2305.03843 · arXiv
- Fardin Ahsan Sakib Saadat Hasan Khan and A. H. M. Karim. 2023. Extending the frontier of ChatGPT: Code generation and debugging. arXiv:2307.08260. Retrieved from https://arxiv.org/abs/2307.08260 · arXiv
- Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ilić Daniel Hesslow Roman Castagné Alexandra Sasha Luccioni François Yvon Matthias Gallé Jonathan Tow Alexander M. Rush Stella Biderman Albert Webson Pawan Sasanka Ammanamanchi Thomas Wang Benoît Sagot Niklas Muennighoff Albert Villanova del Moral Olatunji Ruwase Rachel Bawden Stas Bekman Angelina McMillan-Major Iz Beltagy Huu Nguyen L ucile Saulnier Samson Tan Pedro Ortiz Suarez Victor Sanh Hugo Laurençon Yacine Jernite Julien Launay Margaret Mitchell Colin Raffel Aaron Gokaslan Adi Simhi Aitor Soroa Alham Fikri Aji Amit Alfassy Anna Rogers Ariel Kreisberg Nitzav Canwen Xu Chenghao Mou Chris Emezue Christopher Klamm Colin Leong Daniel van Strien David Ifeoluwa Adelani Dragomir Radev Eduardo González Ponferrada Efrat Levkovizh Ethan Kim Eyal Bar Natan Francesco De Toni Gérard Dupont Germán Kruszewski Giada Pistilli Hady Elsahar Hamza Benyamina Hieu Tran Ian Yu Idris Abdulmumin Isaac Johnson Itziar Gonzalez-Dios Javier de la Rosa Jenny Chim Jesse Dodge Jian Zhu Jonathan Chang Jörg Frohberg Joseph Tobing Joydeep Bhattacharjee Khalid Almubarak Kimbo Chen Kyle Lo Leandro Von Werra Leon Weber Long Phan Loubna Ben allal Ludovic Tanguy Manan Dey Manuel Romero Muñoz Maraim Masoud María Grandury Mario Šaško Max Huang Maximin Coavoux Mayank Singh Mike Tian-Jian Jiang Minh Chien Vu Mohammad A. Jauhar Mustafa Ghaleb Nishant Subramani Nora Kassner Nurulaqilla Khamis Olivier Nguyen Omar Espejel Ona de Gibert Paulo Villegas et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
- Imanol Schlag Sainbayar Sukhbaatar Asli Celikyilmaz Wen tau Yih Jason Weston Jürgen Schmidhuber and Xian Li. 2023. Large language model programs. arXiv:2305.05364. Retrieved from https://arxiv.org/abs/2305.05364 · arXiv
- Martin Schroder. 2023. AutoScrum: Automating project planning using large language models. arXiv:2306.03197. Retrieved from https://arxiv.org/abs/2306.03197 · arXiv
- Oussama Ben Sghaier and Houari Sahraoui. 2023. A multi-step learning approach to assist code review. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 450–460.
- Murray Shanahan. 2022. Talking about large language models. arXiv:2212.03551. Retrieved from https://arxiv.org/abs/2212.03551 · arXiv
- Anton Shapkin Denis Litvinov and Timofey Bryksin. 2023. Entity-augmented code generation. arXiv:2312.08976. Retrieved from https://arxiv.org/abs/2312.08976 · arXiv
- Alexey Shestov Anton Cheshkov Rodion Levichev Ravil Mussabayev Pavel Zadorozhny Evgeny Maslov Chibirev Vadim and Egor Bulychev. 2024. Finetuning large language models for vulnerability detection. arXiv:2401.17010. Retrieved from https://arxiv.org/abs/2401.17010 · arXiv
- Ensheng Shi Yanlin Wang Hongyu Zhang Lun Du Shi Han Dongmei Zhang and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. arXiv:2304.05216. Retrieved from https://arxiv.org/abs/2304.05216 · arXiv
- Ensheng Shi Fengji Zhang Yanlin Wang Bei Chen Lun Du Hongyu Zhang Shi Han Dongmei Zhang and Hongbin Sun. 2023. SoTaNa: The open-source software development assistant. arXiv:2308.13416. Retrieved from https://arxiv.org/abs/2308.13416 · arXiv
- Zejian Shi, Yun Xiong, Xiaolong Zhang, Yao Zhang, Shanshan Li, and Yangyong Zhu. 2022. Cross-modal contrastive learning for code search. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME ’22). IEEE, 94–105.
- Jiho Shin Sepehr Hashtroudi Hadi Hemmati and Song Wang. 2023. Domain adaptation for deep unit test case generation. arXiv:2308.08033. Retrieved from https://arxiv.org/abs/2308.08033 · arXiv
- Jiho Shin Clark Tang Tahmineh Mohati Maleknaz Nayebi Song Wang and Hadi Hemmati. 2023. Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv:2310.10508. Retrieved from https://arxiv.org/abs/2310.10508 · arXiv
- Atsushi Shirafuji Yutaka Watanobe Takumi Ito Makoto Morishita Yuki Nakamura Yusuke Oda and Jun Suzuki. 2023. Exploring the robustness of large language models for solving programming problems. arXiv:2306.14583. Retrieved from https://arxiv.org/abs/2306.14583 · arXiv
- Alexander Shypula Aman Madaan Yimeng Zeng Uri Alon Jacob Gardner Milad Hashemi Graham Neubig Parthasarathy Ranganathan Osbert Bastani and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv:2302.07867. Retrieved from https://arxiv.org/abs/2302.07867 · arXiv
- Mohammed Latif Siddiq Beatrice Casey and Joanna Santos. 2023. A lightweight framework for high-quality code generation. arXiv:2307.08220. Retrieved from https://arxiv.org/abs/2307.08220 · arXiv
- Mohammed Latif Siddiq Joanna Santos Ridwanul Hasan Tanvir Noshin Ulfat Fahmid Al Rifat and Vinicius Carvalho Lopes. 2023. Exploring the effectiveness of large language models in generating unit tests. arXiv:2305.00418. Retrieved from https://arxiv.org/abs/2305.00418 · arXiv
- André Silva Sen Fang and Martin Monperrus. 2023. Repairllama: Efficient representations and fine-tuned adapters for program repair. arXiv:2312.15698. Retrieved from https://arxiv.org/abs/2312.15698 · arXiv
- Benjamin Steenhoek Michele Tufano Neel Sundaresan and Alexey Svyatkovskiy. 2023. Reinforcement learning from automatic feedback for high-quality unit test generation. arXiv:2310.02368. Retrieved from https://arxiv.org/abs/2310.02368 · arXiv
- Hongjin Su Jungo Kasai Chen Henry Wu Weijia Shi Tianlu Wang Jiayi Xin Rui Zhang Mari Ostendorf Luke Zettlemoyer Noah A. Smith and Tao Yu. 2022. Selective annotation makes language models better few-shot learners. arXiv:2209.01975. Retrieved from https://arxiv.org/abs/2209.01975 · arXiv
- Jiamou Sun Zhenchang Xing Qinghua Lu Xiwei Xu Liming Zhu Thong Hoang and Dehai Zhao. 2023. Silent vulnerable dependency alert prediction with vulnerability key aspect explanation. arXiv:2302.07445. Retrieved from https://arxiv.org/abs/2302.07445 · arXiv
- Weisong Sun Chunrong Fang Yudu You Yuchen Chen Yi Liu Chong Wang Jian Zhang Quanjun Zhang Hanwei Qian Wei Zhao Yang Liu and Zhenyu Chen. 2023. A prompt learning framework for source code summarization. arXiv:2312.16066. Retrieved from https://arxiv.org/abs/2312.16066 · arXiv
- Weisong Sun Chunrong Fang Yudu You Yun Miao Yi Liu Yuekang Li Gelei Deng Shenghan Huang Yuchen Chen Quanjun Zhang Hanwei Qian Yang Liu and Zhenyu Chen. 2023. Automatic code summarization via ChatGPT: How far are we? arXiv:2305.12865. Retrieved from https://arxiv.org/abs/2305.12865 · arXiv
- Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Wei Ma Lyuye Zhang Miaolei Shi and Yang Liu. 2024. LLM4Vuln: A unified evaluation framework for decoupling and enhancing LLMS’ vulnerability reasoning. arXiv:2401.16185. Retrieved from https://arxiv.org/abs/2401.16185 · arXiv
- Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Haijun Wang Zhengzi Xu Xiaofei Xie and Yang Liu. 2023. When GPT meets program analysis: Towards intelligent detection of smart contract logic vulnerabilities in gptscan. arXiv:2308.03314. Retrieved from https://arxiv.org/abs/2308.03314 · arXiv
- Zhensu Sun Xiaoning Du Fu Song Shangwen Wang and Li Li. 2024. When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference. arXiv:2401.09964. Retrieved from https://arxiv.org/abs/2401.09964 · arXiv
- Chee Wei Tan Shangxin Guo Man Fai Wong and Ching Nam Hang. 2023. Copilot for xcode: Exploring AI-assisted programming by prompting cloud-based large language models. arXiv:2307.14349. Retrieved from https://arxiv.org/abs/2307.14349 · arXiv
200 of 271 shown
Cited by
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“For example, Hou et al. [14] provided an excellent recent SLR covering 229 research papers from 2017 to 2023 reporting SE tasks tackled, data collection and preprocessing techniques, and strategies for optimising LLM performance (such as prompt engineering).”
BackgroundSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
Self-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
Self-citationLLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
Self-citationWhen LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“Zhao et al. (2023) focuses on how to improve LLM’s semantic understanding of programs through fuzz testing.”
Self-citationLarge Language Models for Cyber Security: A Systematic Literature Review
Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769676 · Cited by 189
Self-citationThe Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06608 · Cited by 118
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education
Pan, W. H., Chok, M. J., Wong, J. L. S., Shin, Y. X., Poon, Y. S., Yang, Z., Chong, C. Y., Lo, D., & Lim, M. K. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640068 · Cited by 66
Self-citationGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
Large Language Model-Based Agents for Software Engineering: A Survey
Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2024) · arXiv · doi:10.48550/arxiv.2409.02977 · Cited by 49
Self-citationShi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
Self-citationOn the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., & Hassan, A. E. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3798166 · Cited by 34
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“Automated software engineering, like code generation and repair [19, 24, 72], has attracted significant interest from academia and industry to facilitate the development of advanced models.”
BackgroundSelf-citationGenerative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions
Xiao, T., Hata, H., Treude, C., & Matsumoto, K. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643773 · Cited by 29
Jelodar, H., Bai, S., Hamedi, P., Mohammadian, H., Razavi-Far, R., & Ghorbani, A. (2026) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2026.104390 · Cited by 27
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?
Wang, G., Sun, Z., Ye, S., Gong, Z., Chen, Y., Zhao, Y., Liang, Q., & Hao, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771933 · Cited by 26
Self-citationToward Automated Simulation Research Workflow through LLM Prompt Engineering Design
Liu, Z., Chai, Y., & Li, J. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01653 · Cited by 24
Human-In-the-Loop Software Development Agents
Takerngsaksiri, W., Pasuksmit, J., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2024) · arXiv · doi:10.48550/arxiv.2411.12924 · Cited by 24
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23
Liu, Z., Zeng, R., Wang, D., Peng, G., Liu, X., Liu, Q., Liu, P., Wang, W., & Wang, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3667895 · Cited by 20
Can GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
Hu, X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 15
Self-citationA disruptive research playbook for studying disruptive innovations
Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3678172 · Cited by 15
Linardon, J., Messer, M., Anderson, C., Liu, C., McClure, Z., Jarman, H. K., Goldberg, S. B., & Torous, J. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2025-301787 · Cited by 10
Security of Language Models for Code: A Systematic Literature Review
Chen, Y., Sun, W., Fang, C., Chen, Z., Ge, Y., Han, T., Zhang, Q., Liu, Y., Chen, Z., & Xu, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735554 · Cited by 10
Self-citationCan Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8
Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective
Ferino, S., Hoda, R., Grundy, J., & Treude, C. (2025) · arXiv · doi:10.48550/arxiv.2511.06428 · Cited by 5
Self-citationCan We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
de Cerqueira, J. A. S., Agbese, M., Rousi, R., Xi, N., Hamari, J., & Abrahamsson, P. (2024) · arXiv · doi:10.48550/arxiv.2411.08881 · Cited by 4
“Several studies are drawing attention to possible problems in adopting LLMs for SE, in particular when syntactically correct but non-functional code is produced, which affects the reliability and effectiveness of LLM-based code generation [14].”
Atkinson, C. F. (2025) · doi:10.21203/rs.3.rs-5702366/v1 · Cited by 3
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Self-citationDeveloping Multi-Agent LLM Applications Through Continuous Human-LLM Co-Programming
Song, H., Goknil, A., Jiang, X., Melum, E., Joe, H., Gazzotti, C., Frascolla, V., Videsjorden, A. N., & Nguyen, P. (2025) · 2025 IEEE/ACM 4th International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00013 · Cited by 3
AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling
Huang, Y., Song, J., Hu, Q., Juefei-Xu, F., & Ma, L. (2024) · arXiv · doi:10.48550/arxiv.2408.03573 · Cited by 2
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1
Context as a Tool: Context Management for Long-Horizon SWE-Agents
Liu, S., Yang, J., Jiang, B., Li, Y., Guo, J., Liu, X., & Dai, B. (2025) · arXiv · doi:10.48550/arxiv.2512.22087 · Cited by 35
Investigating the Impact of Dark Patterns on LLM-Based Web Agents
Ersoy, D., Lee, B., Shreekumar, A., Arunasalam, A., Ibrahim, M., Bianchi, A., & Berkay Celik, Z. (2026) · 2026 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp63933.2026.00042 · Cited by 1
“We develop TrickyArena as a testbed of websites containing dark patterns with which agents interact to complete tasks 1 .”
To Ban or Not to Ban? How Open Source Projects Govern GenAI Contributions
Yang, W., He, R., & Zhou, M. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3837448 · Cited by 1
Self-Admitted GenAI Usage in Open-Source Software
Xiao, T., Fan, Y., Calefato, F., Treude, C., Kula, R. G., Hata, H., & Baltes, S. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3681886 · Cited by 1
Predicate Renaming via Large Language Models
Gentili, E., Ribeiro, T., Riguzzi, F., & Inoue, K. (2026) · Machine Learning · doi:10.1007/s10994-026-07075-1 · Cited by 1
AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work
Choudhuri, R., Badea, C., Bird, C., Butler, J. L., DeLine, R., & Houck, B. (2026) · Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3786583.3786885 · Cited by 1
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Zheng, M., Han, K., Li, B., Xu, H., Tian, Y., He, W., Zhou, H., Guo, J., Hu, H., Ma, L., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.12344
“The minimum acceptable sizes range from 8 to 10 instances per language ( 𝐾 ∗ ∈ [ 8, 10 ] ).”
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Chen, Z., Sun, Z., Shi, Y., Lo, D., & Jiang, L. (2026) · arXiv · doi:10.48550/arxiv.2607.01211 · Cited by 9
Self-citation"We Permit the Use of AI, but [...]": The Landscape of AI Policies in Popular Open Source Projects
Hora, A., Robbes, R., & Zacchiroli, S. (2026) · arXiv · doi:10.48550/arxiv.2609.07542
Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code
Khojah, R., Gomes de Oliveira Neto, F., Mohamad, M., Frattini, J., & Leitner, P. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3834423
AI Policy, Disclosure, and Human in the Loop: How Are Contribution Guidelines Adapting to GenAI?
Hora, A., & Robbes, R. (2026) · arXiv · doi:10.48550/arxiv.2605.16706
Making AI Visible, Not Vanished: How AI Policies Reshape Developer Experience on GitHub
Chen, Y., Zimmermann, T., & Trinkenreich, B. (2026) · arXiv · doi:10.48550/arxiv.2608.03329
Do AI-Generated Pull Requests Get Rejected More? (Yes but Why?)
Wang, Y., & Yang, Z. (2026) · Proceedings of the 23rd International Conference on Mining Software Repositories · doi:10.1145/3793302.3793568
Self-citationLarbi, M., Akli, A., Papadakis, M., Bouyousfi, R., Cordy, M., Sarro, F., & Traon, Y. L. (2025) · arXiv · doi:10.48550/arxiv.2507.20439
Metadata
- Authors
- Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H.
- Affiliations
- Huazhong University of Science and Technology (CN)
- Monash University (AU)
- Singapore Management University (SG)
- Beihang University (CN)
- Hong Kong Polytechnic University (HK)
- ORCID
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-11-30
- Volume
- 33
- Issue
- 8
- Pages
- 1-79
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402665833
- Cited by
- 1,029
- References
- 321
- Citations by year
- 2026: 469 · 2025: 417 · 2024: 135
- FWCI
- 58.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Funders
- National Natural Science Foundation of China (62072046)
- Key R&D Program of Hubei (2023BAB017, 2023BAB079)
- National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, software engineering tasks, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.