CoChat

Large Language Models for Software Engineering: A Systematic Literature Review

Hou, Xinyi, Zhao, Yanjie, Liu, Yue, Yang, Zhou, Wang, Kailong, Li, Li, Luo, Xiapu, Lo, David, Grundy, John, Wang, Haoyu · 2024 · ACM Transactions on Software Engineering and Methodology

The DOI matches its Crossref record; 321 of the 321 references it cites were found in a registry or an index.

Abstract

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR .

Overview

Integrity signals

Registry

Crossref · 10.1145/3695988

Cites

321 of 321 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Research works printed but not linked: 271

  1. Mayank Agarwal Yikang Shen Bailin Wang Yoon Kim and Jie Chen. 2024. Structured code representations enable data-efficient adaptation of code language models (2024). arXiv:2401.10716. Retrieved from https://arxiv.org/abs/2401.10716 · arXiv
  2. Lakshya Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram Rajamani. 2023. Monitor-guided decoding of code lms with static analysis of repository context. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 32270–32298. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/662b1774ba8845fc1fa3d1fc0177ceeb-Paper-Conference.pdf
  3. Baleegh Ahmad Shailja Thakur Benjamin Tan Ramesh Karri and Hammond Pearce. 2023. Fixing hardware security bugs with large language models. arXiv:2302.01215. · arXiv
  4. Toufique Ahmed Kunal Suresh Pai Premkumar Devanbu and Earl T. Barr. 2023. Improving few-shot prompts with relevant static analysis products. arXiv:2304.06815. · arXiv
  5. Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen. 2023. Extending source code pre-trained language models to summarise decompiled binarie. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 260–271.
  6. Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. 2016. A study of visual studio usage in practice. In Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER ’16), Vol. 1. IEEE, 124–134.
  7. Patrick Bareiß Beatriz Souza Marcelo d’Amorim and Michael Pradel. 2022. Code generation tools (almost) for free? A study of few-shot pre-trained language models on code. arXiv:2206.01335. · arXiv
  8. Nghi D. Q. Bui Hung Le Yue Wang Junnan Li Akhilesh Deepak Gotmare and Steven C. H. Hoi. 2023. Codetf: One-stop transformer library for state-of-the-art code LLM. arXiv:2306.00029. · arXiv
  9. Jialun Cao Meiziniu Li Ming Wen and Shing-chi Cheung. 2023. A study on prompt design advantages and limitations of chatgpt for deep learning program repair. arXiv:2304.08191. · arXiv
  10. Aaron Chan Anant Kharkar Roshanak Zilouchian Moghaddam Yevhen Mohylevskyy Alec Helyar Eslam Kamal Mohamed Elkamhawy and Neel Sundaresan. 2023. Transformer-based vulnerability detection in code at edittime: Zero-shot few-shot or fine-tuning? arXiv:2306.01754. · arXiv
  11. Yiannis Charalambous Norbert Tihanyi Ridhi Jain Youcheng Sun Mohamed Amine Ferrag and Lucas C. Cordeiro. 2023. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv:2305.14752. · arXiv
  12. Angelica Chen Jérémy Scheurer Tomasz Korbak Jon Ander Campos Jun Shern Chan Samuel R. Bowman Kyunghyun Cho and Ethan Perez. 2023. Improving code generation by training with natural language feedback. arXiv:2303.16749. · arXiv
  13. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. · arXiv
  14. Meng Chen Hongyu Zhang Chengcheng Wan Zhao Wei Yong Xu Juhong Wang and Xiaodong Gu. 2023. On the effectiveness of large language models in domain-specific code generation. arXiv:2312.01639. · arXiv
  15. Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv:2304.05128. · arXiv
  16. Xinyun Chen Chang Liu and Dawn Song. 2017. Towards synthesizing complex programs from input-output examples. arXiv:1706.01284. · arXiv
  17. Agnieszka Ciborowska and Kostadin Damevski. 2023. Too few bug reports? Exploring data augmentation for improved changeset-based bug localization. arXiv:2305.16430. · DOI
  18. Arghavan Moradi Dakhel Amin Nikanjam Vahid Majdinasab Foutse Khomh and Michel C. Desmarais. 2023. Effective test generation using pre-trained large language models and mutation testing. arXiv:2308.16557. · arXiv
  19. Pantazis Deligiannis Akash Lal Nikita Mehrotra and Aseem Rastogi. 2023. Fixing rust compilation errors using llms. arXiv:2308.05177. · arXiv
  20. Yinlin Deng Chunqiu Steven Xia Chenyuan Yang Shizhuo Dylan Zhang Shujing Yang and Lingming Zhang. 2023. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. arXiv:2304.02014. · DOI · Paper page
  21. Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. 2024. Large language models of code fail at completing code with potential bugs. In Advances in Neural Information Processing Systems. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 41386–41412. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/file/819cebb05f993840e8a52d7564c5c282-Paper-Conference.pdf
  22. Guanting Dong Hongyi Yuan Keming Lu Chengpeng Li Mingfeng Xue Dayiheng Liu Wei Wang Zheng Yuan Chang Zhou and Jingren Zhou. 2023. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv:2310.05492. · arXiv
  23. Yihong Dong Jiazheng Ding Xue Jiang Ge Li Zhuo Li and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. arXiv:2301.09043. · arXiv
  24. Xueying Du Mingwei Liu Juntao Li Hanlin Wang Xin Peng and Yiling Lou. 2023. Resolving crash bugs via large language models: An empirical study. arXiv:2312.10448. · arXiv
  25. Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv:2308.01861. Retrieved from https://arxiv.org/abs/2308.01861 · arXiv
  26. Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for llm-based code completion. arXiv:2401.01701. · arXiv
  27. Abdelkarim El-Hajjami Nicolas Fafin and Camille Salinesi. 2023. Which ai technique is better to classify requirements? An experiment with SVM LSTM and ChatGPT. arXiv:2311.11547. · arXiv
  28. Madeline Endres Sarah Fakhoury Saikat Chakraborty and Shuvendu K. Lahiri. 2023. Formalizing natural language intent into program specifications via large language models. arXiv:2310.01831. Retrieved from https://arxiv.org/abs/2310.01831 · arXiv
  29. Sarah Fakhoury Saikat Chakraborty Madan Musuvathi and Shuvendu K. Lahiri. 2023. Towards generating functionally correct code edits from natural language issue descriptions. arXiv:2304.03816. Retrieved from https://arxiv.org.abs/2304.03816 · arXiv
  30. Zhiyu Fan Xiang Gao Abhik Roychoudhury and Shin Hwei Tan. 2022. Automated repair of programs from large language models. arXiv:2205.10583. Retrieved from https://arxiv.org/abs/2205.10583 · arXiv
  31. Mohamed Amine Ferrag Ammar Battah Norbert Tihanyi Merouane Debbah Thierry Lestable and Lucas C. Cordeiro. 2023. Securefalcon: The next cyber reasoning system for cyber security. arXiv:2307.06616. Retrieved from https://arxiv.org/abs/2307.06616 · arXiv
  32. Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv:2204.05999. Retrieved from https://arxiv.org/abs/2204.05999 · arXiv
  33. Apurva Gandhi Thong Q. Nguyen Huitian Jiao Robert Steen and Ameya Bhatawdekar. 2023. Natural language commanding via program synthesis. arXiv:2306.03460. Retrieved from https://arxiv.org/abs/2306.03460 · arXiv
  34. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite Noa Nabeshima Shawn Presser and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https://arxiv.org/abs/2101.00027 · arXiv
  35. Shuzheng Gao Wenxin Mao Cuiyun Gao Li Li Xing Hu Xin Xia and Michael R. Lyu. 2024. Learning in the wild: Towards leveraging unlabeled data for effectively tuning pre-trained code models. arXiv:2401.01060. Retrieved from https://arxiv.org/abs/2401.01060 · arXiv
  36. Shuzheng Gao Xin-Cheng Wen Cuiyun Gao Wenxuan Wang and Michael R. Lyu. 2023. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. arXiv:2304.07575. Retrieved from https://arxiv.org/abs/2304.07575 · arXiv
  37. Zeyu Gao Hao Wang Yuchen Zhou Wenyu Zhu and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv:2311.12420. Retrieved from https://arxiv.org/abs/2311.12420 · arXiv
  38. Jian Gu, Pasquale Salza, and Harald C. Gall. 2022. Assemble foundation models for automatic code summarization. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 935–946.
  39. Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu Michele Tufano Shao Kun Deng Colin Clement Dawn Drain Neel Sundaresan Jian Yin Daxin Jiang and Ming Zhou. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv:2009.08366. Retrieved from https://arxiv.org/abs/2009.08366 · arXiv
  40. Daya Guo Canwen Xu Nan Duan Jian Yin and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. arXiv:2306.14893. Retrieved from https://arxiv.org/abs/2306.14893 · arXiv
  41. Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li Fuli Luo Yingfei Xiong and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv:2401.14196. Retrieved from https://arxiv.org/abs/2401.14196 · arXiv
  42. Priyanshu Gupta Avishree Khare Yasharth Bajpai Saikat Chakraborty Sumit Gulwani Aditya Kanade Arjun Radhakrishna Gustavo Soares and Ashish Tiwari. 2023. Grace: Generation using associated code edits. arXiv:2305.14129. Retrieved from https://arxiv.org/abs/2305.14129 · arXiv
  43. Patrick Hajali and Ignas Budvytis. 2023. Function-constrained program synthesis. arXiv:2311.15500. Retrieved from https://arxiv.org/abs/2311.15500 · arXiv
  44. Yu Hao Weiteng Chen Ziqiao Zhou and Weidong Cui. 2023. E & v: Prompting large language models to perform static analysis by pseudo-code execution and verification. arXiv:2312.08477. Retrieved from https://arxiv.org/abs/2312.08477 · arXiv
  45. Junda He Zhou Xin Bowen Xu Ting Zhang Kisub Kim Zhou Yang Ferdian Thung Ivana Irsan and David Lo. 2023. Representation learning for stack overflow posts: How far are we? arXiv:2303.06853. Retrieved from https://arxiv.org/abs/2303.06853 · arXiv
  46. Jordan Henkel, Denini Silva, Leopoldo Teixeira, Marcelo d’Amorim, and Thomas Reps. 2021. Shipwright: A human-in-the-loop system for dockerfile repair. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 1148–1160.
  47. Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. Norbert: Transfer learning for requirements classification. In Proceedings of the 2020 IEEE 28th International Requirements Engineering Conference (RE ’20). IEEE, 169–179.
  48. Sirui Hong Xiawu Zheng Jonathan Chen Yuheng Cheng Jinlin Wang Ceyao Zhang Zili Wang Steven Ka Shing Yau Zijuan Lin Liyang Zhou Chenyu Ran Lingfeng Xiao Chenglin Wu and Jürgen Schmidhuber. 2023. METAGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https://arxiv.org/abs/2308.00352 · arXiv
  49. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790–2799.
  50. Jie Hu Qian Zhang and Heng Yin. 2023. Augmenting greybox fuzzing with generative AI. arXiv:2306.06782. Retrieved from https://arxiv.org/abs/2306.06782 · arXiv
  51. Xueyu Hu Kun Kuang Jiankai Sun Hongxia Yang and Fei Wu. 2024. Leveraging print debugging to improve code generation in large language models. arXiv:2401.05319. Retrieved from https://arxiv.org/abs/2401.05319 · arXiv
  52. Dong Huang Qingwen Bu and Heming Cui. 2023. Codecot and beyond: Learning to program and test like a developer. arXiv: 2308.08784. Retrieved from https://arxiv.org/abs/2308.08784 · arXiv
  53. Dong Huang Qingwen Bu Jie M. Zhang Michael Luck and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv: 2312.13010. Retrieved from https://arxiv.org/abs/2312.13010 · arXiv
  54. Di Huang Ziyuan Nan Xing Hu Pengwei Jin Shaohui Peng Yuanbo Wen Rui Zhang Zidong Du Qi Guo Yewen Pu and Yunji Chen. 2023. anpl: Compiling natural programs with interactive decomposition. arXiv:2305.18498. Retrieved from https://arxiv.org/abs/2305.18498 · arXiv
  55. Qing Huang Yishun Wu Zhenchang Xing He Jiang Yu Cheng and Huan Jin. 2023. Adaptive intellect unleashed: The feasibility of knowledge transfer in large language models. arXiv:2308.04788. Retrieved from https://arxiv.org/abs/2308.04788 · arXiv
  56. Qing Huang Jiahui Zhu Zhenchang Xing Huan Jin Changjing Wang and Xiwei Xu. 2023. A chain of ai-based solutions for resolving fqns and fixing syntax errors in partial code. arXiv:2306.11981. Retrieved from https://arxiv.org/abs/2306.11981 · arXiv
  57. Qing Huang Zhou Zou Zhenchang Xing Zhenkang Zuo Xiwei Xu and Qinghua Lu. 2023. Ai chain on large language model for unsupervised control flow graph generation for statically-typed partial code. arXiv: 2306.00757. Retrieved from https://arxiv.org/abs/2306.00757 · arXiv
  58. Ali Reza Ibrahimzada Yang Chen Ryan Rong and Reyhaneh Jabbarvand. 2023. Automated bug generation in the era of large language models. arXiv: 2310.02407. Retrieved from https://arxiv.org/abs/2310.02407 · arXiv
  59. Nafis Tanveer Islam and Peyman Najafirad. 2024. Code security vulnerability repair using reinforcement learning with large language models. arXiv:2401.07031. Retrieved from https://arxiv.org/abs/2401.07031 · arXiv
  60. Abhinav Jain Chima Adiole Thomas Reps Swarat Chaudhuri and Chris Jermaine. 2024. Coarse-tuning models of code with reinforcement learning feedback. Retrieved from https://openreview.net/forum?id=vLqkCvjHRD
  61. Naman Jain Tianjun Zhang Wei-Lin Chiang Joseph E Gonzalez Koushik Sen and Ion Stoica. 2023. LLM-assisted code cleaning for training accurate code generators. arXiv:2311.14904. Retrieved from https://arxiv.org/abs/2311.14904 · arXiv
  62. Prithwish Jana Piyush Jha Haoyang Ju Gautham Kishore Aryan Mahajan and Vijay Ganesh. 2023. Attention compilation and solver-based symbolic analysis are all you need. arXiv:2306.06755. Retrieved from https://arxiv.org/abs/2306.06755 · arXiv
  63. Zhenlan Ji Pingchuan Ma Zongjie Li and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach. arXiv:2310.06680. Retrieved from https://arxiv.org/abs/2310.06680 · arXiv
  64. Nan Jiang Kevin Liu Thibaud Lutellier and Lin Tan. 2023. Impact of code language models on automated program repair. arXiv:2302.05020. Retrieved from https://arxiv.org/abs/2302.05020 · arXiv
  65. Nan Jiang Chengxiao Wang Kevin Liu Xiangzhe Xu Lin Tan and Xiangyu Zhang. 2023. Nova \({}^{+}\) : Generative language models for binaries. arXiv:2311.13721. Retrieved from https://arxiv.org/abs/2311.13721 · arXiv
  66. Shuyang Jiang Yuhao Wang and Yu Wang. 2023. Selfevolve: A code evolution framework via large language models. arXiv:2306.02907. Retrieved from https://arxiv.org/abs/2306.02907 · arXiv
  67. Xue Jiang Yihong Dong Lecheng Wang Qiwei Shang and Ge Li. 2023. Self-planning code generation with large language model. arXiv:2303.06689. Retrieved from https://arxiv.org/abs/2303.06689 · arXiv
  68. Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world GitHub issues? arXiv:2310.06770. Retrieved from https://arxiv.org/abs/2310.06770 · arXiv
  69. Matthew Jin Syed Shahriar Michele Tufano Xin Shi Shuai Lu Neel Sundaresan and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with LLMS. arXiv:2303.07263. Retrieved from https://arxiv.org/abs/2303.07263 · arXiv
  70. Xin Jin Jonathan Larson Weiwei Yang and Zhiqiang Lin. 2023. Binary code summarization: Benchmarking CHATGPT/GPT-4 and other large language models. arXiv:2312.09601. Retrieved from https://arxiv.org/abs/2312.09601 · arXiv
  71. Md Mahir Asef Kabir Sk Adnan Hassan Xiaoyin Wang Ying Wang Hai Yu and Na Meng. 2023. An empirical study of ChatGPT-3.5 on question answering and code maintenance. arXiv:2310.02104. Retrieved from https://arxiv.org/abs/2310.02104 · arXiv
  72. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the International Conference on Machine Learning. PMLR, 5110–5121.
  73. Sungmin Kang Gabin An and Shin Yoo. 2023. A preliminary evaluation of LLM-based fault localization. arXiv:2308.05487. Retrieved from https://arxiv.org/abs/2308.05487 · arXiv
  74. Sungmin Kang Juyeon Yoon Nargiz Askarbekkyzy and Shin Yoo. 2023. Evaluating diverse large language models for automatic and general bug reproduction. arXiv:2311.04532. Retrieved from https://arxiv.org/abs/2311.04532 · arXiv
  75. Sungmin Kang Juyeon Yoon and Shin Yoo. 2022. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. arXiv:2209.11515. Retrieved from https://arxiv.org/abs/2209.11515 · arXiv
  76. Jai Kannan. 2023. Can llms configure software tools. arXiv:2312.06121. Retrieved from https://arxiv.org/abs/2312.06121 · arXiv
  77. Rafael-Michael Karampatsis and Charles Sutton. 2020. Scelmo: Source code embeddings from language models. arXiv:2004.13214. Retrieved from https://arxiv.org/abs/2004.13214 · arXiv
  78. Adam Khakhar Stephen Mell and Osbert Bastani. 2023. PAC prediction sets for large language models of code. arXiv:2302.08703. Retrieved from https://arxiv.org/abs/2302.08703 · arXiv
  79. Junaed Younus Khan, Md Tawkat Islam Khondaker, Gias Uddin, and Anindya Iqbal. 2021. Automatic detection of five API documentation smells: Practitioners’ perspectives. In Proceedings of the 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’21). IEEE, 318–329.
  80. Junaed Younus Khan and Gias Uddin. 2022. Automatic detection and analysis of technical debts in peer-review documentation of r packages. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22). IEEE, 765–776.
  81. Mohammad Abdullah Matin Khan M. Saiful Bari Xuan Long Do Weishi Wang Md Rizwan Parvez and Shafiq Joty. 2023. Xcodeeval: A large scale multilingual multitask benchmark for code understanding generation translation and retrieval. arXiv:2303.03004. Retrieved from https://arxiv.org/abs/2303.03004 · arXiv
  82. Muhammad Fawad Akbar Khan Max Ramsdell Erik Falor and Hamid Karimi. 2023. Assessing the promise and pitfalls of chatgpt for automated code generation. arXiv:2311.02640. Retrieved from https://arxiv.org/abs/2311.02640 · arXiv
  83. Hiroyuki Kirinuki and Haruto Tanno. 2024. Chatgpt and human synergy in black-box testing: A comparative analysis. arXiv:2401.13924. Retrieved from https://arxiv.org/abs/2401.13924 · arXiv
  84. Barbara Ann Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical Report EBSE 2007-001. Keele University and Durham University Joint Report. Retrieved from https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf
  85. 10.5555/1987360.1987364 · doi:10.5555/1987360.1987364
  86. Takashi Koide Naoki Fukushi Hiroki Nakano and Daiki Chiba. 2023. Detecting phishing sites using chatgpt. arXiv:2306.05816. Retrieved from https://arxiv.org/abs/2306.05816 · arXiv
  87. Bonan Kou Muhao Chen and Tianyi Zhang. 2023. Automated summarization of stack overflow posts. arXiv:2305.16680. Retrieved from https://arxiv.org/abs/2305.16680 · arXiv
  88. Bonan Kou Shengmai Chen Zhijie Wang Lei Ma and Tianyi Zhang. 2023. Is model attention aligned with human attention? An empirical study on large language models for code generation. arXiv:2306.01220. Retrieved from https://arxiv.org/abs/2306.01220 · arXiv
  89. Shuvendu K Lahiri Aaditya Naik Georgios Sakkas Piali Choudhury Curtis von Veh Madanlal Musuvathi Jeevana Priya Inala Chenglong Wang and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. arXiv:2208.05950. Retrieved from https://arxiv.org/abs/2208.05950 · arXiv
  90. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning. PMLR, 18319–18345.
  91. Md Tahmid Rahman Laskar M. Saiful Bari Mizanur Rahman Md Amran Hossen Bhuiyan Shafiq Joty and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. arXiv:2305.18486. Retrieved from https://arxiv.org/abs/2305.18486 · arXiv
  92. Hung Le Hailin Chen Amrita Saha Akash Gokul Doyen Sahoo and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv:2310.08992. Retrieved from https://arxiv.org/abs/2310.08992 · arXiv
  93. Dong Li Yelong Shen Ruoming Jin Yi Mao Kuan Wang and Weizhu Chen. 2022. Generation-augmented query expansion for code retrieval. arXiv:2212.10692. Retrieved from https://arxiv.org/abs/2212.10692 · arXiv
  94. Feng-Lin Li, Jennifer Horkoff, John Mylopoulos, Renata S. S. Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014. Non-functional requirements as qualities, with a spice of ontology. In Proceedings of the 2014 IEEE 22nd International Requirements Engineering Conference (RE ’14). IEEE, 293–302.
  95. Haochen Li Xin Zhou and Zhiqi Shen. 2024. Rewriting the code: A simple method for large language model augmented code search. arXiv:2401.04514. Retrieved from https://arxiv.org/abs/2401.04514 · arXiv
  96. Jingyao Li Pengguang Chen and Jiaya Jia. 2023. Motcoder: Elevating large language models with modular of thought for challenging programming tasks. arXiv:2312.15960. Retrieved from https://arxiv.org/abs/2312.15960 · arXiv
  97. Jingxuan Li, Rui Huang, Wei Li, Kai Yao, and Weiguo Tan. 2021. Toward less hidden cost of code completion with acceptance and ranking models. In Proceedings of the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME ’21). IEEE, 195–205.
  98. Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Enabling programming thinking in large language models toward code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
  99. Jia Li Ge Li Yongmin Li and Zhi Jin. 2023. Structured chain-of-thought prompting for code generation. arXiv:2305.06599. Retrieved from https://arxiv.org/abs/2305.06599 · arXiv
  100. Jia Li Ge Li Yunfei Zhao Yongmin Li Zhi Jin Hao Zhu Huanyu Liu Kaibo Liu Lecheng Wang Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yihong Dong Yuqi Zhu Bin Gu and Mengfei Yang. 2024. Deveval: Evaluating code generation in practical software projects. arXiv:2401.06401. Retrieved from https://arxiv.org/abs/2401.06401 · arXiv
  101. Peng Li Tianxiang Sun Qiong Tang Hang Yan Yuanbin Wu Xuanjing Huang and Xipeng Qiu. 2023. Codeie: Large code generation models are better few-shot information extractors. arXiv:2305.05711. Retrieved from https://arxiv.org/abs/2305.05711 · arXiv
  102. Tsz-On Li Wenxi Zong Yibo Wang Haoye Tian Ying Wang and Shing-Chi Cheung. 2023. Finding failure-inducing test cases with ChatGPT. arXiv:2304.11686. Retrieved from https://arxiv.org/abs/2304.11686 · arXiv
  103. Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 14–26.
  104. Xin-Ye Li Jiang-Tian Xue Zheng Xie and Ming Li. 2023. Think outside the code: Brainstorming boosts large language models in code generation. arXiv:2305.10679. Retrieved from https://arxiv.org/abs/2305.10679 · arXiv
  105. Yichen Li Yintong Huo Zhihan Jiang Renyi Zhong Pinjia He Yuxin Su and Michael R. Lyu. 2023. Exploring the effectiveness of LLMS in automated logging generation: An empirical study. arXiv:2307.05950. Retrieved from https://arxiv.org/abs/2307.05950 · arXiv
  106. Youjia Li Jianjun Shi and Zheng Zhang. 2023. A novel approach for rapid development based on chatgpt and prompt engineering. arXiv:2312.13115. Retrieved from https://arxiv.org/abs/2312.13115 · arXiv
  107. Yao Li Tao Zhang Xiapu Luo Haipeng Cai Sen Fang and Dawei Yuan. 2022. Do pre-trained language models indeed understand software engineering tasks? arXiv:2211.10623. Retrieved from https://arxiv.org/abs/2211.10623 · arXiv
  108. Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. Cctest: Testing and repairing code completion systems. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1238–1250.
  109. Zongjie Li Chaozheng Wang Zhibo Liu Haoxuan Wang Shuai Wang and Cuiyun Gao. 2022. Cctest: Testing and repairing code completion systems. arXiv:2208.08289. Retrieved from https://arxiv.org/abs/2208.08289 · arXiv
  110. Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland-Huang. 2021. Traceability transformed: Generating more accurate links with pre-trained bert models. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 324–335.
  111. Yu-Chen Lin Akhilesh Kumar Wen-Liang Zhang Norman Chang Muhammad Zakir Rucha Apte Chao Wang and Jyh-Shing Roger Jang. 2023. Applications of large language models in data processing: Innovative approaches to segmenting and renewing information. arXiv:2311.16267. Retrieved from https://arxiv.org/abs/2311.16267 · arXiv
  112. Chao Liu Xuanlin Bao Hongyu Zhang Neng Zhang Haibo Hu Xiaohong Zhang and Meng Yan. 2023. Improving ChatGPT prompt for code generation. arXiv:2305.08360. Retrieved from https://arxiv.org/abs/2305.08360 · arXiv
  113. Jinrun Liu Xinyu Tang Linlin Li Panpan Chen and Yepang Liu. 2023. Which is a better programming assistant? A comparative study between chatgpt and stack overflow. arXiv:2308.13851. Retrieved from https://arxiv.org/abs/2308.13851 · arXiv
  114. Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. arXiv:2305.01210. Retrieved from https://arxiv.org/abs/2305.01210 · arXiv
  115. Puzhuo Liu Chengnian Sun Yaowen Zheng Xuan Feng Chuan Qin Yuncheng Wang Zhi Li and Limin Sun. 2023. Harnessing the power of llm to support binary taint analysis. arXiv:2310.08275. Retrieved from https://arxiv.org/abs/2310.08275 · arXiv
  116. Shangqing Liu Bozhi Wu Xiaofei Xie Guozhu Meng and Yang Liu. 2023. Contrabert: Enhancing code pre-trained models via contrastive learning. arXiv:2301.09072. Retrieved from https://arxiv.org/abs/2301.09072 · arXiv
  117. Tianyang Liu Canwen Xu and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv:2306.03091. Retrieved from https://arxiv.org/abs/2306.03091 · arXiv
  118. Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2023. Prompt injection attack against LLM-integrated applications. arXiv:2306.05499. Retrieved from https://arxiv.org/abs/2306.05499 · arXiv
  119. Yue Liu Thanh Le-Cong Ratnadira Widyasari Chakkrit Tantithamthavorn Li Li Xuan-Bach D. Le and David Lo. 2023. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. arXiv:2307.12596. Retrieved from https://arxiv.org/abs/2307.12596 · arXiv
  120. Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https://arxiv.org/abs/1907.11692 · arXiv
  121. Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the blank: Context-aware automated text input generation for mobile GUI testing. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE ’23). IEEE, 1355–1367.
  122. Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2023. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. arXiv:2310.15657. Retrieved from https://arxiv.org/abs/2310.15657 · arXiv
  123. Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE ’23). IEEE, 647–658.
  124. Dipeeka Luitel Shabnam Hassani and Mehrdad Sabetzadeh. 2023. Improving requirements completeness: Automated assistance through large language models. arXiv:2308.03784. Retrieved from https://arxiv.org/abs/2308.03784 · arXiv
  125. Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv:2306.08568. Retrieved from https://arxiv.org/abs/2306.08568 · arXiv
  126. Lezhi Ma Shangqing Liu Yi Li Xiaofei Xie and Lei Bu. 2024. SpecGen: Automated generation of formal program specifications via large language models. arXiv:2401.08807. Retrieved from https://arxiv.org/abs/2401.08807 · arXiv
  127. Wei Ma Shangqing Liu Wenhan Wang Qiang Hu Ye Liu Cen Zhang Liming Nie and Yang Liu. 2023. The scope of chatgpt in software engineering: A thorough investigation. arXiv:2305.12138. Retrieved from https://arxiv.org/abs/2305.12138 · arXiv
  128. Shantanu Mandal Adhrik Chethan Vahid Janfaza S. M. Mahmud Todd A Anderson Javier Turek Jesmin Jahan Tithi and Abdullah Muzahid. 2023. Large language models based automatic synthesis of software specifications. arXiv:2304.09181. Retrieved from https://arxiv.org/abs/2304.09181 · arXiv
  129. Antonio Mastropaolo, Massimiliano Di Penta, and Gabriele Bavota. 2023. Towards automatically addressing self-admitted technical debt: How far are we?. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 585–597.
  130. Mohammad Mahdi Mohajer Reem Aleithan Nima Shiri Harzevili Moshi Wei Alvine Boaye Belle Hung Viet Pham and Song Wang. 2023. Skipanalyzer: An embodied agent for code analysis with large language models. arXiv:2310.18532. Retrieved from https://arxiv.org/abs/2310.18532 · arXiv
  131. Seungjun Moon Yongho Song Hyungjoo Chae Dongjin Kang Taeyoon Kwon Kai Tzu-iunn Ong Seung-won Hwang and Jinyoung Yeo. 2023. Coffee: Boost your code LLMS by fixing bugs with feedback. arXiv:2311.07215. Retrieved from https://arxiv.org/abs/2311.07215 · arXiv
  132. 10.5555/1858842.1858883 · doi:10.5555/1858842.1858883
  133. Fangwen Mu Lin Shi Song Wang Zhuohao Yu Binquan Zhang Chenxue Wang Shichao Liu and Qing Wang. 2023. ClarifyGPT: Empowering LLM-based code generation with intention clarification. arXiv:2310.10996. Retrieved from https://arxiv.org/abs/2310.10996 · arXiv
  134. Manisha Mukherjee and Vincent J Hellendoorn. 2023. Stack over-flowing with results: The case for domain-specific pre-training over one-size-fits-all models. arXiv:2306.03268. Retrieved from https://arxiv.org/abs/2306.03268 · arXiv
  135. Daye Nam Andrew Macvean Vincent Hellendoorn Bogdan Vasilescu and Brad Myers. 2023. In-IDE generation-based information support with a large language model. arXiv:2307.08177. Retrieved from https://arxiv.org/abs/2307.08177 · arXiv
  136. Nathalia Nascimento Paulo Alencar and Donald Cowan. 2023. Comparing software developers with ChatGPT: An empirical investigation. arXiv:2305.11837. Retrieved from https://arxiv.org/abs/2305.11837 · arXiv
  137. Phuong T. Nguyen Juri Di Rocco Claudio Di Sipio Riccardo Rubei Davide Di Ruscio and Massimiliano Di Penta. 2023. Is this snippet written by chatgpt? An empirical study with a codebert-based classifier. arXiv:2307.09381. Retrieved from https://arxiv.org/abs/2307.09381 · arXiv
  138. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the International Conference on Machine Learning. PMLR, 26106–26128.
  139. Ansong Ni Pengcheng Yin Yilun Zhao Martin Riddell Troy Feng Rui Shen Stephen Yin Ye Liu Semih Yavuz Caiming Xiong Shafiq Joty Yingbo Zhou Dragomir Radev and Arman Cohan. 2023. L2ceval: Evaluating language-to-code generation capabilities of large language models. arXiv:2309.17446. Retrieved from https://arxiv.org/abs/2309.17446 · arXiv
  140. Erik Nijkamp Hiroaki Hayashi Caiming Xiong Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for training llms on programming and natural languages. arXiv:2305.02309. Retrieved from https://arxiv.org/abs/2305.02309 · arXiv
  141. Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https://arxiv.org/abs/2203.13474 · arXiv
  142. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:252668917
  143. David Noever. 2023. Can large language models find and fix vulnerable software? arXiv:2308.10345. Retrieved from https://arxiv.org/abs/2308.10345 · arXiv
  144. Marcel Ochs, Krishna Narasimhan, and Mira Mezini. 2023. Evaluating and improving transformers pre-trained on ASTS for code completion. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’16). IEEE, 834–844.
  145. OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
  146. Shuyin Ouyang Jie M. Zhang Mark Harman and Meng Wang. 2023. Llm is like a box of chocolates: The non-determinism of chatgpt in code generation. arXiv:2308.02828. Retrieved from https://arxiv.org/abs/2308.02828 · arXiv
  147. Jialing Pan Adrien Sadé Jin Kim Eric Soriano Guillem Sole and Sylvain Flamant. 2023. Stelocoder: A decoder-only LLM for multi-language to python code translation. arXiv:2310.15539. Retrieved from https://arxiv.org/abs/2310.15539 · arXiv
  148. Rangeet Pan Ali Reza Ibrahimzada Rahul Krishna Divya Sankar Lambert Pouguem Wassi Michele Merler Boris Sobolev Raju Pavuluri Saurabh Sinha and Reyhaneh Jabbarvand. 2023. Understanding the effectiveness of large language models in code translation. arXiv:2308.03109. Retrieved from https://arxiv.org/abs/2308.03109 · arXiv
  149. Bhargavi Paranjape Scott Lundberg Sameer Singh Hannaneh Hajishirzi Luke Zettlemoyer and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014. Retrieved from https://arxiv.org/abs/2303.09014 · arXiv
  150. Emilio Parisotto Abdel-rahman Mohamed Rishabh Singh Lihong Li Dengyong Zhou and Pushmeet Kohli. 2016. Neuro-symbolic program synthesis. arXiv:1611.01855. Retrieved from https://arxiv.org/abs/1611.01855 · arXiv
  151. Arkil Patel Siva Reddy Dzmitry Bahdanau and Pradeep Dasigi. 2023. Evaluating in-context learning of libraries for code generation. arXiv:2311.09635. Retrieved from https://arxiv.org/abs/2311.09635 · arXiv
  152. Rishov Paul Md Mohib Hossain Masum Hasan and Anindya Iqbal. 2023. Automated program repair based on code review: How do pre-trained transformer models perform? arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
  153. Rishov Paul Md. Mohib Hossain Mohammed Latif Siddiq Masum Hasan Anindya Iqbal and Joanna C. S. Santos. 2023. Enhancing automated program repair through fine-tuning and prompt engineering. arXiv:2304.07840. Retrieved from https://arxiv.org/abs/2304.07840 · arXiv
  154. Tommaso Pegolotti Elias Frantar Dan Alistarh and Markus Püschel. 2023. QIGen: Generating efficient kernels for quantized inference on large language models. arXiv:2307.03738. Retrieved from https://arxiv.org/abs/2307.03738 · arXiv
  155. Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2023. Can large language models reason about program invariants? In Proceedings of the 40th International Conference on Machine Learning (ICML’23). Vol. 202, 27496–27520.
  156. Sanyogita Piya and Allison Sullivan. 2023. LLM4TDD: Best practices for test driven development using large language models. arXiv:2312.04687. Retrieved from https://arxiv.org/abs/2312.04687 · arXiv
  157. Amrit Poudel Jinfeng Lin and Jane Cleland-Huang. 2023. Leveraging transformer-based language models to automate requirements satisfaction assessment. arXiv:2312.04463. Retrieved from https://arxiv.org/abs/2312.04463 · arXiv
  158. Rohith Pudari and Neil A. Ernst. 2023. From copilot to pilot: Towards AI supported software development. arXiv:2303.04142. Retrieved from https://arxiv.org/abs/2303.04142 · arXiv
  159. Mengnan Qi Yufan Huang Maoquan Wang Yongqiang Yao Zihan Liu Bin Gu Colin Clement and Neel Sundaresan. 2023. Sut: Active defects probing for transcompiler models. arXiv:2310.14209. Retrieved from https://arxiv.org/abs/2310.14209 · arXiv
  160. Vu Le Anh Quan Chau Thuan Phat Kiet Van Nguyen Phan The Duy and Van-Hau Pham. 2023. XGV-BERT: Leveraging contextualized language model and graph neural network for efficient software vulnerability detection. arXiv:2309.14677. Retrieved from https://arxiv.org/abs/2309.14677 · arXiv
  161. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. Retrieved from https://api.semanticscholar.org/CorpusID:49313245
  162. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
  163. 10.5555/3455716.3455856 · doi:10.5555/3455716.3455856
  164. Sajjad Rahmani AmirHossein Naghshzan and Latifa Guerrouj. 2023. Improving code example recommendations on informal documentation using bert and query-aware LSH: A comparative study. arXiv:2305.03017. Retrieved from https://arxiv.org/abs/2305.03017 · arXiv
  165. Abhinav Rao Sachin Vashistha Atharva Naik Somak Aditya and Monojit Choudhury. 2023. Tricking llms into disobedience: Understanding analyzing and preventing jailbreaks. arXiv:2305.14965. Retrieved from https://arxiv.org/abs/2305.14965 · arXiv
  166. Nikitha Rao Jason Tsay Kiran Kate Vincent J. Hellendoorn and Martin Hirzel. 2023. AI for low-code for AI. arXiv:2305.20015. Retrieved from https://arxiv.org/abs/2305.20015 · arXiv
  167. Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, and Xiaohu Yang. 2023. From misuse to mastery: Enhancing code generation with knowledge-driven ai chaining. In Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE ’23). IEEE, 976–987.
  168. Tal Ridnik Dedy Kredo and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv:2401.08500. Retrieved from https://arxiv.org/abs/2401.08500 · arXiv
  169. Baptiste Roziere Marie-Anne Lachaux Marc Szafraniec and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. arXiv:2102.07492. Retrieved from https://arxiv.org/abs/2102.07492 · arXiv
  170. Fernando Vallecillos Ruiz Anastasiia Grishina Max Hort and Leon Moonen. 2024. A novel approach for automatic program repair using round-trip translation with large language models. arXiv:2401.07994. Retrieved from https://arxiv.org/abs/2401.07994 · arXiv
  171. Iman Saberi Fatemeh Fard and Fuxiang Chen. 2023. Multilingual adapter-based knowledge aggregation on code summarization for low-resource languages. arXiv:2307.07854. Retrieved from https://arxiv.org/abs/2307.07854 · arXiv
  172. Ahmed Sadik Antonello Ceravola Frank Joublin and Jibesh Patra. 2023. Analysis of ChatGPT on source code. arXiv:2306.00597. Retrieved from https://arxiv.org/abs/2306.00597 · arXiv
  173. Anthony Saieva Saikat Chakraborty and Gail Kaiser. 2023. On contrastive learning of semantic similarity forcode to code search. arXiv:2305.03843. Retrieved from https://arxiv.org/abs/2305.03843 · arXiv
  174. Fardin Ahsan Sakib Saadat Hasan Khan and A. H. M. Karim. 2023. Extending the frontier of ChatGPT: Code generation and debugging. arXiv:2307.08260. Retrieved from https://arxiv.org/abs/2307.08260 · arXiv
  175. Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ilić Daniel Hesslow Roman Castagné Alexandra Sasha Luccioni François Yvon Matthias Gallé Jonathan Tow Alexander M. Rush Stella Biderman Albert Webson Pawan Sasanka Ammanamanchi Thomas Wang Benoît Sagot Niklas Muennighoff Albert Villanova del Moral Olatunji Ruwase Rachel Bawden Stas Bekman Angelina McMillan-Major Iz Beltagy Huu Nguyen L ucile Saulnier Samson Tan Pedro Ortiz Suarez Victor Sanh Hugo Laurençon Yacine Jernite Julien Launay Margaret Mitchell Colin Raffel Aaron Gokaslan Adi Simhi Aitor Soroa Alham Fikri Aji Amit Alfassy Anna Rogers Ariel Kreisberg Nitzav Canwen Xu Chenghao Mou Chris Emezue Christopher Klamm Colin Leong Daniel van Strien David Ifeoluwa Adelani Dragomir Radev Eduardo González Ponferrada Efrat Levkovizh Ethan Kim Eyal Bar Natan Francesco De Toni Gérard Dupont Germán Kruszewski Giada Pistilli Hady Elsahar Hamza Benyamina Hieu Tran Ian Yu Idris Abdulmumin Isaac Johnson Itziar Gonzalez-Dios Javier de la Rosa Jenny Chim Jesse Dodge Jian Zhu Jonathan Chang Jörg Frohberg Joseph Tobing Joydeep Bhattacharjee Khalid Almubarak Kimbo Chen Kyle Lo Leandro Von Werra Leon Weber Long Phan Loubna Ben allal Ludovic Tanguy Manan Dey Manuel Romero Muñoz Maraim Masoud María Grandury Mario Šaško Max Huang Maximin Coavoux Mayank Singh Mike Tian-Jian Jiang Minh Chien Vu Mohammad A. Jauhar Mustafa Ghaleb Nishant Subramani Nora Kassner Nurulaqilla Khamis Olivier Nguyen Omar Espejel Ona de Gibert Paulo Villegas et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
  176. Imanol Schlag Sainbayar Sukhbaatar Asli Celikyilmaz Wen tau Yih Jason Weston Jürgen Schmidhuber and Xian Li. 2023. Large language model programs. arXiv:2305.05364. Retrieved from https://arxiv.org/abs/2305.05364 · arXiv
  177. Martin Schroder. 2023. AutoScrum: Automating project planning using large language models. arXiv:2306.03197. Retrieved from https://arxiv.org/abs/2306.03197 · arXiv
  178. Oussama Ben Sghaier and Houari Sahraoui. 2023. A multi-step learning approach to assist code review. In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’23). IEEE, 450–460.
  179. Murray Shanahan. 2022. Talking about large language models. arXiv:2212.03551. Retrieved from https://arxiv.org/abs/2212.03551 · arXiv
  180. Anton Shapkin Denis Litvinov and Timofey Bryksin. 2023. Entity-augmented code generation. arXiv:2312.08976. Retrieved from https://arxiv.org/abs/2312.08976 · arXiv
  181. Alexey Shestov Anton Cheshkov Rodion Levichev Ravil Mussabayev Pavel Zadorozhny Evgeny Maslov Chibirev Vadim and Egor Bulychev. 2024. Finetuning large language models for vulnerability detection. arXiv:2401.17010. Retrieved from https://arxiv.org/abs/2401.17010 · arXiv
  182. Ensheng Shi Yanlin Wang Hongyu Zhang Lun Du Shi Han Dongmei Zhang and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. arXiv:2304.05216. Retrieved from https://arxiv.org/abs/2304.05216 · arXiv
  183. Ensheng Shi Fengji Zhang Yanlin Wang Bei Chen Lun Du Hongyu Zhang Shi Han Dongmei Zhang and Hongbin Sun. 2023. SoTaNa: The open-source software development assistant. arXiv:2308.13416. Retrieved from https://arxiv.org/abs/2308.13416 · arXiv
  184. Zejian Shi, Yun Xiong, Xiaolong Zhang, Yao Zhang, Shanshan Li, and Yangyong Zhu. 2022. Cross-modal contrastive learning for code search. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME ’22). IEEE, 94–105.
  185. Jiho Shin Sepehr Hashtroudi Hadi Hemmati and Song Wang. 2023. Domain adaptation for deep unit test case generation. arXiv:2308.08033. Retrieved from https://arxiv.org/abs/2308.08033 · arXiv
  186. Jiho Shin Clark Tang Tahmineh Mohati Maleknaz Nayebi Song Wang and Hadi Hemmati. 2023. Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv:2310.10508. Retrieved from https://arxiv.org/abs/2310.10508 · arXiv
  187. Atsushi Shirafuji Yutaka Watanobe Takumi Ito Makoto Morishita Yuki Nakamura Yusuke Oda and Jun Suzuki. 2023. Exploring the robustness of large language models for solving programming problems. arXiv:2306.14583. Retrieved from https://arxiv.org/abs/2306.14583 · arXiv
  188. Alexander Shypula Aman Madaan Yimeng Zeng Uri Alon Jacob Gardner Milad Hashemi Graham Neubig Parthasarathy Ranganathan Osbert Bastani and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv:2302.07867. Retrieved from https://arxiv.org/abs/2302.07867 · arXiv
  189. Mohammed Latif Siddiq Beatrice Casey and Joanna Santos. 2023. A lightweight framework for high-quality code generation. arXiv:2307.08220. Retrieved from https://arxiv.org/abs/2307.08220 · arXiv
  190. Mohammed Latif Siddiq Joanna Santos Ridwanul Hasan Tanvir Noshin Ulfat Fahmid Al Rifat and Vinicius Carvalho Lopes. 2023. Exploring the effectiveness of large language models in generating unit tests. arXiv:2305.00418. Retrieved from https://arxiv.org/abs/2305.00418 · arXiv
  191. André Silva Sen Fang and Martin Monperrus. 2023. Repairllama: Efficient representations and fine-tuned adapters for program repair. arXiv:2312.15698. Retrieved from https://arxiv.org/abs/2312.15698 · arXiv
  192. Benjamin Steenhoek Michele Tufano Neel Sundaresan and Alexey Svyatkovskiy. 2023. Reinforcement learning from automatic feedback for high-quality unit test generation. arXiv:2310.02368. Retrieved from https://arxiv.org/abs/2310.02368 · arXiv
  193. Hongjin Su Jungo Kasai Chen Henry Wu Weijia Shi Tianlu Wang Jiayi Xin Rui Zhang Mari Ostendorf Luke Zettlemoyer Noah A. Smith and Tao Yu. 2022. Selective annotation makes language models better few-shot learners. arXiv:2209.01975. Retrieved from https://arxiv.org/abs/2209.01975 · arXiv
  194. Jiamou Sun Zhenchang Xing Qinghua Lu Xiwei Xu Liming Zhu Thong Hoang and Dehai Zhao. 2023. Silent vulnerable dependency alert prediction with vulnerability key aspect explanation. arXiv:2302.07445. Retrieved from https://arxiv.org/abs/2302.07445 · arXiv
  195. Weisong Sun Chunrong Fang Yudu You Yuchen Chen Yi Liu Chong Wang Jian Zhang Quanjun Zhang Hanwei Qian Wei Zhao Yang Liu and Zhenyu Chen. 2023. A prompt learning framework for source code summarization. arXiv:2312.16066. Retrieved from https://arxiv.org/abs/2312.16066 · arXiv
  196. Weisong Sun Chunrong Fang Yudu You Yun Miao Yi Liu Yuekang Li Gelei Deng Shenghan Huang Yuchen Chen Quanjun Zhang Hanwei Qian Yang Liu and Zhenyu Chen. 2023. Automatic code summarization via ChatGPT: How far are we? arXiv:2305.12865. Retrieved from https://arxiv.org/abs/2305.12865 · arXiv
  197. Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Wei Ma Lyuye Zhang Miaolei Shi and Yang Liu. 2024. LLM4Vuln: A unified evaluation framework for decoupling and enhancing LLMS’ vulnerability reasoning. arXiv:2401.16185. Retrieved from https://arxiv.org/abs/2401.16185 · arXiv
  198. Yuqiang Sun Daoyuan Wu Yue Xue Han Liu Haijun Wang Zhengzi Xu Xiaofei Xie and Yang Liu. 2023. When GPT meets program analysis: Towards intelligent detection of smart contract logic vulnerabilities in gptscan. arXiv:2308.03314. Retrieved from https://arxiv.org/abs/2308.03314 · arXiv
  199. Zhensu Sun Xiaoning Du Fu Song Shangwen Wang and Li Li. 2024. When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference. arXiv:2401.09964. Retrieved from https://arxiv.org/abs/2401.09964 · arXiv
  200. Chee Wei Tan Shangxin Guo Man Fai Wong and Ching Nam Hang. 2023. Copilot for xcode: Exploring AI-assisted programming by prompting cloud-based large language models. arXiv:2307.14349. Retrieved from https://arxiv.org/abs/2307.14349 · arXiv

200 of 271 shown

Cited by

See all 1,029

Metadata

Authors
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H.
Affiliations
  • Huazhong University of Science and Technology (CN)
  • Monash University (AU)
  • Singapore Management University (SG)
  • Beihang University (CN)
  • Hong Kong Polytechnic University (HK)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2024-11-30
Volume
33
Issue
8
Pages
1-79
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4402665833
Cited by
1,029
References
321
Citations by year
2026: 469 · 2025: 417 · 2024: 135
FWCI
58.78
Type
journal article
Language
en
Open access
closed
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
Funders
  • National Natural Science Foundation of China (62072046)
  • Key R&D Program of Hubei (2023BAB017, 2023BAB079)
  • National Research Foundation, under its Investigatorship (NRF-NRFI08-2022-0002)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, software engineering tasks, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.