CoChat
Sign Up
Cited by 650Software Engineering ResearchSoftware Engineering Techniques and PracticesAdvanced Software Engineering Methodologies

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Large Language Models for Software Engineering: Survey and Open Problems

Open access
Fan, Angela, Gokkaya, Beliz, Harman, Mark, Lyubarskiy, Mitya, Sengupta, Shubho, Yoo, Shin, Zhang, Jie M.·2023·2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)
DOI 10.1109/icse-fose59343.2023.00008
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper provides a survey of the emerging area of Large Language Models (LLMs) for Software Engineering (SE). It also sets out open research challenges for the application of LLMs to technical problems faced by software engineers. LLMs' emergent properties bring novelty and creativity with applications right across the spectrum of Software Engineering activities including coding, design, requirements, repair, refactoring, performance improvement, documentation and analytics. However, these very same emergent properties also pose significant technical challenges; we need techniques that can reliably weed out incorrect solutions, such as hallucinations. Our survey reveals the pivotal role that hybrid techniques (traditional SE plus LLMs) have to play in the development and deployment of reliable, efficient and effective LLM-based SE.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icse-fose59343.2023.00008

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Fig. 1 A mapping between software development activities, research domains, and the paper structure
Fig. 1 A mapping between software development activities, research domains, and the paper structureCC-BY · From the PDF
Fig. 3 Proportions of LLM papers and SE papers about LLMs. By 'about LLMs', we mean that either the title or the abstract of a preprint contains 'LLM', 'Large Language Model', or 'GPT'. The blue line denotes the percentage of the number of preprints about LLMs out of the number of all preprints in the CS category. The orange line denotes the percentage of the number of preprints about LLMs in cs.SE and cs.PL categories out of all preprints about LLMs
Fig. 3 Proportions of LLM papers and SE papers about LLMs. By 'about LLMs', we mean that either the title or the abstract of a preprint contains 'LLM', 'Large Language Model', or 'GPT'. The blue line denotes the percentage of the number of preprints about LLMs out of the number of all preprints in the CS category. The orange line denotes the percentage of the number of preprints about LLMs in cs.SE and cs.PL categories out of all preprints about LLMsCC-BY · From the PDF
Fig. 4 Code generation leaderboard for the HumanEval benchmark. These methods are either based on LLMs or LLMs themselves.
Fig. 4 Code generation leaderboard for the HumanEval benchmark. These methods are either based on LLMs or LLMs themselves.CC-BY · From the PDF
Fig. 5 The Widening Scope of Program Transformation
Fig. 5 The Widening Scope of Program TransformationCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 195

  1. Ma, 2023, The scope of ChatGPT in software engineering: A thorough investigation
  2. 10.1016/j.jss.2013.02.061 · DOI
  3. 10.1145/2408776.2408795 · DOI
  4. 10.1109/ICST.2015.7102580 · DOI
  5. 10.1109/TSE.2014.2372785 · DOI · Paper page
  6. Shin, 2023, Assessing evaluation met-rics for neural test oracle generation, arXiv preprint
  7. 10.1109/TEVC.2017.2693219 · DOI
  8. Cantino, 2021, Prompt engineering tips and tricks with GPT-3
  9. Ouyang, 2023, Llm is like a box of chocolates: the non-determinism of chatgpt in code generation, arXiv prnote
  10. 10.1145/1321631.1321749 · DOI
  11. 10.1145/2379776.2379787 · DOI
  12. 10.1145/1985793.1985795 · DOI
  13. 10.1007/978-3-642-25231-0_1 · DOI
  14. Hou, 2023, Large language models for software engineering: A systematic literature review, arXiv prnote
  15. Goyal, 2023, News summarization and evaluation in the era of gpt-3
  16. Xie, 2023, A survey for biomedical text summarization: From pre-trained to large language models
  17. Kheiri, 2023, Sentimentgpt: Exploiting gpt for advanced sentiment analysis and its departure from current machine learning
  18. Vaswani, 2017, Attention is all you need
  19. Zhao, Github Copilot now has a better AI model and new capabilities
  20. 2023, GitHub CEO says Copilot will write 80% of code sooner than later
  21. 2023, OpenAI, GPT-4 Technical report
  22. Bubeck, 2023, Sparks of Artificial General Intelligence: Early experiments with GPT-4
  23. 10.1109/MS.2003.1241365 · DOI
  24. 10.1007/978-3-540-69062-7_8 · DOI
  25. 10.1109/icngn59831.2023.10396810 · DOI
  26. 10.1145/3551349.3560417 · DOI
  27. Luitel, 2023, Improving requirements completeness: Automated assistance through large language models, arXiv prnote
  28. 10.1145/1595696.1595728 · DOI
  29. Murali, 2023, Codecompose: A large-scale industrial deployment of ai-assisted code authoring
  30. 10.1561/2500000010 · DOI
  31. 10.1109/ICSE.2012.6227135 · DOI · Paper page
  32. 10.1145/2635868.2635898 · DOI
  33. 10.1145/1882291.1882315 · DOI
  34. 10.1145/321992.321996 · DOI
  35. Nijkamp, 2023, Codegen2: Lessons for training llms on programming and natural languages
  36. 10.1145/3491102.3501870 · DOI · Paper page
  37. 10.1615/JMachLearnModelComput.2023048492 · DOI
  38. 10.1109/CAIN58948.2023.00031 · DOI
  39. Li, 2023, Towards Enhancing In-Context Learning for Code Generation
  40. Doderlein, 2023, Pi-loting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic?
  41. He, 2023, Controlling Large Language Models to Generate Secure and Vulnerable Code
  42. White, 2023, ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design
  43. 10.1109/ICSE48619.2023.00179 · DOI · Paper page
  44. 10.1109/iccad57390.2023.10323812 · DOI
  45. Jiang, 2023, Selfevolve: A code evolution framework via large language models, arXiv prnote
  46. 10.18653/v1/2023.acl-long.45 · DOI
  47. Ahmed, 2023, Improving Few-Shot Prompts with Relevant Static Analysis Products
  48. Shin, 2023, Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks, arXiv preprint
  49. Zhang, 2023, Planning with large language models for code generation, arXiv prnote
  50. Jiang, 2023, Self-planning Code Generation with Large Language Model
  51. Zhang, 2023, Toolcoder: Teach code generation models to use api search tools
  52. Chen, 2023, CODET: Code Generation With Generated Tests
  53. Inala, 2022, Fault-aware neural code rankers
  54. 10.1145/3510003.3510203 · DOI
  55. Poldrack, 2023, AI-assisted coding: Experiments with GPT-4
  56. Yetistiren, 2023, Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
  57. Dinh, 2023, Large language models of code fail at completing code with potential bugs
  58. 10.1109/BotSE59190.2023.00008 · DOI
  59. Yan, 2023, Coco: Testing code generation systems via concretized instructions
  60. 10.1145/3545945.3569785 · DOI · Paper page
  61. Noever, 2023, Chatbots as fluent polyglots: Revisiting breakthrough code snippets
  62. Mohammadkhani, 2022, Ex-plainable ai for pre-trained code models: What do they learn? when they do not work?, arXiv preprint
  63. Hajipour, 2023, Systematically Finding Security Vulnerabilities in Black-Box Code Generation Models
  64. 10.1109/SMC53992.2023.10394237 · DOI · Paper page
  65. 10.1145/3611643.3617845 · DOI
  66. 10.1145/3587102.3588792 · DOI · Paper page
  67. Liu, 2023, A comprehensive evalu-ation of ChatGPTs zero-shot Text-to-SQL capability
  68. 10.1145/3597926.3598060 · DOI
  69. 10.1145/3510457.3513061 · DOI
  70. Turing, Checking a large routine, 67
  71. 10.1109/tse.2024.3368208/mm1 · DOI
  72. 10.1109/ICSE48619.2023.00178 · DOI
  73. Yuan, 2023, No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
  74. BareiB, 2022, Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code, arXiv prnote
  75. 10.1145/1297846.1297902 · DOI
  76. Hashtroudi, 2023, Automated test case generation using code models and domain adaptation, arXiv preprint
  77. Siddiq, 2023, Exploring the Effectiveness of Large Language Models in Generating Unit Tests
  78. Hu, 2023, Augmenting greybox fuzzing with generative ai
  79. Xia, 2023, Universal fuzzing via large language models, arXiv preprint
  80. 10.1145/3597926.3598067 · DOI · Paper page
  81. Li, 2023, Finding Failure-Inducing Test Cases with ChatGPT
  82. Sun, 2023, Automatic code summarization via ChatGPT: How far are we
  83. 10.1109/icse48619.2023.00194 · DOI · Paper page
  84. 10.1145/3597503.3608137 · DOI
  85. Yuan, 2023, No more manual tests? evaluating and improving chatgpt for unit test generation
  86. Schafer, 2023, Adaptive Test Generation Using a Large Language Model
  87. Xie, 2023, ChatUniTest: a ChatGPT-based automated unit test generation tool
  88. 10.1145/3377811.3380420 · DOI
  89. 10.1145/3510003.3510206 · DOI
  90. Wang, 2023, Self-consistency improves chain of thought rea-soning in language models
  91. 10.1145/120807.120814 · DOI
  92. 10.1109/FOSE.2007.25 · DOI
  93. 10.1109/TSE.2010.62 · DOI
  94. 10.1016/bs.adcom.2018.03.015 · DOI
  95. 10.1109/ICSE.2017.61 · DOI
  96. Khanfir, 2023, Efficient mutation testing via pre-trained language models
  97. Garg, 2023, Vulnerability Mimicking Mutants
  98. 10.1007/978-3-031-48796-5_13 · DOI
  99. Pan, 2023, LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models
  100. 10.18653/v1/2023.findings-acl.308 · DOI
  101. 10.1109/SCAM.2018.00009 · DOI
  102. 10.1109/AST58925.2023.00018 · DOI
  103. 10.1109/tse.2022.3201209 · DOI
  104. Fatima, 2023, Black-box prediction of flaky test fix categories using language models
  105. 10.1109/ASE.2008.32 · DOI
  106. 10.1002/stvr.435 · DOI
  107. 10.1002/stvr.1696 · DOI
  108. 10.1145/2568225.2568314 · DOI
  109. 10.1109/ASE56229.2023.00148 · DOI · Paper page
  110. Kang, 2023, A preliminary evaluation of llm-based fault localization
  111. 10.1145/3597503.3608137 · DOI
  112. 10.1609/aaai.v37i4.25642 · DOI · Paper page
  113. Wu, 2023, Condefects: A new dataset to address the data leakage concern for llm-based fault localization and program repair
  114. 10.1109/ISSRE59848.2023.00055 · DOI
  115. 10.1109/tse.2011.104 · DOI
  116. Huang, 2023, A sur-vey on automated program repair techniques
  117. 10.1109/ICSE-SEIP.2019.00039 · DOI
  118. 10.1145/3524459.3527353 · DOI
  119. 10.1145/3105906 · DOI
  120. 10.1016/S0950-5849(01)00189-6 · DOI
  121. 10.1145/3540250.3549101 · DOI · Paper page
  122. 10.1109/ICSE48619.2023.00129 · DOI · Paper page
  123. Xia, 2023, Conversational Automated Program Repair
  124. Xia, 2023, Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT
  125. 10.1109/sp46215.2023.10179420 · DOI
  126. Charalambous, 2023, A new era in software security: Towards self-healing software via large language models and formal verification
  127. Cao, 2023, A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair
  128. Fakhoury, 2023, Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions
  129. 10.1109/ICSE48619.2023.00149 · DOI · Paper page
  130. 10.1109/ICSE48619.2023.00125 · DOI · Paper page
  131. 10.1145/3611643.3613892 · DOI · Paper page
  132. Berabi, 2021, TFix: Learning to fix coding errors with a text-to-text transformer, 139, 780
  133. Xia, 2023, Revisiting the plastic surgery hypothesis via large language models
  134. Kang, 2023, Explainable automated debugging via large language model-driven scientific debugging
  135. 10.1007/978-3-031-48796-5_12 · DOI
  136. Lovelace, Sketch of the analytical engine invented by Charles Babbage by L. F. Menabrea of Turin, officer of the military engineers, with notes by the translator, 1843, translation with notes on article in italian in Bibliotheque Universelle de Geneve
  137. Aho, 1986, Compilers: Principles, techniques and tools
  138. 10.1007/BF00263742 · DOI
  139. 10.1007/978-3-642-46490-4_27 · DOI
  140. Koza, 1992, Genetic Programming: On the Programming of Computers by Means of Natural Selection
  141. 10.1145/1629575.1629585 · DOI
  142. 10.1145/2593929.2600116 · DOI
  143. 10.1109/tevc.2013.2281544 · DOI
  144. 10.1145/3540250.3549096 · DOI · Paper page
  145. 10.1109/GI59320.2023.00013 · DOI
  146. Garg, 2023, Rapgen: An approach for fixing code inefficiencies in zero-shot
  147. Chen, 2023, Supersonic: Learning to generate source code optimisations in c/c++
  148. Cummins, 2023, Large language models for compiler optimization
  149. 10.1145/130844.130856 · DOI
  150. 10.1109/ICSE-Companion58688.2023.00013 · DOI
  151. 10.1016/j.jss.2023.111796 · DOI
  152. 10.18653/v1/2023.findings-acl.678 · DOI · Paper page
  153. 10.1016/j.infsof.2016.11.009 · DOI
  154. Halstead, 1977, Elements of Software Science
  155. 10.1109/TSE.1976.233837 · DOI · Paper page
  156. 10.1109/TSE.2018.2827066 · DOI
  157. 10.1145/2568225.2568321 · DOI
  158. 10.1145/1389095.1389437 · DOI
  159. 10.1145/2739480.2754648 · DOI
  160. 10.1145/2351676.2351678 · DOI
  161. Gamma, 1995, Design Patterns
  162. 10.1109/TSE.2021.3067156 · DOI
  163. Sun, 2023, Automatic Code Summarization via ChatGPT: How Far Are We
  164. Geng, 2023, An Empirical Study on Using Large Language Models for Multi-Intent Comment Generation
  165. 10.1145/3597503.3608134 · DOI
  166. 10.1109/MS.2013.86 · DOI
  167. 10.1109/FOSM.2008.4659248 · DOI
  168. 10.1109/TSE.2016.2630689 · DOI · Paper page
  169. 10.1109/MS.2018.290111035 · DOI
  170. 10.1142/9789812775245_0014 · DOI
  171. 10.1007/978-1-4471-2239-5 · DOI
  172. 10.1109/MS.2017.4541027 · DOI
  173. 10.1145/24574.24583 · DOI
  174. 10.1109/MC.2008.541 · DOI
  175. 10.1109/COMPSAC57700.2023.00117 · DOI · Paper page
  176. Pressman, 1992, Software Engineering: A Practitioners Approach
  177. 10.1145/3475716.3484196 · DOI
  178. Cornes, 1995, The coq proof assistant, reference manual, version 5.10
  179. 10.1109/ICSM.1992.242538 · DOI
  180. 10.1109/ICSM.1989.65225 · DOI
  181. 10.1145/3581641.3584037 · DOI
  182. Tian, 2023, Is ChatGPT the Ultimate Programming Assistant - How far is it?
  183. Meckler, 2022, Teachers are on alert for inevitable cheating after release of ChatGPT, The Washington post
  184. Heaven, 2023, ChatGPT is going to change education, not destroy it, MIT Technology review
  185. 10.5220/0011996900003470 · DOI
  186. 10.1145/3501385.3543957 · DOI · Paper page
  187. 10.1145/3597503.3608140 · DOI
  188. Rae, 2022, Scaling language models: Methods, analysis & insights from training gopher
  189. Henderson, 2020, Towards the systematic reporting of the energy and carbon footprints of machine learning, J. Mach. Learn. Res., 21
  190. Hoffmann, 2022, Training compute-optimal large language models
  191. Polino, 2018, Model compression via distillation and quantization
  192. Manes, 2018, The art, science, and engineering of fuzzing: A survey, CoRR, abs/1812.00140 · arXiv
  193. 10.1109/ESEM56168.2023.10304803 · DOI
  194. Tang, 2023, ChatGPT vs SBST: A comparative assessment of unit test suite generation
  195. 10.18653/v1/2022.emnlp-main.823 · DOI · Paper page

Cited by

Sorted by citations

See all 650

Metadata

Authors
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M.
Affiliations
  • Meta (United States) (US)
  • Meta (United Kingdom) (GB)
  • Korea Advanced Institute of Science and Technology (KR)
  • King's College London (GB)
ORCID
Venue
2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)
Publisher
IEEE
Published
2023-05-14
Pages
31-53
DOI
10.1109/icse-fose59343.2023.00008
OpenAlex
W4392414327
Cited by
650
References
45
Citations by year
2026: 227 · 2025: 266 · 2024: 153 · 2023: 4
FWCI
150.01
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
large language models, software engineering, code generation, program repair, refactoring, requirements engineering, software documentation, performance improvement, hallucination mitigation, hybrid technique, emergent properties

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review