CoChat
Sign Up
Cited by 124Software Engineering ResearchTopic ModelingSoftware Engineering Techniques and Practices

A survey on large language models for software engineering

Open access
Zhang, Quanjun, Fang, Chunrong, Xie, Yang, Zhang, Yaxin, Yu, Shengcheng, Sun, Weisong, Yang, Yun, Chen, Zhenyu·2026·Science China Information Sciences
DOI 10.1007/s11432-025-4670-0
Open this paper in CoChat — free accountOpen PDF

Abstract

Software engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing large language models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information on the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 926 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing, and distillation. Finally, we highlight several challenges and potential opportunities in applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository https://github.com/iSEngLab/AwesomeLLM4SE .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11432-025-4670-0

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GenAI is increasingly acting as a community contributor in OSS, submitting issues, pull requests, code reviews, and security reports.

    “It is increasingly acting as a community contributor-submitting issues, pull requests, code reviews, and security reports [14, 24, 73].”

    From To Ban or Not to Ban? How Open Source Projects Govern GenAI Contributions (2026) · cites this paper as [73]

    The section on integrating LLMs into open-source ecosystems (page unavailable) describes developers and AI agents collaborating on issue triage and resolution, pull request submission, and code review. It does not say they submit security reports, and issue triage or resolution is not the same as submitting issues. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 524

  1. Biolchini J, Mian P G, Natali A C C, et al. Systematic review in software engineering. System engineering and computer science department COPPE/UFRJ, Technical Report ES, 2005, 679: 45
  2. Zelkowitz M V. Perspectives in software engineering. ACM Comput Surv, 1978, 10: 197-216 · DOI
  3. Yang Y, Xia X, Lo D, et al. A survey on deep learning for software engineering. ACM Comput Surv, 2022, 54: 1-73 · DOI · Paper page
  4. Wang J, Huang Y, Chen C, et al. Software testing with large language model: survey, landscape, and vision. ArXiv:230707221
  5. Devlin J, Chang M W, Lee K, et al. Bert: pre-training of deep bidirectional transformers for language understanding. ArXiv:181004805
  6. Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019, 1: 9
  7. Zhang Y, Wu Y, Chen T, et al. How do developers talk about github actions? Evidence from online software development community. In: Proceedings of the 46th International Conference on Software Engineering (ICSE), 2024. 492-504
  8. Dong J, Sun J, Zhang W, et al. ConTested: consistency-aided tested code generation with LLM. Proc ACM Softw Eng, 2025, 2: 596-617 · DOI
  9. Fu Y, Li B, Li L, et al. The first prompt counts the most! An evaluation of large language models on iterative example-based code generation. Proc ACM Softw Eng, 2025, 2: 1583-1606 · DOI
  10. Yu H, Shen B, Zhang J, et al. Wenwang: toward effectively generating code beyond standalone functions via generative pre-trained models. ACM Trans Softw Eng Methodol, 2025, 34: 187
  11. Zhang Q, Fang C, Xie Y, et al. A systematic literature review on large language models for automated program repair. ArXiv:240501466
  12. Xia C S, Wei Y, Zhang L. Automated program repair in the era of large pre-trained language models. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 1482-1494
  13. Hu H, He C, Zhang H, et al. Aprmcts: improving LLM-based automated program repair with iterative tree search. ArXiv:250701827
  14. Shang Y, Zhang Q, Fang C, et al. A large-scale empirical study on fine-tuning large language models for unit testing. Proc ACM Softw Eng, 2025, 2: 1678-1700
  15. Zhang Q, Sun W, Fang C, et al. Exploring automated assertion generation via large language models. ACM Trans Softw Eng Methodol, 2025. 34: 1-25
  16. Zhang Q, Fang C, Zheng Y, et al. Improving retrieval-augmented deep assertion generation via joint training. IEEE Trans Softw Eng, 2025, 51: 1232-1247 · DOI
  17. Xia C S, Zhang L. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022. 959-971
  18. Zhang Q, Fang C, Yu B, et al. Pre-trained model-based automated software vulnerability repair: How far are we? IEEE Trans Depend Secure Comput, 2023, 21: 2507-2525
  19. Fried D, Aghajanyan A, Lin J, et al. Incoder: a generative model for code infilling and synthesis. ArXiv:220405999
  20. Wu Y, Jiang N, Pham H V, et al. How effective are neural networks for fixing security vulnerabilities. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023. 1282-1294
  21. Yuan W, Zhang Q, He T, et al. Circle: continual repair across programming languages. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022. 678-690
  22. Hu H, Xie X, Zhang Q. Repair-r1: better test before repair. ArXiv:250722853
  23. Nashid N, Sintaha M, Mesbah A. Retrieval-based prompt selection for code-related few-shot learning. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2450-2462
  24. Zhang Q J, et al. Sci China Inf Sci April 2026, Vol. 69, Iss. 4, 141102:42
  25. Wang S, Huang L, Gao A, et al. Machine/deep learning for software engineering: a systematic literature review. IEEE Trans Software Eng, 2022, 49: 1188-1231
  26. Zhang H Z, Zhang K C, Li Z, et al. Deep learning for code generation: a survey. Sci China Inf Sci, 2024, 67: 191101 · DOI
  27. Niu C, Li C, Luo B, et al. Deep learning meets software engineering: a survey on pre-trained models of source code. In: Proceedings of the 31st International Joint Conference on Artificial Intelligence, 2022. 5546-5555
  28. Zan D, Chen B, Zhang F, et al. Large language models meet nl2code: a survey. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. 7443-7464
  29. Chen X P, Hu X, Huang Y, et al. Deep learning-based software engineering: progress, challenges, and opportunities. Sci China Inf Sci, 2025, 68: 111102 · DOI
  30. Fan A, Gokkaya B, Harman M, et al. Large language models for software engineering: survey and open problems. ArXiv:231003533
  31. Hou X, Zhao Y, Liu Y, et al. Large language models for software engineering: a systematic literature review. ArXiv:230810620
  32. Zheng Z, Ning K, Zhong Q, et al. Towards an understanding of large language models in software engineering tasks. Empir Softw Eng, 2025, 30: 50 · DOI
  33. Zheng Z, Ning K, Wang Y, et al. A survey of large language models for code: evolution, benchmarking, and future trends. ArXiv:231110372
  34. Ran D, Wu M, Yang W, et al. Foundation model engineering: engineering foundation models just as engineering software. ACM Trans Softw Eng Methodol, 2025, 34: 1-18
  35. Liu J, Wang K, Chen Y, et al. Large language model-based agents for software engineering: a survey. ArXiv:240902977
  36. Zhang Q, Fang C, Gu S, et al. Large language models for unit testing: a systematic literature review. ArXiv:250615227
  37. Zhou X, Cao S, Sun X, et al. Large language model for vulnerability detection and repair: literature review and the road ahead. ACM Trans Softw Eng Methodol, 2025, 34: 145
  38. Sheng Z, Chen Z, Gu S, et al. LLMs in software security: a survey of vulnerability detection techniques and insights. ArXiv:2502.07049 · arXiv
  39. Jiang J, Wang F, Shen J, et al. A survey on large language models for code generation. ArXiv:240600515
  40. Gao C, Hu X, Gao S, et al. The current challenges of software engineering in the era of large language models. ACM Trans Softw Eng Methodol, 2025, 34: 127
  41. He J, Treude C, Lo D. LLM-based multi-agent systems for software engineering: literature review, vision and the road ahead. ACM Trans Softw Eng Methodol, 2024, 34: 124
  42. Zhang H, Babar M A, Tell P. Identifying relevant studies in software engineering. Inf Softw Tech, 2011, 53: 625-637 · DOI
  43. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Proceedings of Advances in Neural Information Processing Systems, 2017
  44. Wang Z, Yan M, Chen J, et al. Deep learning library testing via effective model generation. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020. 788-799
  45. Zhang Q, Fang C, Ma Y, et al. A survey of learning-based automated program repair. ACM Trans Softw Eng Methodol, 2023. 33: 1-69
  46. Kanade A, Maniatis P, Balakrishnan G, et al. Learning and evaluating contextual embedding of source code. In: Proceedings of International Conference on Machine Learning, 2020. 5110-5121
  47. Feng Z, Guo D, Tang D, et al. Codebert: a pre-trained model for programming and natural languages. In: Proceedings of Findings of the Association for Computational Linguistics, 2020. 1536-1547
  48. Guo D, Ren S, Lu S, et al. Graphcodebert: pre-training code representations with data flow. ArXiv:200908366
  49. Mukherjee M, Hellendoorn V J. Stack over-flowing with results: the case for domain-specific pre-training over one-size-fits-all models. ArXiv:230603268
  50. Zhang D, Ahmad W, Tan M, et al. Code representation learning at scale. ArXiv:240201935
  51. Lin J, Dong H, Xie Y, et al. Scaling laws behind code understanding model. ArXiv:240212813
  52. Clement C B, Drain D, Timcheck J, et al. Pymt5: multi-mode translation of natural language and Python code with transformers. ArXiv:201003150
  53. Mastropaolo A, Scalabrino S, Cooper N, et al. Studying the usage of text-to-text transfer transformer to support code-related tasks. In: Proceedings of the 43rd International Conference on Software Engineering (ICSE), 2021. 336-347
  54. Ahmad W U, Chakraborty S, Ray B, et al. Unified pre-training for program understanding and generation. ArXiv:210306333
  55. Guo D, Lu S, Duan N, et al. Unixcoder: unified cross-modal pre-training for code representation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022. 7212-7225
  56. Niu C, Li C, Ng V, et al. Spt-code: sequence-to-sequence pre-training for learning source code representations. In: Proceedings of the 44th International Conference on Software Engineering, 2022. 2006-2018
  57. Le H, Wang Y, Gotmare A D, et al. Coderl: mastering code generation through pretrained models and deep reinforcement learning. In: Proceedings of Advances in Neural Information Processing Systems, 2022. 21314-21328
  58. Zhang J, Panthaplackel S, Nie P, et al. Coditt5: pretraining for source code and natural language editing. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022. 1-12
  59. Wang Y, Le H, Gotmare A D, et al. Codet5+: open code large language models for code understanding and generation. ArXiv:230507922
  60. Chandel S, Clement C B, Serrato G, et al. Training and evaluating a jupyter notebook data science assistant. ArXiv:220112901
  61. Chai Y, Wang S, Pang C, et al. Ernie-code: beyond English-centric cross-lingual pretraining for programming languages. ArXiv:221206742
  62. Shojaee P, Jain A, Tipirneni S, et al. Execution-based code generation using deep reinforcement learning. ArXiv:230113816
  63. Liu J, Zhu Y, Xiao K, et al. Rltf: reinforcement learning from unit test feedback. ArXiv:230704349
  64. Lin B, Wang S, Liu Z, et al. Cct5: a code-change-oriented pre-trained model. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. 1509-1521
  65. Yu Z, Tao Y, Chen L, et al. B-coder: value-based deep reinforcement learning for program synthesis. ArXiv:2310.03173 · arXiv
  66. Gong L, Elhoushi M, and Cheung A. Ast-t5: structure-aware pretraining for code generation and understanding. ArXiv:240103003
  67. Zhu Q, Liang Q, Sun Z, et al. Grammart5: grammar-integrated pretrained encoder-decoder neural model for code. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024. 1-13
  68. Lu S, Guo D, Ren S, et al. Codexglue: a machine learning benchmark dataset for code understanding and generation. In: Proceedings of the 35th Conference on Neural Information Processing Systems, 2021. 1-14
  69. Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. ArXiv:210703374
  70. Xu F F, Alon U, Neubig G, et al. A systematic evaluation of large language models of code. In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022. 1-10
  71. Nijkamp E, Pang B, Hayashi H, et al. Codegen: an open large language model for code with multi-turn program synthesis. ArXiv:220313474
  72. Zan D, Chen B, Yang D, et al. Cert: continual pre-training on sketches for library-oriented code generation. ArXiv:220606888
  73. Allal L B, Li R, Kocetkov D, et al. Santacoder: don't reach for the stars! ArXiv:230103988
  74. Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you! ArXiv:230506161
  75. Christopoulou F, Lampouras G, Gritta M, et al. Pangu-coder: program synthesis with function-level language modeling. ArXiv:220711280
  76. Zhang Q J, et al. Sci China Inf Sci April 2026, Vol. 69, Iss. 4, 141102:43
  77. Shen B, Zhang J, Chen T, et al. Pangu-coder2: boosting large language models for code with ranking feedback. ArXiv:230714936
  78. Chowdhery A, Narang S, Devlin J, et al. Palm: scaling language modeling with pathways. ArXiv:220402311
  79. Zheng Q, Xia X, Zou X, et al. Codegeex: a pre-trained model for code generation with multilingual evaluations on Humaneval-X. ArXiv:230317568
  80. Nijkamp E, Hayashi H, Xiong C, et al. Codegen2: lessons for training LLMs on programming and natural languages. ArXiv:230502309
  81. Touvron H, Lavril T, Izacard G, et al. Llama: open and efficient foundation language models. ArXiv:230213971
  82. Workshop B, Scao T L, Fan A, et al. Bloom: a 176b-parameter open-access multilingual language model. ArXiv:221105100
  83. Di P, Li J, Yu H, et al. Codefuse-13b: a pretrained multi-lingual code large language model. In: Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, 2024. 418-429
  84. Xie R, Zeng Z, Yu Z, et al. Codeshell technical report. ArXiv:240315747
  85. Xu Y, Su H, Xing C, et al. Lemur: harmonizing natural language and code for language agents. ArXiv:231006830
  86. Wei Y, Wang Z, Liu J, et al. Magicoder: empowering code generation with OSS-instruct. In: Proceedings of the 41st International Conference on Machine Learning, 2024. 52632-52657
  87. Muennighoff N, Liu Q, Zebaze A, et al. Octopack: instruction tuning code large language models. ArXiv:230807124
  88. Luo Z, Xu C, Zhao P, et al. Wizardcoder: empowering code large language models with evol-instruct. ArXiv:230608568
  89. Song Z, Wang Y, Zhang W, et al. Alchemistcoder: harmonizing and eliciting code capability by hindsight tuning on multi-source data. ArXiv:240519265
  90. Lei B, Li Y, Chen Q. Autocoder: enhancing code large language model with AIEV-Instruct. ArXiv:240514906
  91. Team C. Codegemma: open code models based on Gemma. ArXiv:240611409
  92. Bai J, Bai S, Chu Y, et al. Qwen technical report. ArXiv:230916609
  93. Guo D, Zhu Q, Yang D, et al. Deepseek-coder: when the large language model meets programming-the rise of code intelligence. ArXiv:240114196
  94. Zhu Q, Guo D, Shao Z, et al. Deepseek-coder-v2: breaking the barrier of closed-source models in code intelligence. ArXiv:240611931
  95. Wang Y, He K, Dong G, et al. Dolphcoder: echo-locating code large language models with diverse and multi-objective instruction tuning. ArXiv:240209136
  96. Mishra M, Stallone M, Zhang G, et al. Granite code models: a family of open foundation models for code intelligence. ArXiv:240504324
  97. Ding Y, Liu J, Wei Y, et al. Xft: unlocking the power of code instruction tuning by simply merging upcycled mixture-of-experts. ArXiv:240415247
  98. Wu Y, Huang D, Shi W, et al. Inversecoder: unleashing the power of instruction-tuned code LLMs with inverse-instruct. ArXiv:240705700
  99. Rathinasamy K, Kumar A, Gayari G, et al. Narrow transformer: starcoder-based Java-LM for desktop. ArXiv:240703941
  100. Lozhkov A, Li R, Allal L B, et al. Starcoder 2 and the stack v2: the next generation. ArXiv:240219173
  101. Dou S, Liu Y, Jia H, et al. Stepcoder: improving code generation with reinforcement learning from compiler feedback. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 4571-4585
  102. Sun T, Chai L, Yang J, et al. Unicoder: scaling code large language model via universal code. ArXiv:240616441
  103. Yu Z, Zhang X, Shang N, et al. Wavecoder: widespread and versatile enhancement for code large language models by instruction tuning. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 5140-5153
  104. Zhao W X, Zhou K, Li J, et al. A survey of large language models. ArXiv:230318223
  105. Yin S, Fu C, Zhao S, et al. A survey on multimodal large language models. ArXiv:230613549
  106. Roziere B, Gehring J, Gloeckle F, et al. Code llama: open foundation models for code. ArXiv:230812950
  107. Clark K, Luong M T, Le Q V, et al. Electra: pre-training text encoders as discriminators rather than generators. ArXiv:200310555
  108. Nong Y, Sharma R, Hamou-Lhadj A, et al. Open science in software engineering: a study on deep learning-based vulnerability detection. IEEE Trans Softw Eng, 2022, 49: 1983-2005 · DOI
  109. Moharil A, Sharma A. TABASCO: a transformer based contextualization toolkit. Sci Comput Programm, 2023, 230: 102994 · DOI
  110. Moharil A, Sharma A. Identification of intra-domain ambiguity using transformer-based machine learning. In: Proceedings of the 1st International Workshop on Natural Language-based Software Engineering, 2022. 51-58
  111. Ezzini S, Abualhaija S, Arora C, et al. Automated handling of anaphoric ambiguity in requirements: a multi-solution study. In: Proceedings of the 44th International Conference on Software Engineering, 2022. 187-199
  112. Sridhara G, Ranjani H G, Mazumdar S. ChatGPT: a study on its utility for ubiquitous software engineering tasks. ArXiv:230516837
  113. Kolthoff K, Bartelt C, Ponzetto S P. Data-driven prototyping via natural-language-based GUI retrieval. Autom Softw Eng, 2023, 30: 13 · DOI
  114. Brie P, Burny N, Sluÿters A, et al. Evaluating a large language model on searching for GUI layouts. Proc ACM Hum-Comput Interact, 2023, 7: 1-37 · DOI
  115. Hey T, Keim J, Koziolek A, et al. Norbert: transfer learning for requirements classification. In: Proceedings of the 28th International Requirements Engineering Conference (RE), 2020. 169-179
  116. Khan M A, Khan M S, Khan I, et al. Non functional requirements identification and classification using transfer learning model. IEEE Access, 2023, 11: 74997-75005
  117. Rahman K, Ghani A, Alzahrani A, et al. Pre-trained model-based NFR classification: overcoming limited data challenges. IEEE Access, 2023, 11: 81787-81802
  118. Han L, Zhou Q, Li T. Improving requirements classification models based on explainable requirements concerns. In: Proceedings of the 31st International Requirements Engineering Conference Workshops (REW), 2023. 95-101
  119. Lubos S, Felfernig A, Tran T N T, et al. Leveraging LLMs for the quality assurance of software requirements. In: Proceedings of the 32nd International Requirements Engineering Conference (RE), 2024. 389-397
  120. Preda A R, Mayr-Dorn C, Mashkoor A, et al. Supporting high-level to low-level requirements coverage reviewing with large language models. In: Proceedings of the 21st International Conference on Mining Software Repositories, 2024. 242-253
  121. Poudel A, Lin J, Cleland-Huang J. Leveraging transformer-based language models to automate requirements satisfaction assessment. ArXiv:231204463
  122. Ronanki K, Cabrero-Daniel B, Berger C. ChatGPT as a tool for user story quality evaluation: trustworthy out of the box? In: Proceedings of International Conference on Agile Software Development, 2022. 173-181
  123. Ferrari A, Abualhaija S, Arora C. Model generation from requirements with LLMs: an exploratory study. ArXiv:240406371
  124. Wang B, Wang C, Liang P, et al. How LLMs aid in UML modeling: an exploratory study with novice analysts. ArXiv:240417739
  125. Tinnes C, Welter A, Apel S. Leveraging large language models for software model completion: results from industrial and public datasets. ArXiv:240617651
  126. Xie D, Yoo B, Jiang N, et al. Impact of large language models on generating software specifications. ArXiv:230603324
  127. Ma L, Liu S, Li Y, et al. Specgen: automated generation of formal program specifications via large language models. ArXiv:240108807
  128. Mandal S, Chethan A, Janfaza V, et al. Large language models based automatic synthesis of software specifications. ArXiv:230409181
  129. Hasan M R, Li J, Ahmed I, et al. Automated repair of declarative software specifications in the era of large language models. ArXiv:231012425
  130. Li Z, Dutta S, Naik M. LLM-assisted static analysis for detecting security vulnerabilities. ArXiv:240517238
  131. Luitel D, Hassani S, Sabetzadeh M. Improving requirements completeness: automated assistance through large language models. Requir Eng, 2024, 29: 73-95
  132. Ren S, Nakagawa H, Tsuchiya T. Combining prompts with examples to enhance LLM-based requirement elicitation. In: Proceedings of Zhang Q J, et al. Sci China Inf Sci April 2026, Vol. 69, Iss. 4, 141102:44
  133. the 48th Annual Computers, Software, and Applications Conference (COMPSAC), 2024. 1376-1381
  134. Sami M A, Rasheed Z, Waseem M, et al. Prioritizing software requirements using large language models. ArXiv:240501564
  135. Jain C, Anish P R, Singh A, et al. A transformer-based approach for abstractive summarization of requirements from obligations in software engineering contracts. In: Proceedings of the 31st International Requirements Engineering Conference (RE), 2023. 169-179
  136. Guo J L, Steghöfer J P, Vogelsang A, et al. Natural language processing for requirements traceability. ArXiv:240510845
  137. Lin J, Liu Y, Zeng Q, et al. Traceability transformed: generating more accurate links with pre-trained BERT models. In: Proceedings of the 43rd International Conference on Software Engineering (ICSE), 2021. 324-335
  138. Zhang S, Wang J, Dong G, et al. Experimenting a new programming practice with LLMs. ArXiv:240101062
  139. Wei M, Harzevili N S, Huang Y, et al. Clear: contrastive learning for API recommendation. In: Proceedings of the 44th International Conference on Software Engineering, 2022. 376-387
  140. Li Z, Li C, Tang Z, et al. PTM-apirec: leveraging pre-trained models of source code in api recommendation. ACM Trans Softw Eng Methodol, 2024, 33: 1-30
  141. Huang Q, Wan Z, Xing Z, et al. Let's chat to find the APIs: connecting human, LLM and knowledge graph through AI chain. In: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023. 471-483
  142. Chen Y, Gao C, Zhu M, et al. Apigen: generative API method recommendation. ArXiv:240115843
  143. Mastropaolo A, Aghajani E, Pascarella L, et al. An empirical study on code comment completion. In: Proceedings of IEEE International Conference on Software Maintenance and Evolution (ICSME), 2021. 159-170
  144. Nie P, Banerjee R, Li J J, et al. Learning deep semantics for test completion. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2111-2123
  145. Li Z, Wang C, Liu Z, et al. Cctest: testing and repairing code completion systems. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 1238-1250
  146. Liu W, Yu A, Zan D, et al. Graphcoder: enhancing repository-level code completion via code context graph-based retrieval and language model. ArXiv:240607003
  147. Cheng W, Wu Y, Hu W. Dataflow-guided retrieval augmentation for repository-level code completion. ArXiv:240519782
  148. Wu D, Ahmad W U, Zhang D, et al. Repoformer: selective retrieval for repository-level code completion. ArXiv:240310059
  149. Phan H N, Phan H N, Nguyen T N, et al. Repohyper: better context retrieval is all you need for repository-level code completion. ArXiv:240306095
  150. Liu J, Chen Y, Liu M, et al. Stall+: boosting LLM-based repository-level code completion with static analysis. ArXiv:240610018
  151. Wang Y, Wang Y, Guo D, et al. Rlcoder: reinforcement learning for repository-level code completion. ArXiv:240719487
  152. Ciniselli M, Cooper N, Pascarella L, et al. An empirical study on the usage of BERT models for code completion. In: Proceedings of the 18th International Conference on Mining Software Repositories (MSR), 2021. 108-119
  153. Ciniselli M, Cooper N, Pascarella L, et al. An empirical study on the usage of transformer models for code completion. IEEE Trans Softw Eng, 2021, 48: 4818-4837
  154. van Dam T, Izadi M, van Deursen A. Enriching source code with contextual data for code completion models: an empirical study. ArXiv:230412269
  155. Liu C, Cai Y, Lin Y, et al. Coedpilot: recommending code edits with learned prior edit relevance, project-wise awareness, and interactive nature. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024. 466-478
  156. Li J, Li G, Li Z, et al. Codeeditor: learning to edit source code with pre-trained models. ACM Trans Softw Eng Methodol, 2023, 32: 1-22
  157. Gupta P, Khare A, Bajpai Y, et al. Grace: generation using associated code edits. ArXiv:230514129
  158. Liu C, Cetin P, Patodia Y, et al. Automated code editing with search-generate-modify. In: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024. 398-399
  159. Yang G, Zhou Y, Chen X, et al. A syntax-guided multi-task learning approach for Turducken-style code generation. Empir Softw Eng, 2023, 28: 141 · DOI
  160. Jiang X, Dong Y, Wang L, et al. Self-planning code generation with large language models. ACM Trans Softw Eng Methodol, 2024, 33: 1-30
  161. Zhang S, Chen Z, Shen Y, et al. Planning with large language models for code generation. ArXiv:230305510
  162. Zhang Q, Fang C, Shang Y, et al. No man is an island: towards fully automatic programming by code search, code generation and program repair. ArXiv:240903267
  163. Han H, Kim J, Yoo J, et al. Archcode: incorporating software requirements in code generation with large language models. ArXiv:240800994
  164. Mu F, Shi L, Wang S, et al. ClarifyGPT: empowering LLM-based code generation with intention clarification. ArXiv:231010996
  165. Li J, Zhao Y, Li Y, et al. Acecoder: an effective prompting technique specialized in code generation. ACM Trans Softw Eng Methodol, 2024, 33: 204
  166. Ni A, Iyer S, Radev D, et al. Lever: learning to verify language-to-code generation with execution. In: Proceedings of International Conference on Machine Learning, 2023. 26106-26128
  167. Chen X, Lin M, Schärli N, et al. Teaching large language models to self-debug. ArXiv:230405128
  168. Chen A, Scheurer J, Korbak T, et al. Improving code generation by training with natural language feedback. ArXiv:230316749
  169. Zhang K, Li Z, Li J, et al. Self-edit: fault-aware code editor for code generation. ArXiv:230504087
  170. Mastropaolo A, Pascarella L, Guglielmi E, et al. On the robustness of code generation techniques: an empirical study on github copilot. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2149-2160
  171. Zhong L, Wang Z. Can LLM replace stack overflow? A study on robustness and reliability of large language model code generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2024. 21841-21849
  172. Coignion T, Quinton C, Rouvoy R. A performance study of LLM-generated code on LeetCode. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024. 79-89
  173. Jin K, Wang C Y, Pham H V, et al. Can ChatGPT support developers? An empirical evaluation of large language models for code generation. In: Proceedings of the 21st International Conference on Mining Software Repositories (MSR), 2024. 167-171
  174. Feng Y, Vanam S, Cherukupally M, et al. Investigating code generation performance of ChatGPT with crowdsourcing social data. In: Proceedings of the 47th Annual Computers, Software, and Applications Conference (COMPSAC), 2023. 876-885
  175. Kou B, Chen S, Wang Z, et al. Do large language models pay similar attention like human programmers when generating code? Proc ACM Softw Eng, 2024, 1: 2261-2284
  176. Liu C, Bao X, Zhang H, et al. Guiding ChatGPT for better code generation: an empirical study. In: Proceedings of IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2024. 102-113
  177. Yan D, Gao Z, Liu Z. A closer look at different difficulty levels code generation abilities of ChatGPT. In: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023. 1887-1898
  178. Liu C, Bao X, Zhang H, et al. Improving ChatGPT prompt for code generation. ArXiv:230508360
  179. Yu H, Shen B, Ran D, et al. Codereval: a benchmark of pragmatic code generation with generative pre-trained models. In: Proceedings
  180. Zhang Q J, et al. Sci China Inf Sci April 2026, Vol. 69, Iss. 4, 141102:45 of the 46th IEEE/ACM International Conference on Software Engineering, 2024. 1-12
  181. Chen M, Zhang H, Wan C, et al. On the effectiveness of large language models in domain-specific code generation. ArXiv:231201639
  182. Ma Z, An S, Xie B, et al. Compositional API recommendation for library-oriented code generation. In: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024. 87-98
  183. Liu M, Yang T, Lou Y, et al. Codegen4libs: a two-stage approach for library-oriented code generation. In: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023. 434-445
  184. Li J, Li Y, Li G, et al. Skcoder: a sketch-based approach for automatic code generation. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2124-2135
  185. Lin F, Kim D J, Chen T H. When LLM-based code generation meets the software development process. ArXiv:240315852
  186. Huang D, Bu Q, Zhang J M, et al. Agentcoder: multi-agent-based code generation with iterative testing and optimisation. ArXiv:231213010
  187. Zhang K, Li J, Li G, et al. Codeagent: enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. ArXiv:240107339
  188. Liu F, Li G, Zhao Q H, et al. Learning to represent code semantics. Sci China Inf Sci, 2025, 68: 172101 · DOI
  189. Qin Y H, Wang S W, Lin B, et al. GTE: learning code AST representation efficiently and effectively. Sci China Inf Sci, 2025, 68: 139101 · DOI
  190. Agarwal M, Shen Y, Wang B, et al. Structured code representations enable data-efficient adaptation of code language models. ArXiv:240110716
  191. He J, Zhou X, Xu B, et al. Representation learning for stack overflow posts: How far are we? ACM Trans Softw Eng Methodol, 2024, 33: 1-24
  192. Cui L, Yin J, Cui J, et al. Api2vec++: boosting API sequence representation for malware detection and classification. IEEE Trans Softw Eng, 2024, 50: 2142-2162
  193. Lin Y, Wan C, Bai S, et al. Vargan: adversarial learning of variable semantic representations. IEEE Trans Softw Eng, 2024, 50: 1505-1517
  194. Li J, Liu F, Li J, et al. Mcodesearcher: multi-view contrastive learning for code search. In: Proceedings of the 14th Asia-Pacific Symposium on Internetware, 2023. 270-280
  195. Liu S, Wu B, Xie X, et al. Contrabert: enhancing code pre-trained models via contrastive learning. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2476-2487
  196. Shi Z, Xiong Y, Zhang X, et al. Cross-modal contrastive learning for code search. In: Proceedings of IEEE International Conference on Software Maintenance and Evolution (ICSME), 2022. 94-105
  197. Shi Z, Xiong Y, Zhang Y, et al. Improving code search with multi-modal momentum contrastive learning. In: Proceedings of the 31st International Conference on Program Comprehension (ICPC), 2023. 280-291
  198. Li X, Gong Y, Shen Y, et al. Coderetriever: a large scale contrastive pre-training method for code search. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022. 2898-2910
  199. Shi E, Wang Y, Gu W, et al. Cocosoda: effective contrastive learning for code search. In: Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023. 2198-2210
  200. Salza P, Schwizer C, Gu J, et al. On the effectiveness of transfer learning for code search. IEEE Trans Softw Eng, 2023, 49: 1804-1822

200 of 524 shown

Cited by

Sorted by citations

Authors
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z.
Affiliations
  • Nanjing University (CN)
  • Nanyang Technological University (SG)
  • Swinburne University of Technology (AU)
Corresponding author
  • Fang, Chunrong
  • Chen, Zhenyu
ORCID
Venue
Science China Information Sciences
Publisher
Springer Science and Business Media LLC
Published
2026-03-05
Volume
69
Issue
4
ISSN
  • 1869-1919 (ISSN-L)
  • 1674-733X
DOI
10.1007/s11432-025-4670-0
OpenAlex
W7136464128
Cited by
124
References
17
Citations by year
2026: 78 · 2025: 26 · 2024: 20
FWCI
354.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Science China Information Sciences (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (U24A20337, 62372228)
  • National Key Research and Development Program of China
  • Fundamental Research Funds for the Central Universities (14380029)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
large language models, software engineering, code generation, pre-training objectives, model architecture, downstream tasks, empirical evaluation, benchmarking, security and reliability, model compression, knowledge distillation, evaluation dataset, domain-specific LLMs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review