Towards an understanding of large language models in software engineering tasks
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in text generation and reasoning tasks. Derivative products, like ChatGPT, have been extensively deployed and highly sought after. Meanwhile, the evaluation and optimization of LLMs in software engineering tasks, such as code generation, have become a research focus. However, there is still a lack of systematic research on applying and evaluating LLMs in software engineering. Therefore, this paper comprehensively investigate and collate the research and products combining LLMs with software engineering, aiming to answer two questions: (1) What are the current integrations of LLMs with software engineering? (2) Can LLMs effectively handle software engineering tasks? To find the answers, we have collected related literature as extensively as possible from seven mainstream databases and selected 123 timely papers published starting from 2022 for analysis. We have categorized these papers in detail and reviewed the current research status of LLMs from the perspective of seven major software engineering tasks, hoping this will help researchers better grasp the research trends and address the issues when applying LLMs. Meanwhile, we have also organized and presented papers with evaluation content to reveal the performance and effectiveness of LLMs in various software engineering tasks, guiding researchers and developers to optimize.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“According to the scaling law of large language models (Kaplan et al. 2020), as the model parameters and training data increase, the model’s capacity and capabilities also improve.”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 2×
“Currently, as more and more LLMs designed for software engineering tasks are deployed (Huaggingface, 2021; Nijkamp et al., 2023; Zheng et al., 2023; Wang et al., 2021; Li et al., 2022b; Chai et al., 2022; Wang et al., 2023d), many research works focused on the application of LLMs in the software engineering domain (Ah…”
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Self-citationLin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Self-citationLin, T., Wang, Y., Liu, X., & Qiu, X. (2021) · arXiv · doi:10.48550/arxiv.2106.04554 · Cited by 1,420
Self-citationRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 5×
“Yang et al. (2023b) investigates the application of Transformer-based PLMs for the Controllable Text Generation (CTG) task, summarizing typical applications, key methodologies, and evaluation systems of PLMs in CTG. Min et al. (2023) explores three trending paradigms of using pre-trained language models for NLP, They…”
InfluentialWhy Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 2×
“(Zamfirescu-Pereira et al., 2023) discusses the role of non-AI practitioners in “user cue engineering” and designs BotDesigner, a cue-based chatbot design tool; Ross et al.”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · cited 5×
“…et al. 2023), propose their own solutions to existing problems, such as addressing performance issues (Jain et al. 2023), develop tools or datasets, such as code-text datasets (Manh et al. 2023; Liu et al. 2023c), or identify issues related to LLMs (Treude and Hata 2023; Khlaaf et al. 2022).”
InfluentialLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1×
“ThemostadvancedLLMstodate,suchasGPT-4(Liuetal.2023e;Savelkaetal.2023) andLaMDA(Thoppilanetal.2022),havedemonstratedremarkablelanguagecomprehension andgenerationcapabilities,abletoperformwellonavarietyofnaturallanguageprocessing tasks,suchastextsummarization(Tangetal.2023a).”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 1×
“Such as in-context learning (Brown et al. 2020), instruction following (Sanh et al. 2021; Ouyang et al. 2022; Wei et al. 2021), and step-by-step reasoning (Shanahan 2022).”
A Survey on Large Language Models for Recommendation
Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.19860 · Cited by 515
Self-citationA survey on large language models for recommendation
Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01291-2 · Cited by 515
Self-citationAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 5×
“Some works evaluate the code summarization capabilities of different LLMs, such as ChatGPT (Sun et al. 2023), Codex (Sarsa et al. 2022; Leinonen et al. 2023b), GPT-3 (MacNeil et al. 2022c), among others.”
InfluentialCodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
Self-citationCodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.68 · Cited by 459
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2022) · arXiv · doi:10.48550/arxiv.2212.10403 · Cited by 428
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390 · cited 5×
“…et al. 2023), propose their own solutions to existing problems, such as addressing performance issues (Jain et al. 2023), develop tools or datasets, such as code-text datasets (Manh et al. 2023; Liu et al. 2023c), or identify issues related to LLMs (Treude and Hata 2023; Khlaaf et al. 2022).”
InfluentialSelf-citationLost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 349
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · cited 2×
“…code may contain potential vulnerabilities (Thakur et al. 2023a), automatically generated test cases often fail to achieve comprehensive coverage (Lemieux et al. 2023), etc. Fortunately, LLMs have great potential to solve the above problems due to their excellent performance on complex tasks…”
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281
Self-citationChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270 · cited 3×
“, 2023a), answering software testing questions (Jalil et al., 2023), and test case generation (Tang et al.”
InfluentialAutomated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256 · cited 1×
“However, two works evaluating LLMs in automated program repair (Fan et al. 2023b; Xia et al. 2022) present very positive findings.”
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · cited 2×
“…the quality of code generated by LLMs (Zhang et al. 2023a; Wang et al. 2023d; Barke et al. 2023a; Mouselinos et al. 2023; Lahiri et al. 2022; Dong et al. 2023; Jiang et al. 2023a), addressing security and reliability concerns (Poesia et al. 2022; Zhu and Zhang 2023; Ke et al. 2023),…”
ChatGPT is not all you need. A state-of-the-art review of large generative AI models
Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023) · arXiv · doi:10.48550/arxiv.2301.04655 · Cited by 245
Maddigan, P., & Susnjak, T. (2023) · IEEE Access · doi:10.1109/access.2023.3274199 · Cited by 242
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240 · cited 1×
“Other works assess the capabilities of LLMs in code summarization from different perspectives, such as causal reasoning abilities (Liu et al. 2023d), effectiveness with minimal training (Ahmed and Devanbu 2022), and interpretability (Leinonen et al. 2023a).”
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction
Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00194 · Cited by 204
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195
Towards Human-Bot Collaborative Software Architecting with ChatGPT
Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M. S., & Mikkonen, T. (2023) · Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3593434.3593468 · Cited by 168 · cited 2×
“…et al. 2021; Li et al. 2022c; Chai et al. 2022; Wang et al. 2023e), many research works focused on the application of LLMs in the software engineering domain (Ahmad et al. 2023a; Zhang et al. 2023b; Li et al. 2023a; Barke et al. 2023a; Zhuo et al. 2023; Gozalo-Brizuela and Garrido-Merchan 2023).”
Natural Attack for Pre-trained Models of Code
Yang, Z., Shi, J., He, J., & Lo, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510146 · Cited by 166
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160 · cited 2×
“From the table and the above articles, we can reach a preliminary conclusion:”
An extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · cited 2×
“Some commonly known models based on the prefix decoder architecture include U-PaLM (Tay et al. 2022) and GLM-130B (Zeng et al. 2022a).”
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · cited 2×
“Moreover, a considerable portion of the research evaluates the performance of LLMs in education and programming courses (Leinonen et al. 2023a; Hellas et al. 2023; Sarsa et al. 2022; Savelka et al. 2023), with positive feedback.”
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · cited 2×
“Moreover, a considerable portion of the research evaluates the performance of LLMs in education and programming courses (Leinonen et al. 2023a; Hellas et al. 2023; Sarsa et al. 2022; Savelka et al. 2023), with positive feedback.”
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137 · cited 2×
“Watson et al. (2022) provides a systematic literature review of the intersection between SE and deep learning (DL).”
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119 · cited 5×
“…et al. 2023), propose their own solutions to existing problems, such as addressing performance issues (Jain et al. 2023), develop tools or datasets, such as code-text datasets (Manh et al. 2023; Liu et al. 2023c), or identify issues related to LLMs (Treude and Hata 2023; Khlaaf et al. 2022).”
InfluentialInvestigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112
Can OpenAI's codex fix bugs?: an evaluation on QuixBugs
Prenner, J. A., Babii, H., & Robbes, R. (2022) · Proceedings of the Third International Workshop on Automated Program Repair · doi:10.1145/3524459.3527351 · Cited by 99 · cited 1×
“…or affirm the potential of LLMs in those tasks, such as code generation (Vaithilingam et al. 2022; Kande et al. 2023; Thakur et al. 2023b) and error fixing (Prenner et al. 2022). gangz (2023) evaluates the ability of large models to generate test cases on a simple game, reporting positive results.”
Automatic Code Documentation Generation Using GPT-3
Khan, J. Y., & Uddin, G. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559548 · Cited by 96 · cited 1×
“There are also works that utilize LLMs for code summarization to construct documentation, generate test cases, or address limitations in LLM-based code summarization (Khan and Uddin, 2023; Zhang et al., 2023b).”
ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks
Sridhara, G., G., R. H., & Mazumdar, S. (2023) · arXiv · doi:10.48550/arxiv.2305.16837 · Cited by 68 · cited 4×
“A significant portion of the work conducted has empirically analyzed the performance of ChatGPT, one of the most popular LLM models, as a programming assistant (Tian et al., 2023; Sridhara et al., 2023; Li et al., 2023d; Liu et al., 2023a).”
InfluentialEvaluating the Text-to-SQL Capabilities of Large Language Models
Rajkumar, N., Li, R., & Bahdanau, D. (2022) · arXiv · doi:10.48550/arxiv.2204.00498 · Cited by 56
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
Diet code is healthy: simplifying programs for pre-trained models of code
Zhang, Z., Zhang, H., Shen, B., & Gu, X. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549094 · Cited by 41 · cited 4×
“Today, the causal decoder architecture has become the prevailing choice for large language model architectures, giving rise to a wide range of powerful LLMs such as PaLM (Chowdhery et al. 2022), LLaMA (Touvron et al. 2023), OPT (Zhang et al. 2022c), Bloom (Scao et al. 2022).”
InfluentialCodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Software Development
Potluri, V., Pandey, M., Begel, A., Barnett, M., & Reitherman, S. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544812 · Cited by 39 · cited 1×
“There are also detailed evaluations of LLMs’ performance in specific tasks, such as reverse engineering (Pearce et al. 2022b), code explanation (Zhuo et al. 2023), code analysis (Feiyu 2023), and vulnerability repair (Pearce et al. 2022a).”
AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., & Li, Y.-F. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.77 · Cited by 31
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
A Hazard Analysis Framework for Code Synthesis Large Language Models
Khlaaf, H., Mishkin, P., Achiam, J., Krueger, G., & Brundage, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14157 · Cited by 23
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2022) · arXiv · doi:10.48550/arxiv.2212.06742 · Cited by 18
A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages
Buscemi, A. (2023) · arXiv · doi:10.48550/arxiv.2308.04477 · Cited by 18 · cited 1×
“Such as performance status, defects, potential, completion, deficien-cies, etc.”
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Chai, Y., Wang, S., Pang, C., Sun, Y., Tian, H., & Wu, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.676 · Cited by 18
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2021) · arXiv · doi:10.48550/arxiv.2112.02125 · Cited by 17
Is Self-Repair a Silver Bullet for Code Generation?
Olausson, T. X., Inala, J. P., Wang, C., Gao, J., & Solar-Lezama, A. (2023) · arXiv · doi:10.48550/arxiv.2306.09896 · Cited by 13
Understanding Programs by Exploiting (Fuzzing) Test Cases
Zhao, J., Rong, Y., Guo, Y., He, Y., & Chen, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.678 · Cited by 13
Who wrote this code? Watermarking for code generation
Lee, T., Hong, S., Ahn, J., Hong, I., Lee, H., Yun, S., Shin, J., & Kim, G. (2023) · arXiv · doi:10.48550/arxiv.2305.15060 · Cited by 4
Cited by
A survey on large language models for software engineering
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4670-0 · Cited by 124
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
“At the current stage, LLMs can serve as useful assistants to software maintainers, but they are very far from replacing completely humans [143].”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
“To overcome traditional program analysis techniques that may not scale or incorporate natural language, one direction of current software engineering research has is to use neural networks, including LMs, to automate real-world software development processes (Maniatis et al., 2023; Zheng et al., 2023; Hou et al., 2023…”
MethodCodepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Metadata
- Authors
- Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y.
- Venue
- Empirical Software Engineering
- Published
- 2024-12-26
- Volume
- 30
- Issue
- 2
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W4405796591
- Cited by
- 116
- References
- 72
- FWCI
- 44.01
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.