Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 15

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Hu, Xing, Niu, Feifei, Chen, Junkai, Zhou, Xin, Zhang, Junwei, He, Junda, Xia, Xin, Lo, David·2025·ACM Transactions on Software Engineering and Methodology
DOI 10.1145/3786771
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are gaining increasing popularity in software engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available , how benchmarks are constructed , and the future outlook for these benchmarks . We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOps, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3786771

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 370

  1. Sallam Abualhaija, Davide Fucci, Fabiano Dalpiaz, and Xavier Franch. 2020. 3rd workshop on natural language processing for requirements engineering (NLP4RE’20). In Joint 26th International Conference on Requirements Engineering: Foundation for Software Quality Workshops, Doctoral Symposium, Live Studies Track, and Poster Track, Pisa; Italy; 24 March 2020 through 27 March 2020, Vol. 2584. CEUR-WS.
  2. Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2023. AVATAR: A Parallel Corpus for Java-Python Program Translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023. Association for Computational Linguistics, 2268–2281.
  3. Md Basim Uddin Ahmed, Nima Shiri Harzevili, Jiho Shin, Hung Viet Pham, and Song Wang. 2025. SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection. arXiv preprint arXiv:2505.19828 (2025). · arXiv
  4. 10.1145/3663533.3664036 · DOI
  5. Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar T. Devanbu, and Arie van Deursen. 2023. Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries. CoRR abs/2301.01701 (2023). · arXiv
  6. Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023. A3Test: Assertion-Augmented Automated Test Case Generation. CoRR abs/2302.10352 (2023). · arXiv
  7. 10.1109/ICSME58846.2023.00013 · DOI
  8. Md Tauseef Alam, Raju Halder, and Abyayananda Maiti. 2024. Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities. arXiv preprint arXiv:2409.10574 (2024). · arXiv
  9. Miltiadis Allamanis and Pengcheng Yin. 2025. Disproving Program Equivalence with LLMs. arXiv:2502.18473 [cs.SE] https://arxiv.org/abs/2502.18473 · arXiv
  10. Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and Jonathan Bell. 2021. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 1572–1584.
  11. Sima Arasteh, Georgios Nikitopoulos, Wei-Cheng Wu, Nicolaas Weideman, Aaron Portnoy, Mukund Raghothaman, and Christophe Hauser. 2025. BinPool: A Dataset of Vulnerabilities for Binary Security Analysis. arXiv preprint arXiv:2504.19055 (2025). · arXiv
  12. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868 (2022). · arXiv
  13. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). · arXiv
  14. Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2023. StudentEval: a benchmark of student-written prompts for large language models of code. arXiv preprint arXiv:2306.04556 (2023). · arXiv
  15. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72.
  16. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017, Volume 2: Short Papers. Asian Federation of Natural Language Processing, 314–319.
  17. Chip R Bell and Ron E Zemke. 1987. Service breakdown: the road to recovery. Management review 76, 10 (1987), 32.
  18. 10.1007/978-3-031-46002-9_23 · DOI
  19. 10.1145/3475960.3475985 · DOI · Paper page
  20. Paul E Black et al. 2017. Sard: A software assurance reference dataset. In Anonymous Cybersecurity Innovation Forum.(), Vol. 85.
  21. Paul E Black and Paul E Black. 2018. Juliet 1.3 test suite: Changes from 1.2. US Department of Commerce, National Institute of Standards and Technology ….
  22. 10.1145/3213846.3213872 · DOI
  23. Marcel Böhme, Ezekiel Olamide Soremekun, Sudipta Chattopadhyay, Emamurho Juliet Ugherughe, and Andreas Zeller. 2017. How developers debug software—the dbgbench dataset. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 244–246.
  24. 10.1109/MSR.2015.21 · DOI
  25. 10.1145/3524842.3528482 · DOI
  26. Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, et al. 2025. How Should I Build A Benchmark? arXiv preprint arXiv:2501.10711 (2025). · arXiv
  27. 10.1145/3691620.3695470 · DOI
  28. Jialun Cao, Meiziniu Li, Ming Wen, and Shing-chi Cheung. 2023. A study on prompt design, advantages and limitations of chatgpt for deep learning program repair. arXiv preprint arXiv:2304.08191 (2023). · arXiv
  29. Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, et al. 2025. Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks. arXiv preprint arXiv:2504.18838 (2025). · arXiv
  30. Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, and Smaranda Muresan. 2023. Creativity support in the age of large language models: An empirical study involving emerging writers. arXiv preprint arXiv:2309.12570 (2023). · arXiv
  31. Partha Chakraborty, Mahmoud Alfadel, and Meiyappan Nagappan. 2024. BLAZE: Cross-Language and Cross-Project Bug Localization via Dynamic Chunking and Hard Example Learning. arXiv preprint arXiv:2407.17631 (2024). · arXiv
  32. 10.1109/TSE.2024.3423712 · DOI
  33. 10.18653/v1/2023.findings-emnlp.614 · DOI · Paper page
  34. Shubham Chandel, Colin B Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and evaluating a jupyter notebook data science assistant. arXiv preprint arXiv:2201.12901 (2022). · arXiv
  35. Mahinthan Chandramohan, Dai Quoc Nguyen, Padmanabhan Krishnan, and Jovan Jancic. 2024. Supporting Cross-language Cross-project Bug Localization Using Pre-trained Language Models. arXiv preprint arXiv:2407.02732 (2024). · arXiv
  36. 10.1145/3641289 · DOI · Paper page
  37. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. GitHub repository (2023).
  38. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  39. 10.1145/3597503.3639085 · DOI
  40. 10.1109/ICSE55347.2025.00012 · DOI
  41. Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, et al. 2024. A survey on evaluating large language models in code generation tasks. arXiv preprint arXiv:2408.16498 (2024). · arXiv
  42. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arXiv:2107.03374 [cs.LG] · arXiv
  43. Meng Chen, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, Juhong Wang, and Xiaodong Gu. 2023. On the effectiveness of large language models in domain-specific code generation. arXiv preprint arXiv:2312.01639 (2023). · arXiv
  44. Simin Chen, Pranav Pusarla, and Baishakhi Ray. 2025. Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination. arXiv preprint arXiv:2503.04149 (2025). · arXiv
  45. 10.1145/3607199.3607242 · DOI
  46. 10.1109/ICSME55016.2022.00080 · DOI
  47. 10.1145/3510003.3510042 · DOI
  48. Jane Cleland-Huang, Michael Vierhauser, and Sean Bayley. 2018. Dronology: An incubator for cyber-physical system research. arXiv preprint arXiv:1804.02423 (2018). · arXiv
  49. Alireza DaghighFarsoodeh, Chung-Yu Wang, Hamed Taherkhani, Melika Sepidband, Mohammad Abdollahi, Hadi Hemmati, and Hung Viet Pham. 2025. Deep-Bench: Deep Learning Benchmark Dataset for Code Generation. CoRR abs/2502.18726 (2025). · arXiv
  50. 10.1609/aaai.v39i1.31988 · DOI
  51. 10.1109/RE.2019.00025 · DOI
  52. Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. CoRR abs/2304.02014 (2023). · arXiv
  53. Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624 (2024). · arXiv
  54. Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems 36 (2024).
  55. Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, William K. Robertson, Frederick Ulrich, and Ryan Whelan. 2016. LAVA: Large-Scale Automated Vulnerability Addition. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016. IEEE Computer Society, 110–121.
  56. Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang. 2023. Towards understanding the capability of large language models on code clone detection: a survey. arXiv preprint arXiv:2308.01191 (2023). · arXiv
  57. Mingzhe Du, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. 2024. Mercury: An efficiency benchmark for llm code synthesis. arXiv preprint arXiv:2402.07844 (2024). · arXiv
  58. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861 (2023). · arXiv
  59. Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2024. Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. arXiv preprint arXiv:2406.11147 (2024). · arXiv
  60. Richard A Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C Cordeiro, and Norbert Tihanyi. 2025. Castle: Benchmarking dataset for static code analyzers and llms towards cwe detection. arXiv preprint arXiv:2503.09433 (2025). · arXiv
  61. 10.1145/3510454.3528648 · DOI
  62. Abdelkarim El-Hajjami, Nicolas Fafin, and Camille Salinesi. 2023. Which AI Technique Is Better to Classify Requirements? An Experiment with SVM, LSTM, and ChatGPT. arXiv preprint arXiv:2311.11547 (2023). · arXiv
  63. Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. 2023. Formalizing natural language intent into program specifications via large language models. arXiv preprint arXiv:2310.01831 (2023). · arXiv
  64. 10.1002/9780470146354 · doi:10.1002/9780470146354
  65. 10.1145/3510003.3510157 · DOI
  66. 10.1145/3379597.3387501 · DOI · Paper page
  67. 10.1109/ICSE48619.2023.00128 · DOI · Paper page
  68. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144 13 (2024), 14. · arXiv
  69. 10.1145/3691620.3695552 · DOI
  70. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020). · arXiv
  71. Mohamed Amine Ferrag, Ammar Battah, Norbert Tihanyi, Merouane Debbah, Thierry Lestable, and Lucas C Cordeiro. 2023. Securefalcon: The next cyber reasoning system for cyber security. arXiv preprint arXiv:2307.06616 (2023). · arXiv
  72. 10.1145/3324884.3415298 · DOI
  73. Giorgio Franceschelli and Mirco Musolesi. 2024. On the creativity of large language models. AI & SOCIETY (2024), 1–11.
  74. 10.1145/2685612 · DOI
  75. Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940 (2023). · arXiv
  76. 10.1145/3632746 · DOI
  77. 10.1145/3540250.3549098 · DOI
  78. Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383 (2024). · arXiv
  79. Xiang Gao, Bo Wang, Gregory J Duck, Ruyi Ji, Yingfei Xiong, and Abhik Roychoudhury. 2021. Beyond tests: Program vulnerability repair via crash constraint extraction. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 2 (2021), 1–27.
  80. Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420 (2023). · arXiv
  81. 10.1145/3395363.3397385 · DOI
  82. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. CRUXEval: a benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning. 16568–16621.
  83. Tianyang Gu, Jingjin Wang, Zhihao Zhang, and HaoHong Li. 2024. LLMs can realize combinatorial creativity: generating creative ideas via LLMs for scientific research. arXiv preprint arXiv:2412.14141 (2024). · arXiv
  84. 10.1145/3597503.3623306 · DOI
  85. 10.1007/978-3-031-70879-4_14 · DOI
  86. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736 (2023). · arXiv
  87. Mohammad Kasra Habib, Daniel Graziotin, and Stefan Wagner. 2025. ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation. arXiv preprint arXiv:2505.17632 (2025). · arXiv
  88. Nam Le Hai, Dung Manh Nguyen, and Nghi D. Q. Bui. 2025. On the Impacts of Contexts on Repository-Level Code Generation. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025. Association for Computational Linguistics, 1496–1524.
  89. Hojae Han, Seung-won Hwang, Rajhans Samdani, and Yuxiong He. 2025. ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
  90. Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei. 2022. Aixbench: A code generation benchmark dataset. arXiv preprint arXiv:2206.13179 (2022). · arXiv
  91. Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with NumPy. Nature 585, 7825 (2020), 357–362.
  92. Junda He, Christoph Treude, and David Lo. 2024. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead. ACM Transactions on Software Engineering and Methodology (2024).
  93. 10.1145/3576915.3623175 · DOI
  94. Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2019. Global relational models of source code. In International conference on learning representations.
  95. Robert Kraig Helmeczi, Mucahit Cevik, and Savas Yıldırım. 2023. Few-shot learning for sentence pair classification and its applications in software engineering. arXiv preprint arXiv:2306.08058 (2023). · arXiv
  96. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021).
  97. Tobias Hey, Jan Keim, Anne Koziolek, and Walter F Tichy. 2020. Norbert: Transfer learning for requirements classification. In 2020 IEEE 28th international requirements engineering conference (RE). IEEE, 169–179.
  98. 10.1145/3691620.3695072 · DOI · Paper page
  99. 10.1109/TPS-ISA58951.2023.00044 · DOI
  100. Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, and Kaidi Xu. 2025. DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation. CoRR abs/2503.10452 (2025). · arXiv
  101. 10.1145/3196321.3196334 · DOI · Paper page
  102. 10.5555/3304889.3304975 · doi:10.5555/3304889.3304975
  103. Yucheng Hu and Yuxing Lu. 2024. RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing. arXiv preprint arXiv:2404.19543 (2024). · arXiv
  104. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037 (2024). · arXiv
  105. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403 (2022). · arXiv
  106. Qing Huang, Yishun Wu, Zhenchang Xing, He Jiang, Yu Cheng, and Huan Jin. 2023. Adaptive Intellect Unleashed: The Feasibility of Knowledge Transfer in Large Language Models. arXiv preprint arXiv:2308.04788 (2023). · arXiv
  107. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs/1909.09436 (2019). · arXiv
  108. 10.1145/3729379 · DOI
  109. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 (2024). · arXiv
  110. Prithwish Jana, Piyush Jha, Haoyang Ju, Gautham Kishore, Aryan Mahajan, and Vijay Ganesh. 2023. CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution. arXiv preprint arXiv:2306.06755 (2023). · arXiv
  111. Sehun Jeong, Hakjoo Oh, Minje Choi, and Jaegul Choo. 2017. End-to-End prediction of buffer overruns from raw source code via neural memory networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization.
  112. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. CoRR abs/2406.00515 (2024). · arXiv
  113. Nan Jiang, Chengxiao Wang, Kevin Liu, Xiangzhe Xu, Lin Tan, and Xiangyu Zhang. 2023. Nova+: Generative Language Models for Binaries. CoRR abs/2311.13721 (2023). · arXiv
  114. 10.1145/3650212.3652123 · DOI
  115. Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen. 2023. On the Evaluation of Neural Code Translation: Taxonomy and Benchmark. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 1529–1541.
  116. Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In ICLR.
  117. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479 (2024). · arXiv
  118. 10.1145/3611643.3613892 · DOI · Paper page
  119. Xin Jin, Jonathan Larson, Weiwei Yang, and Zhiqiang Lin. 2023. Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models. CoRR abs/2312.09601 (2023). · arXiv
  120. 10.1145/3660769 · DOI
  121. Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems 35 (2022), 11785–11799.
  122. 10.1145/2610384.2628055 · DOI · Paper page
  123. Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2023. Explainable automated debugging via large language model-driven scientific debugging. arXiv preprint arXiv:2304.02195 (2023). · arXiv
  124. 10.1145/3635059.3635104 · DOI · Paper page
  125. Arshiya Khan, Guannan Liu, and Xing Gao. 2024. Code vulnerability repair with large language model using context-aware prompt tuning. arXiv preprint arXiv:2409.18395 (2024). · arXiv
  126. Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Do Long, Weishi Wang, Md. Rizwan Parvez, and Shafiq Joty. 2024. XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. Association for Computational Linguistics, 6766–6805.
  127. Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1–26.
  128. Kristian Kolthoff, Christian Bartelt, and Simone Paolo Ponzetto. 2023. Data-driven prototyping via natural-language-based GUI retrieval. Automated software engineering 30, 1 (2023), 13.
  129. 10.1109/RE59067.2024.00045 · DOI
  130. 10.1145/3729390 · DOI
  131. Vasileios Kouliaridis, Georgios Karopoulos, and Georgios Kambourakis. 2024. Assessing the effectiveness of llms in android application vulnerability analysis. In International Conference on Attacks and Defenses for Internet-of-Things. Springer, 139–154.
  132. 10.1109/RE59067.2024.00056 · DOI
  133. Ivan Victor Krsul. 1998. Software vulnerability analysis. Purdue University.
  134. 10.18653/v1/2025.naacl-long.348 · DOI
  135. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning. PMLR, 18319–18345.
  136. 10.1109/ICST.2019.00038 · DOI
  137. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurIPS.
  138. 10.1145/3589335.3651463 · DOI
  139. 10.1109/ICSE.2019.00087 · DOI · Paper page
  140. 10.1145/3213846.3213856 · DOI
  141. Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, and Sundong Kim. 2024. Reasoning abilities of large language models: In-depth analysis on the abstraction and reasoning corpus. arXiv preprint arXiv:2403.11793 (2024). · arXiv
  142. Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, and Pilsung Kang. 2024. CheckEval: Robust Evaluation Framework using Large Language Model via Checklist. arXiv preprint arXiv:2403.18771 (2024). · arXiv
  143. VI Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Proceedings of the Soviet physics doklady (1966).
  144. Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. 2024. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604 (2024). · arXiv
  145. Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2404.00599 (2024). · arXiv
  146. Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et al. 2024. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2405.19856 (2024). · arXiv
  147. Jun Li, Lixian Li, Jin Liu, Xiao Yu, Xiao Liu, and Jacky Wai Keung. 2024. Large language model ChatGPT versus small deep learning models for self-admitted technical debt detection: Why not together? Software: Practice and Experience (2024).
  148. 10.1145/3540250.3549099 · DOI
  149. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852 (2023). · arXiv
  150. Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation. CoRR abs/2503.06680 (2025). · arXiv
  151. 10.1109/ICST49551.2021.00034 · DOI
  152. Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. CoRR abs/2203.07814 (2022). · arXiv
  153. Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, and Michael R Lyu. 2023. Exploring the effectiveness of llms in automated logging generation: An empirical study. arXiv preprint arXiv:2307.05950 (2023). · arXiv
  154. 10.1145/3643754 · DOI
  155. Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, Peng Cheng, Kangjie Lu, and Ting Wang. 2021. UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021. USENIX Association, 2777–2794.
  156. Z Li, S Dutta, and M Naik. [n. d.]. LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. arXiv 2024. arXiv preprint arXiv:2405.17238 ([n. d.]). · arXiv
  157. 10.1145/3540250.3549081 · DOI
  158. 10.1145/3597503.3639218 · doi:10.1145/3597503.3639218
  159. 10.1109/TDSC.2021.3076142 · DOI · Paper page
  160. 10.1109/TDSC.2021.3051525 · DOI · Paper page
  161. Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018). · arXiv
  162. Qingyuan Liang, Zeyu Sun, Qihao Zhu, Wenjie Zhang, Lian Yu, Yingfei Xiong, and Lu Zhang. 2021. Lyra: A benchmark for turducken-style code generation. arXiv preprint arXiv:2108.12144 (2021). · arXiv
  163. Shanchao Liang, Yiran Hu, Nan Jiang, and Lin Tan. 2024. Can Language Models Replace Programmers? REPOCOD Says’ Not Yet’. arXiv preprint arXiv:2410.21647 (2024). · arXiv
  164. Bo Lin, Shangwen Wang, Liqian Chen, and Xiaoguang Mao. 2024. There are More Fish in the Sea: Automated Vulnerability Repair via Binary Templates. arXiv preprint arXiv:2411.18088 (2024). · arXiv
  165. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
  166. 10.1145/3135932.3135941 · DOI · Paper page
  167. 10.1145/3133956.3138840 · DOI
  168. 10.1109/ICSE43902.2021.00040 · DOI
  169. Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. 2018. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. arXiv preprint arXiv:1802.08979 (2018). · arXiv
  170. 10.1145/3533767.3534380 · DOI
  171. 10.48550/ARXIV.2502.02757 · arXiv · doi:10.48550/arxiv.2502.02757
  172. Ethan TS Liu, Austin Wang, Spencer Mateega, Carlos Georgescu, and Danny Tang. 2025. VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation. arXiv preprint arXiv:2505.19395 (2025). · arXiv
  173. 10.1145/3597926.3598092 · DOI
  174. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024).
  175. Kaiyuan Liu, Youcheng Pan, Jing Li, Daojing He, Yang Xiang, Yexing Du, and Tianrun Gao. 2025. ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation. CoRR abs/2503.07010 (2025). · arXiv
  176. Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091 (2023). · arXiv
  177. Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024. Vuldetectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595 (2024). · arXiv
  178. Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1355–1367.
  179. 10.18653/v1/2022.findings-emnlp.449 · DOI
  180. 10.1109/TSE.2024.3392499 · DOI · Paper page
  181. Chaomeng Lu, Tianyu Li, Toon Dehaene, and Bert Lagaisse. 2025. ICVul: A Well-labeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs. arXiv preprint arXiv:2505.08503 (2025). · arXiv
  182. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021). · arXiv
  183. 10.1109/ASP-DAC58780.2024.10473904 · DOI · Paper page
  184. 10.1145/3551349.3560417 · DOI
  185. Yu Luo, Weifeng Xu, Karl Andersson, Mohammad Shahadat Hossain, and Dianxiang Xu. 2024. FELLMVP: An Ensemble LLM Framework for Classifying Smart Contract Vulnerabilities. In 2024 IEEE International Conference on Blockchain (Blockchain). IEEE, 89–96.
  186. Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu. 2024. SpecGen: Automated Generation of Formal Program Specifications via Large Language Models. arXiv preprint arXiv:2401.08807 (2024). · arXiv
  187. Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, and Yang Liu. 2024. Combining fine-tuning and llm-based agents for intuitive smart contract auditing with justifications. arXiv preprint arXiv:2403.16073 (2024). · arXiv
  188. 10.1109/SANER.2019.8667991 · DOI
  189. 10.48550/arXiv.2502.16704 · arXiv · doi:10.48550/arxiv.2502.16704
  190. 10.1109/ICICCT.2018.8473199 · DOI
  191. Shantanu Mandal, Adhrik Chethan, Vahid Janfaza, SM Mahmud, Todd A Anderson, Javier Turek, Jesmin Jahan Tithi, and Abdullah Muzahid. 2023. Large language models based automatic synthesis of software specifications. arXiv preprint arXiv:2304.09181 (2023). · arXiv
  192. Ruchira Manke, Mohammad Wardat, Foutse Khomh, and Hridesh Rajan. 2024. Leveraging data characteristics for bug localization in deep learning programs. ACM Transactions on Software Engineering and Methodology (2024).
  193. 10.1145/3290605.3300819 · DOI
  194. 10.1145/3597503.3639174 · DOI
  195. 10.1145/3510003.3511561 · DOI
  196. 10.1109/ICSE48619.2023.00181 · DOI
  197. Xiang Mei, Pulkit Singh Singaria, Jordi Del Castillo, Haoran Xi, Tiffany Bao, Ruoyu Wang, Yan Shoshitaishvili, Adam Doupé, Hammond Pearce, Brendan Dolan-Gavitt, et al. 2024. ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software. arXiv preprint arXiv:2408.02153 (2024). · arXiv
  198. Jonathan Metzman, László Szekeres, Laurent Simon, Read Sprabery, and Abhishek Arya. 2021. FuzzBench: an open fuzzer benchmarking platform and service. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021. ACM, 1393–1403.
  199. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? arXiv preprint arXiv:2502.12115 (2025). · arXiv
  200. 10.1145/3643763 · DOI · Paper page

200 of 370 shown

Metadata

Authors
Hu, X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D.
Venue
ACM Transactions on Software Engineering and Methodology
Publisher
Association for Computing Machinery (ACM)
Published
2025-12-29
ISSN
  • 1049-331X
  • 1557-7392
DOI
10.1145/3786771
Cited by
15
References
53
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.