The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Open accessAbstract
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Similarly, MBPP+ offers 35 times more test cases than the original MBPP.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the second case, a finite test suite necessarily under-approximates an executable's full specification (Liu et al., 2023; Le Goues et al., 2015).”
From ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026) · cites this paper as Liu et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“To further ensure accurate evaluation, EvalPlus (Liu et al., 2023) extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions in HumanEval.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 61
- T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-5, 2022.
- L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. · arXiv
- S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. In Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II, pages 386-407. Springer, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209-224, 2008.
- F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.
- S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725-741. IEEE, 2015.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023.
- Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE), 2024.
- U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM (JACM), 45(4):634-652, 1998.
- D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence, pages 202-222. Elsevier, 1981.
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not., 46(1):317-330, jan 2011.
- S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1-119, 2017.
- T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 21st USENIX Security Sym- posium (USENIX Security 12), pages 445-458, Bellevue, WA, Aug. 2012. USENIX Association.
- M. Ivanković, G. Petrović, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 955-963, 2019.
- S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, 2018.
- N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020, 2023. · arXiv
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. · arXiv
- A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In International Conference on Learning Representations, 2018.
- J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385-394, 1976.
- M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511, 2020. · arXiv
- K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pages 348-370. Springer, 2010.
- Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, 2022.
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages, 6(OOPSLA1):1-26, Apr. 2022.
- Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355-1367. IEEE, 2023.
- N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 65-79, 2021.
- S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. · arXiv
- Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. · arXiv
- Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
- W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
- B. Meyer. Applying'design by contract'. Computer, 25(10):40-51, 1992.
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32-44, 1990.
- G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83-94, 2000.
- E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint, 2023.
- E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2):58-62, 2005.
- OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
- G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice, pages 163-171, 2018.
- G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
- B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773, 2021. · arXiv
- K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev), pages 157-157. IEEE, 2016.
- D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In IJCAI, volume 75, pages 260-267, 1975.
- A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pages 246-256, 2014.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 241-252, 1969.
- Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023. · arXiv
- Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint arXiv:2309.00608, 2023. · arXiv
- D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1-25, 2020.
- C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE), 2024.
- C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
- C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program re- pair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-971, 2022.
- F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.
- T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
- L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 170-179. IEEE, 2011.
- Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, 2023. · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“to ChatGPT using a larger parameter setting, reaching about a 78% match (Liu et al., 2023b).”
MethodSelf-citationLarge language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Liu et al. [90] draw attention to the problem, showing how existing test suites can lead to high degrees of false positive conclusions (also a serious problem for online judge platforms [89]).”
BackgroundUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
“Humans bring unparalleled creativity, critical thinking, ethical judgment, and domain-specific knowledge [95].”
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
“Humans bring unparalleled creativity, critical thinking, ethical judgment, and domain-specific knowledge [95].”
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Jun-wei, L., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639219 · Cited by 271
“We select the state-of-the-art LLMs that have been widely studied in recent code generation work [44, 49].”
BackgroundMethodRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
“Specifically, popular benchmarks like HumanEval [9] encompassing 164 Python tasks, and MBPP [3] containing 974 programming tasks, have been widely used by prior research [7–9 , 32 ].”
MethodGenerative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128
“Kou et al. [88] investigate the attention alignment between the nature language description from humans and the code generation by LLMs. Liu et al. [25] propose EvalPlus framework to enhance code generation benchmarks.”
BackgroundSelf-citationYang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
“• Coding Tasks : EvalPlus (Liu et al., 2023a) (0-shot) (Average of HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), Humaneval+, MBPP+) (Liu et al., 2023a), MultiPL-E (Cassano et al., 2023) (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript), MBPP-3shot (Austin et al., 2021), CRUX-O of CRUXEva…”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 160
RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation Technique
Liu, S., Fang, W., Lu, Y., Wang, J., Zhang, Q., Zhang, H., & Xie, Z. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3483089 · Cited by 77
“Table VI shows pass@1 results on Veilog tasks from VerilogEval-Human [13], python tasks from HumanEval + [37], Cpp, and Sh programming tasks from [38].”
ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74
“Following prior work [29], ClarifyGPT leverages LLMs as the test input generator and generates test inputs by adopting a two-step approach (i.e., seed input initialization and type-aware mutation).”
Influential(2024) · doi:10.1109/sp54263.2024.00182 · Cited by 62
“LLMs have already demonstrated prowess in generating source code across various programming languages [35], [36].”
BackgroundHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Zhong, L., & Wang, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30185 · Cited by 96
“Liu et al. (2023) enlarges HumanEval (Chen et al. 2021) by generating test cases with higher coverage which serve as an add-on to the existing programming benchmarks but the evaluation still focuses on functional correctness and simple programming questions far from software development.”
BackgroundThe FormAI Dataset: Generative AI in Software Security through the Lens of Formal Verification
Tihanyi, N., Bisztray, T., Jain, R., Ferrag, M. A., Cordeiro, L. C., & Mavroeidis, V. (2023) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3617555.3617874 · Cited by 70
The future of AI-driven software engineering
Terragni, V., Vella, A., Roop, P., & Blincoe, K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715003 · Cited by 62
Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 70
“Second, recent studies [25], [4], [26] show that ChatGPT has potential in performing various software engineering tasks for Python programs.”
BackgroundSelf-citationPrompt-based automation of building code information transformation for compliance checking
Yang, F., & Zhang, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105817 · Cited by 59
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 75
“Recently, large language models (LLMs), such as GPT4 [3], have demonstrated impressive performance in code generation tasks [4].”
BackgroundMethodInfluentialChen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 66
“While LLMs might outperform average humans in specific, structured tasks[58], the evaluation criteria might have been flawed[50], as LLMs struggled to combine existing solutions for a novel challenge[23].”
BackgroundLLM4VV: Developing LLM-driven testsuite for compiler validation
Munley, C., Jarmusch, A., & Chandrasekaran, S. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.034 · Cited by 55
Xpert: empowering incident management with query recommendations via large language models
Jiang, Y., Zhang, C., He, S., Yang, Z., Ma, M., Qin, S., Kang, Y., Dang, Y., Rajmohan, S., Lin, Q., et al. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639081 · Cited by 45
DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming
Giunchi, D., Numan, N., Gatti, E., & Steed, A. J. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00078 · Cited by 57
“These tasks include generating boilerplate code, refactoring existing code, and translating natural language descriptions into new code functionalities [48].”
Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study
Wu, Y., Wan, Y., Zhang, H., Sui, Y., Wei, W., Zhao, W., Xu, G., & Jin, H. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654992 · Cited by 68
How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts
Wang, B., Wang, C., Liang, P., Li, B., & Zeng, C. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00046 · Cited by 46
“Especially, in Software Engineering (SE), LLMs have been effectively employed in code generation [1] [2], software architecture [3], software testing [4], etc.”
MethodQwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65
“5, a top-tier open-source LLM, Qwen2.”
BackgroundSelf-citationTihanyi, N., Charalambous, Y., Jain, R., Ferrag, M. A., & Cordeiro, L. C. (2025) · International Conference/Workshop on Automation of Software Test · doi:10.1109/ast66626.2025.00020 · Cited by 108
“Nonetheless, studies show that the codes and patches synthesized by GPT models may be incorrect and untrustworthy [25], [34], [85]–[88].”
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
Xiong, L., Wang, H., Chen, X., Sheng, L., Xiong, Y., Liu, J., Xiao, Y., Chen, H., Han, Q., & Tang, Y. S. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125495 · Cited by 50
“In 2022, OpenAI launched ChatGPT, a generative AI chat-bot based on LLMs, which is suitable for natural language processing tasks, such as question answering [19], [20], conversational agents [21], [22], [23], and code completion [24], [25].”
Self-citationThe Plastic Surgery Hypothesis in the Era of Large Language Models
Xia, C. S., Ding, Y., & Zhang, L. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00047 · Cited by 67
“Using this paradigm, researchers build larger and larger models (as large as 540B in the number of model parameters [54]) and demonstrated impressive results on code-related tasks, especially on program synthesis [44], [55], [56], [57], [58].”
Self-citationHajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 87
“For more details, please refer to Liu et al. [55]).”
BackgroundInfluentialCould ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 56
“For instance, observations from various works (and our study) show that models excel in generating code to solve software problems [26, 31, 32, 55, 65].”
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 128
“Protected attributes are highlighted in bold.”
Tosi, D. (2024) · Future Internet · doi:10.3390/fi16060188 · Cited by 49
“In [5], the authors focus on the implementation of EvalPlus [6], a benchmarking framework designed to assess the correctness of LLM-generated code by supplementing a substantial quantity of test cases using an LLM-based and mutation-based (test case) generation approach.”
MethodWickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 58
“Even with the benchmarking rates from HumanEval, Liu et al. [64] raised questions about the accuracy of the functionality of the generated code.”
Automatic smart contract comment generation via large language models and in-context learning
Zhao, J. L., Chen, X., Yang, G., & Shen, Y. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107405 · Cited by 34
Demi̇r, G. B., Sükut, Y., Duran, G. S., Topsakal, K. G., & Görgülü, S. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae011 · Cited by 34
An Empirical Study on Challenges for LLM Application Developers
Chen, X., Gao, C., Chen, C., Zhang, G., & Liu, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715007 · Cited by 73
“General Questions encompass challenges that lack specific implementation details and are often posed by developers seeking foundational knowledge in LLM development.”
InfluentialA Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT
Yan, D., Gao, Z., & Liu, Z. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00096 · Cited by 47
“Ad-ditionally, in 2023, Liu et al. [42] proposed EvalPlus, an evaluation framework that precisely assesses the functional correctness of LLM-generated code.”
Calibration and Correctness of Language Models for Code
Spieß, C. K., Gros, D., Pai, K. S., Pradel, M., Rabin, M. R. I., Alipour, M. A., Jha, S., Dévanbu, P., & Ahmed, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00040 · Cited by 102
“For function synthesis task, temperature zero is most accurate and is fairly standard practice when doing pass@1 with only one solution to generate and present [13], [19].”
CodeV: Empowering LLMs With HDL Generation Through Multilevel Summarization
Zhao, Y., Huang, D. S., Li, C., Jin, P., Song, M., Xu, Y., Nan, Z., Gao, M., Ma, T., Lei, Q., et al. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3604320 · Cited by 118
“For instance, while GPT-4 achieves an 88.4% pass@1 score on the Python benchmark HumanEval [18], [19], it only reaches 43.5% on the VerilogEval-Human benchmark [20].”
InfluentialSeveral Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
“WizardCoder [63] Four code generation benchmarks: HumanEval[125], HumanEval+[126], MBPP[127], and DS-1000[128] Initialized it with the 20K instructionfollowing dataset called Code Alpaca[67].”
MethodSOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents
Lin, F., Kim, D. J., & Chen, T.-H. P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00140 · Cited by 150
“…: We conduct a fine-grained evaluation on the quality of the generated code using four popular code generation benchmarks: Humaneval [8], Humaneval-ET [10], MBPP [3], MBPP-ET [21], comparing agent interactions and their effect on both accuracy (Pass@1) and other code quality metrics (e.g., smells).”
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
Peng, J., Cui, L., Huang, K., Yang, J., & Ray, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00009 · Cited by 84
“Apart from enabling the cross-check with coding task specifications and test oracles, reference implementations also lay a foundation for future extensions, such as the differential testing and test oracle augmentation in EvalPlus [11].”
InfluentialBigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“Previous works like EvalPlus [35] have attempted to extend the limited test cases by augmenting the input-output pairs via LLM-and mutation-based strategies.”
MethodInfluentialSelf-citationBai, X., Huang, S., Wei, C., & Wang, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126357 · Cited by 34
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00080 · Cited by 45
“LLMs have been extensively studied for function-level coding tasks and have shown promising results in function-level benchmarks such as HumanEval [11], [12] and MBPP [13].”
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
“This underscores the interaction of diverse code knowledge types and stresses the significance of efficiently mining and utilizing code knowledge in future studies.”
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
“LLMs have been extensively studied for function-level coding tasks and have shown promising results in function-level benchmarks such as HumanEval [11], [12] and MBPP [13].”
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 34
“…Datasets: HumanEval, MBPP CoderEval [27] Repository: Github Multipl-E [28] Datasets: HumanEval, MBPP DS-1000 [29] Forum: Stack Overflow HumanEval+ [30] Dataset: HumanEval CONCODE [31] Repository: Github R-benchmark [32] Text Books JuICe [33] Repository: Github (Jupyter, nbgrader) Exec-CSN [34]…”
Chain-of-programming (CoP): empowering large language models for geospatial code generation task
Hou, S., Jiao, H., Shen, Z., Liang, J., Zhao, A., Zhang, X., Wang, J., & Wu, H. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2509812 · Cited by 39
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Artificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 42
“Convolutional neural networks and long short-term memory-based models, for instance, Bi-directional (Bi-)long short-term memory and Gate Recurrent Units, emerge as the two most frequently employed deep learning techniques in software testing.”
Rane, N., Choudhary, S., & Rane, J. (2024) · Studies in Economics and Business Relations · doi:10.48185/sebr.v5i1.1051 · Cited by 30
HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineers
Yang, Y., Teng, F., Liu, P., Qi, M., Lv, C., Li, J., Zhang, X., & He, Z. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993072 · Cited by 24
"Generate" the Future of Work through AI: Empirical Evidence from Online Labor MarketsPreprint
Liu, J., Xu, X.-C., Li, Y.-J., & Tan, Y. (2023) · Social Science Research Network · doi:10.2139/ssrn.4529739 · Cited by 33
Self-citationFu, Y., Xiong, D., & Dong, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29756 · Cited by 37
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715004 · Cited by 53
“As explained in Section IV-A, to evaluate whether our approach can better utilize the LLM-generated patches in program repair, we compare the results of G IANT R EPAIR with the repair results when using the LLM-generated patches directly.”
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
“In contrast, LUNA is task-universal and more flexible with different trustworthiness perspectives.”
BackgroundTowards Sustainable Large Language Model Serving
Nguyen, S., Zhou, B., Ding, Y., & Liu, S. (2024) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3727200.3727220 · Cited by 29
“Large language models (LLMs) have revolutionized numerous industries, including search engines, natural language processing, and programming [14, 26, 32].”
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 68
“Algorithm-centric benchmarks, including HumanEval [7], MBPP [8], and their extended counterparts (e.g., Hu-manEval+, MultiPL, MBPP+) [32], primarily focus on small-scale code generation but fall short in evaluating broader comprehension.”
SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidance
Vungarala, D., Alam, S., Ghosh, A., & Angizi, S. (2024) · IEEE International Conference on Rebooting Computing (ICRC) · doi:10.1109/icrc64395.2024.10937006 · Cited by 29
“5 on the HumanEval [27] coding tasks [28].”
AutoVerus: Automated Proof Generation for Rust Code
Yang, C., Li, X., Misu, M. R. H., Yao, J., Cui, W., Gong, Y., Hawblitzel, C., Lahiri, S., Lorch, J. R., Lu, S., et al. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763174 · Cited by 20
“Furthermore, although Verus proof annotations (short as proof) are written in Rust syntax, their usage is very different from regular Rust executable code and require expertise in verification to master.”
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 27
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24
“However, instead of embedding all files in the repository, A GENTLESS first filters out irrelevant folders.”
BackgroundSelf-citationLPR: Large Language Models-Aided Program Reduction
Zhang, M., Tian, Y., Xu, Z., Dong, Y., Tan, S. H., & Sun, C. P. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652126 · Cited by 21
“Recently, LLMs have been applied and assessed on various software engineering tasks, such as automatic program repair [9, 12, 39–41] and program generation [20, 32, 46].”
MethodResultSelf-HWDebug: Automation of LLM Self-Instructing for Hardware Security Verification
Akyash, M., & Kamali, H. M. (2024) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi61997.2024.00077 · Cited by 20
“…intervention in implementation tasks [14], [15], (2) acting as a substitute for conventional hardware generators, e.g., high-level synthesis (HLS) [16], (3) addressing the persistent issue of inadequate HDL codebase availability [14], (4) facilitating the acceleration of time-to-market (TTM) in…”
BackgroundMa, D., Zhang, C., Xu, Q., & Zhou, G. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103078 · Cited by 21
The Impact of Prompt Programming on Function-Level Code Generation
Khojah, R., Neto, F. G. D. O., Mohamad, M., & Leitner, P. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3587794 · Cited by 59
“However, the increased adoption of LLMs for code-related tasks has unveiled risks and limitations, such as hallucinations, in-accuracies, and potential vulnerabilities [15], [16].”
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“HumanEval (Chen et al., 2021; Liu et al., 2023b) has emerged as the standard for this task.”
BackgroundMethodRug: Turbo Llm for Rust Unit Test Generation
Cheng, X. C., Sang, F., Zhai, Y., Zhang, X., & Kim, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00097 · Cited by 26
“The powerful induction ability of LLMs shows their application in software engineering [25], especially in code generation testing [31], [35]–[39].”
Isolating Compiler Bugs by Generating Effective Witness Programs With Large Language Models
Tu, H., Zhou, Z., Jiang, H., Yusuf, I. N. B., Li, Y., & Jiang, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3397822 · Cited by 28
“3) Large Language Models (LLMs): Recently, pre-trained Large Language Models (LLMs) such as ChatGPT [5] become ubiquitous and have exhibited remarkable performance in numerous tasks, such as machine translation [17], text summarization [18], classification [19] and code generation [20].”
BackgroundInfluentialJohns, B., Carroll, K., Medina, C., Lewark, R., & Walliser, J. C. (2024) · INCOSE International Symposium · doi:10.1002/iis2.13201 · Cited by 17
“While attention has been dedicated to assessing GPT’s proficiency in generating syntactically correct programming code (Biswas, 2023; Liu et al., 2023; Sakib et al., 2023), there has been limited exploration of its capacity to develop comprehensive conceptual models and system models.”
Black, G., Rimal, B. P., & Vaidyan, V. M. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3446695 · Cited by 15
Assessing the Performance of AI-Generated Code: A Case Study on GitHub Copilot
Li, S., Cheng, Y., Chen, J., Xuan, J., He, S., & Shang, W. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00030 · Cited by 25
“Liu et al. [33] found three instances of inefficient implementations within the HumanEval ground truth, which caused slow performance on inputs of reasonable size.”
Discriminating Human-authored from ChatGPT-Generated Code Via Discernable Feature Analysis
Li, K., Hong, S., Fu, C., Zhang, Y., & Liu, M. (2023) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew60843.2023.00059 · Cited by 18
“The opaque nature of ChatGPT’s training data spawns uncertainties regarding the provenance of its generated code snippets and the possibility of their harboring security vulnerabilities or unsafe code fragments[8, 10], which could entangle developers in copyright disputes and code security quandaries.”
BackgroundMethodSpennemann, D. H. R. (2023) · Preprints.org · doi:10.20944/preprints202308.1271.v1 · Cited by 14
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3657432 · Cited by 356
Self-citationChatGPT and computational-based research: benefits, drawbacks, and machine learning applications
Atkinson, C. F. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00091-3 · Cited by 18
“Another problem is that code may be ‘functionally correct’, insofar as it runs, however, it may not be true [70].”
BackgroundInfluentialOptLLM: Optimal Assignment of Queries to Large Language Models
Liu, Y., Zhang, H., Miao, Y., Le, V.-H., & Li, Z. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00098 · Cited by 28
“These models are trained on extensive text data [1] and have demonstrated impressive abilities to simulate human linguistic capabilities, posing a significant impact across multiple domains such as natural language processing (NLP) [2]–[4] and programming tasks [5]–[7].”
BackgroundFixing Large Language Models' Specification Misunderstanding for Better Code Generation
Tian, Z., Chen, J.-J., & Zhang, X.-Y. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00108
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
Translating code with Large Language Models and human-in-the-loop feedback
Siano, G. D. D., Fasolino, A. R., Sperlí, G., & Vignali, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107785 · Cited by 16
CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph
Xu, H., Ma, W., Zhou, T., Zhao, Y., Chen, K., Hu, Q., Liu, Y., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00079 · Cited by 37
“Due to hallucinations and a lack of high-quality training data, code generated by LLMs often contains syntax and semantic errors [22], [32], [34], which may affect the reliability of fuzz drivers.”
“Who will be left behind?”: A Swedish case of learning AI in vocational education
Metreveli, A., Chen, X., Hedman, A., & Sergeeva, A. (2025) · International Journal of Educational Research · doi:10.1016/j.ijer.2025.102697 · Cited by 17
From Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 23
“The rest of this section is designed to take the reader on a journey from the early methodologies to the current, advanced applications of AI in SE such as GitHub Copilot [6] and ChatGPT [2].”
Unified modeling language code generation from diagram images using multimodal large language models
Bates, A., Vavricka, R., Carleton, S., Shao, R., & Pan, C. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100660 · Cited by 25
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 17
“Although there emerges a substantial number of evaluation on code completion, a body of research focused on assessing the overall accuracy across various benchmarks [11], [30], [31], [32], [33].”
Self-citationA Student-Centric Evaluation Survey to Explore the Impact of LLMs on UML Modeling
Al-Ahmad, B. I., Alsobeh, A., Meqdadi, O., Shaikh, N., & Kabir, M. F. (2025) · Information · doi:10.3390/info16070565 · Cited by 12
AI Meets the Classroom: When Do Large Language Models Harm Learning?
Lehmann, M., Cornelius, P. B., & Sting, F. J. (2024) · arXiv · doi:10.48550/arxiv.2409.09047 · Cited by 11
“LLMs also exhibit strong coding abilities, which includes problem-solving and the ability to reason about self or user-generated code (Touvron et al. 2023, Gemini Team et al. 2023, Liu et al. 2024a).”
Correctness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 39
Carbon Footprint Evaluation of Code Generation through LLM as a Service
Vartziotis, T., Schmidt, M., Dasoulas, G., Dellatolas, I., Attademo, S., Le, V. D., Wiechmann, A., Hoffmann, T., Keckeisen, M., & Kotsopoulos, S. D. (2024) · Proceedings · doi:10.1007/978-3-658-45010-6_15 · Cited by 11
LLMs for Code: The Potential, Prospects, and Problems
Sharma, T. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00067 · Cited by 11
“However, they bring diverse challenges and problems, such as the generation of inaccurate, buggy, and vulnerable code [4].”
Background
Metadata
- Authors
- Liu, J., Xia, C. S., Wang, Y., & Zhang, L.
- ORCID
- Xia, Chunqiu Steven
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 21558-21572
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133198585
- Cited by
- 384
- References
- 0
- Citations by year
- 2026: 143 · 2025: 116 · 2024: 102 · 2023: 23
- FWCI
- 98.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2131943, 2141474)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, ChatGPT, program correctness, program verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
