The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Open accessAbstract
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Similarly, MBPP+ offers 35 times more test cases than the original MBPP.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the second case, a finite test suite necessarily under-approximates an executable's full specification (Liu et al., 2023; Le Goues et al., 2015).”
From ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026) · cites this paper as Liu et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“To further ensure accurate evaluation, EvalPlus (Liu et al., 2023) extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions in HumanEval.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 61
- T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-5, 2022.
- L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. · arXiv
- S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. In Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II, pages 386-407. Springer, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209-224, 2008.
- F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.
- S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725-741. IEEE, 2015.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023.
- Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE), 2024.
- U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM (JACM), 45(4):634-652, 1998.
- D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence, pages 202-222. Elsevier, 1981.
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not., 46(1):317-330, jan 2011.
- S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1-119, 2017.
- T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 21st USENIX Security Sym- posium (USENIX Security 12), pages 445-458, Bellevue, WA, Aug. 2012. USENIX Association.
- M. Ivanković, G. Petrović, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 955-963, 2019.
- S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, 2018.
- N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020, 2023. · arXiv
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. · arXiv
- A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In International Conference on Learning Representations, 2018.
- J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385-394, 1976.
- M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511, 2020. · arXiv
- K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pages 348-370. Springer, 2010.
- Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, 2022.
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages, 6(OOPSLA1):1-26, Apr. 2022.
- Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355-1367. IEEE, 2023.
- N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 65-79, 2021.
- S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. · arXiv
- Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. · arXiv
- Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
- W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
- B. Meyer. Applying'design by contract'. Computer, 25(10):40-51, 1992.
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32-44, 1990.
- G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83-94, 2000.
- E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint, 2023.
- E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2):58-62, 2005.
- OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
- G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice, pages 163-171, 2018.
- G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
- B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773, 2021. · arXiv
- K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev), pages 157-157. IEEE, 2016.
- D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In IJCAI, volume 75, pages 260-267, 1975.
- A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pages 246-256, 2014.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 241-252, 1969.
- Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023. · arXiv
- Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint arXiv:2309.00608, 2023. · arXiv
- D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1-25, 2020.
- C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE), 2024.
- C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
- C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program re- pair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-971, 2022.
- F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.
- T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
- L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 170-179. IEEE, 2011.
- Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, 2023. · arXiv
Cited by
Sorted by citations
Items 101–200 of 425
Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks
Smutný, P., & Bojko, M. (2024) · Applied Sciences · doi:10.3390/app142110048 · Cited by 14
AlOmar, E. A. (2025) · ACM Transactions on Computing Education · doi:10.1145/3722229 · Cited by 18
“Liu et al. [29] answered whether the code generated by ChatGPT is correct.”
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Iyanda, A. R., Bukar, U. A., & Hanapi, Z. M. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 17
“5, particularly in handling Java code [10], while GPT-4 studies focus on improving code efficiency [38] and enhancing code readability and quality [39].”
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 21
What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs
Bano, M., Gunatilake, H., & Hoda, R. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1109/icse-seis66351.2025.00023 · Cited by 24
“LLMs, widely used in SE for tasks such as recruitment [35], [36], requirements engineering [37], code generation [38], [39], and testing [40], pose significant risks of reinforcing societal biases.”
Basha, M., & Rodríguez-Pérez, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107804 · Cited by 15
Mokav: Execution-driven differential testing with LLMs
Etemadi, K., Mohammadi, B., Su, Z., & Monperrus, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112571 · Cited by 11
“Mokav significantly outperforms Pynguin [41] and DPP [33], the closest, state-of-the-art test generation tools for the considered task.”
Evaluating quantized Large Language Models for code generation on low-resource language benchmarks
Nyamsuren, E. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101351 · Cited by 18
Vulnerabilities Introduced by LLMs Through Code Suggestions
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_9 · Cited by 10
Khanshan, A., Gorp, P. V., & Markopoulos, P. (2024) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3661143 · Cited by 12
“…assistants are perceived to be aiding with productivity [11, 19, 53], the generated code often falls below standards of secure coding [18, 34], calling for further exploration and investigations both for prompt-response evaluation and inspecting the generated code in solving complex tasks [41].”
BackgroundSrcMarker: Dual-Channel Source Code Watermarking via Scalable Code Transformations
Yang, B., Li, W., Xiang, L., & Li, B. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00097 · Cited by 15
DeepMig: A transformer-based approach to support coupled library and code migrations
Rocco, J. D., Nguyen, P. T., Sipio, C. D., Rubei, R., Ruscio, D. D., & Penta, M. D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107588 · Cited by 13
Gopali, S., Siami‐Namini, S., Abri, F., & Namin, A. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100120 · Cited by 13
Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy
Alizadeh, N., Belchev, B., Saurabh, N., Kelbert, P., & Castor, F. (2025) · Empirical Software Engineering · doi:10.1109/msr66628.2025.00109 · Cited by 24
Large Language Models are Catalyzing Chemistry EducationPreprint
Du, Y., Duan, C., Bran, A. M., Sotnikova, A., Qu, Y., Kulik, H. J., Bosselut, A., Xu, J., & Schwaller, P. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-h722v · Cited by 10
Translating between SQL Dialects for Cloud Migration
Zmigrod, R., Alamir, S., & Liu, X. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3639727 · Cited by 9
“Research has demonstrated that LLMs are prone to hallucinations [12, 18, 21], meaning that there may be syntactic or semantic errors in any SQL segments generated.”
Real-GPT: Efficiently Tailoring LLMs for Informed Decision-Making in the Real Estate Industry
Gloria, B., Melsbach, J. W., Bienert, S., & Schoder, D. (2024) · Journal of Real Estate Portfolio Management · doi:10.1080/10835547.2024.2372748 · Cited by 10
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far are We?
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00064 · Cited by 17
A Survey on Self-Evolution of Large Language Models
Tao, Z., Lin, T.-E., Chen, X., Li, H., Wu, Y., Li, Y., Jin, Z., Huang, F., Tao, D., & Zhou, J. (2024) · arXiv · doi:10.48550/arxiv.2404.14387 · Cited by 9
“Despite these advancements, humans anticipate that the emerging generation of LLMs can be tasked with assignments of greater complexity, such as scientific discovery [81] and future events forecasting [95].”
BackgroundGenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
Hamid, A., Samidi, H. R., Pappachan, P., Finin, T., & Yus, R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0081 · Cited by 21
Don’t Stop Believin’: A Unified Evaluation Approach for LLM Honeypots
Weber, S. B., Feger, M., & Pilgermann, M. (2024) · IEEE Access · doi:10.1109/access.2024.3472460 · Cited by 15
“Recent research indicates that while code generated by LLMs may appear correct at first glance, it is often prone to errors [12].”
BackgroundChatGPT-driven machine learning code generation for android malware detection
Nelson, J., Pavlidis, M., Fish, A., Kapetanakis, S., & Polatidis, N. (2024) · The Computer Journal · doi:10.1093/comjnl/bxae114 · Cited by 10
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 15
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 11
LLM-Sentry: A Model-Agnostic Human-in-the-Loop Framework for Securing Large Language Models
Irtiza, S., Akbar, K. A., Yasmeen, A., Khan, L. R., Daescu, O., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00036 · Cited by 10
AutoPAC: Exploring LLMs for Automating Policy to Code Conversion in Business Organizations
Chowdhary, N., Dutta, T., Chattopadhyay, S., & Chakraborty, S. (2025) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets63942.2025.10885751 · Cited by 10
“However, as the t5-base model uses Masked Span Prediction (MSP) 14 , it is more sensitive towards declarative languages.”
InfluentialLeveraging LLMs for Dynamic IoT Systems Generation Through Mixed-Initiative Interaction
Adnan, B., Miryala, S., Sambu, A., Vaidhyanathan, K., Sanctis, M. D., & Spalazzese, R. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00073 · Cited by 8
“We chose these based on the EvalPlus [8] leaderboard.”
Gao, C., Chen, X., & Zhang, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107803 · Cited by 14
“In recent years, LLMs have achieved significant performance in various software engineering tasks, such as code generation [5, 6], code completion, source code summarization [5, 9], source code repair [21], and vulnerability detection [7, 8].”
Salerno, F., Al-Kaswan, A., & Izadi, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00080 · Cited by 8
“Starcoder2 and its derivatives are the best-performing models with both open weights and open data on the EvalPlus Leaderboard [38].”
Can LLMs Generate Higher Quality Code Than Humans? An Empirical Study
Jamil, M. T., Abid, S., & Shamail, S. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00081 · Cited by 15
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 17
Forgetful Large Language Models: Lessons Learned from Using LLMs in Robot Programming
Chen, J.-T., & Huang, C. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27721 · Cited by 14
Honarvar, S., van der Wilk, M., & Donaldson, A. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989005 · Cited by 8
“However, current instruction-tuned LLMs often generate incorrect code [5]–[8].”
Frankford, E., Höhn, I., Sauerwein, C., & Breu, R. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10662990 · Cited by 13
“Liu et al. [24] proposed EvalPlus , which builds upon HumanEval , adding the ability to detect more incorrectly synthesized code.”
BackgroundZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets Using LLMs
Kabir, A., Wang, S., Tian, Y., Chen, T.-H. P., Asaduzzaman, M., & Zhang, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702979 · Cited by 13
“For instance, on average, ZS4C − 4 achieves 0.98, 0.966, and 0.995 in terms of F1, Precision, and Recall on StatType-SO, respectively, as detailed in Table 3.”
InfluentialChen, Z., & Jiang, L. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00068 · Cited by 23
“Incorporate code quality metrics such as complexity, duplication, and code smells into the assessment criteria to ensure agents produce maintainable code [46], [47].”
Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00058 · Cited by 10
“The evaluation of these approaches is nowadays based on publicly available code benchmarks [5], [8], [16]–[20], being collections of well-documented programming tasks paired with ad-hoc test suites.”
Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
Zhang, Y., Xie, Y., Lit, S., Liu, P., Wang, C., Jia, Z., Huang, X., Song, J., Luo, C., Zheng, Z., et al. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00082 · Cited by 7
Promoting open science in test-driven software experiments
Kessel, M., & Atkinson, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.111971 · Cited by 7
Boosting LLM-Based Software Generation by Aligning Code with Requirements
Yaacov, T., Elyasaf, A., & Weiss, G. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00045 · Cited by 10
“While LLMs are proving useful in bridging requirements and their implementation, there is still a gap between the current state-of-the-art and the vision of code-less programming [6].”
BackgroundBenchmarking Large Language Models for Ethereum Smart Contract Development
Daspe, E., Durand, M., Hatin, J., & Bradai, S. (2024) · Conference on Blockchain Research & Applications for Innovative Networks and Services (BRAINS) · doi:10.1109/brains63024.2024.10732686 · Cited by 7
“This allows us to clarify the queries, thereby avoiding a common pitfall identified in the HumanEval benchmark [7].”
BackgroundFully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 12
Automating the Detection of Code Vulnerabilities by Analyzing GitHub Issues
Cipollone, D., Wang, C., Scazzariello, M., Ferlin, S., Izadi, M., Kostić, D., & Chiesa, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00010 · Cited by 11
“Also, LLMs have the capability to analyze and generate code across various programming languages, as demonstrated by their performance across multiple benchmarks [11].”
Jiang, X., Dong, Y., Tao, Y., Liu, H., Jin, Z. G., & Li, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00133 · Cited by 18
A Systematic Approach for Assessing Large Language Models’ Test Case Generation Capability
Chang, H., & Shirazi, M. S. (2025) · Software · doi:10.3390/software4010005 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“…GPT-3 (Brown et al., 2020), and ChatGPT, used as approximate stand-ins for shifts in how the community uses and therefore evaluates LMs. Common practice for evaluating autoregressive language models today diverges from the method described in the paper for all listed tasks except MMLU and MATH.”
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6
“4B [39], CodeGen-2B [8], CodeLlama-7B [9] on three typical benchmark datasets, i.e., HumanEval [1], HumanEvalPlus [36] and MBPP [37].”
An Empirical Study on Commit Message Generation Using LLMs via In-Context Learning
Wu, Y., Wang, Y., Li, Y., Tao, W., Yu, S., Yang, H.-W., Jiang, W., & Li, J. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00091 · Cited by 22
“E. Evaluation Models 1) LLMs: We select six LLM families with widespread popularity and exceptional performance in code generation [53].”
Self-citationJonnala, R., Yang, J. Y., Lee, Y., Liang, G., & Cao, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3585742 · Cited by 5
“While recent benchmarks, such as [12] and [13], have largely focused on the functional correctness of code generated by LLMs, the EFFIBENCH benchmark [14] uniquely assesses both correctness and efficiency.”
Lebioda, K., Petrović, N., Pan, F., Zolfaghari, V., Schamschurko, A., & Knoll, A. (2025) · IEEE Access · doi:10.1109/access.2025.3597748 · Cited by 5
“Many efforts are concentrated on benchmark tasks [6], [9], [24].”
Zhang, X., Wang, Y., He, A., Wang, H., & Li, T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129423 · Cited by 9
Self-citationIdentifying Security Bugs in Issue Reports: Comparison of BERT, N-gram IDF and ChatGPT
Yokoyama, D., Nishiura, K., & Monden, A. (2024) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera61261.2024.10685583 · Cited by 5
“Its advanced language generation capabilities and task flexibility have attracted the attention of many researchers, and its usefulness and appropriate applications have been studied in many fields, including software development and education [8] [11].”
BackgroundThe evolution of ChatGPT for programming: a comparative study
Jain, R., Thanvi, J., & Akila Subasinghe, A. (2025) · Engineering Research Express · doi:10.1088/2631-8695/ada51d · Cited by 8
Evaluating the Generalizability of LLMs in Automated Program Repair
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00024 · Cited by 11
“Our selection of LLMs is based on the EvalPlus leader-board [30], [31], which offers more rigorous tests for evaluating coding capabilities of LLMs.”
Can Large Language Models Better Predict Software Vulnerability?
Katsadouros, E., Patrikakis, C. Z., & Hurlburt, G. F. (2023) · IT Professional · doi:10.1109/mitp.2023.3284628 · Cited by 5
Efficient Text-to-Code Retrieval with Cascaded Fast and Slow Transformer Models
Gotmare, A. D., Li, J., Joty, S., & Hoi, S. C. H. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616369 · Cited by 7
“Besides, some recent studies [33] have found limitations with popular benchmarks like HumanEval that have been relied on to measure the correctness of model generated programs, suggesting that the synthesized program correctness scores of code LLMs have been overestimated.”
ResultOracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5
“However, a recent study [25] shows that the validation test suite of HumanEval and MBPP benchmarks are relatively weak.”
Symbolic Execution with Test Cases Generated by Large Language Models
Xu, J., Xu, J., Chen, T., & Ma, X. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00031 · Cited by 5
Robustness evaluation of code generation systems via concretizing instructions
Yan, M. X., Chen, J., Zhang, J. M., Cao, X., Yang, C., & Harman, M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107645 · Cited by 18
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
Aggarwal, P., Parno, B., & Welleck, S. (2024) · arXiv · doi:10.48550/arxiv.2412.06176 · Cited by 4
“Achieving perfect test coverage is typically infeasible Li et al. (2022b); Liu et al. (2023), and incomplete coverage leads to unreliable signals that models can exploit Pan et al. (2022); Liu et al. (2023); Denison et al. (2024).”
Evaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4
Chen, Z., Guo, W., Zhang, Y., Liu, X., Li, B., & Wang, H. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2634251 · Cited by 5
LLM-In-The-Loop: A Framework for Enhancing Drilling Data Analytics
Amin, S. T. A., Wu, K., Mathur, A., & Koritala, C. (2025) · SPE/IADC Drilling Conference · doi:10.2118/223805-ms · Cited by 6
Braess’s Paradox of Generative AI
Taitler, B., & Ben-Porat, O. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i13.33548 · Cited by 4
On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT
Colombo, G., Mariani, L., Micucci, D., & Riganelli, O. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00060 · Cited by 4
“However, the recommendation of using high-temperature values to avoid unwanted plagiarism has to be carefully balanced with settings recommended for code generation, where low-temperature values tend to produce better results [16], [25], [26].”
Fu, Y., Li, B., Li, L., Zhang, W., & Xie, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728947 · Cited by 7
“HumanEval has been widely used [27] and extended [38, 42] by existing studies, and many LLMs have achieved promising performance on this benchmark.”
TCSR‐SQL: Towards Table Content‐Aware Text‐to‐SQL With Self‐Retrieval
Xu, W., Liang, Y., Liu, C., Han, P., Zhu, H., Xu, Y., Liang, Y., & Zhang, B. (2025) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.70071 · Cited by 9
“This situation causes the standard SQL query to apply the knowledge of the relationship matching table (i.e., 𝑟𝑜𝑤𝑜𝑟𝑑𝑒𝑟 = 5 ).”
Automatically Testing Functional Properties of Code Translation Models
Enişer, H. F., Wüstholz, V., & Christakis, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30097 · Cited by 4
Exploring the Effectiveness of Abstract Syntax Tree Patterns for Algorithm Recognition
Neumüller, D., Sihler, F., Straub, R., & Tichy, M. (2024) · International Conference on Code Quality (ICCQ) · doi:10.1109/iccq60895.2024.10576984 · Cited by 4
Poster: Benchmarking of Code Generative LLMs
Rahman, M. M., Kundu, A., & Bertino, E. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00145 · Cited by 4
“Functional requirements mandate semantic correctness and completeness of the code [3].”
BackgroundScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00015 · Cited by 4
N-Version Assessment and Enhancement of Generative AI: Differential GAI
Kessel, M., & Atkinson, C. (2024) · IEEE Software · doi:10.1109/ms.2024.3469388 · Cited by 4
“5/-4 Turbo, Codex (GitHub CoPilot), CodeGen and InCoder (see 9 ), which are all based on the Transformer-driven LLM technology popularized by ChatGPT.”
GlueTest: Testing Code Translation via Language Interoperability
Abid, M. S., Pawagi, M., Adhikari, S., Cheng, X., Badr, R., Wahiduzzaman, M., Rathi, V., Qi, R., Li, C., Liu, L., et al. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00061 · Cited by 4
“One approach is limited to tests with inputs of primitive type , e.g., integers or strings [9]– [12], or types that can be easily translated from one language to another, e.g., Go structs to Rust structs [13].”
Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?
Vitale, A., Mastropaolo, A., Oliveto, R., Penta, M. D., & Scalabrino, S. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00033 · Cited by 8
“This approach has been proven effective for several SE tasks, including code generation [16], [17], [18], [19], program repair [20], [21], [22], [23], and, indeed, code summarization [24], [25], [10], [12], [26], [11].”
Comparative Analysis of ChatGPT, DeepSeek, and Gemini for Automated Code Generation
Shahzad, K., & Iqbal, S. (2025) · International Conference on Engineering of Modern Electric Systems · doi:10.1109/emes65692.2025.11045587 · Cited by 5
“In the same year 2023, J. Liu et al. introduced EvalPlus, a framework for evaluating the functional correctness of LLM-generated code by expanding test cases through LLM and mutation-based strategies [10].”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 804
“For Code-R1 evaluation, we follow the original codebase and adopt the default templates from the benchmarks, including LiveCodeBench prompt (Figure 14), HumanEval+, and MBPP+ prompt (Figure 15).”
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
BackgroundAre “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Wang, Y., Pradel, M., & Liu, Z. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764576 · Cited by 3
“We envision PatchDiff to be useful for building a sustainable and robust patch validation ecosystem for assessing issue-solving tools.”
Self-citationSteer Your Model: Secure Code Generation With Contrastive Decoding
Huang, L., Yan, M. L., Yin, T., Sun, W., Liu, Z., Zhang, H., & Lo, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3650127 · Cited by 4
“Previous studies have predominantly focused on the functional accuracy of code generated by LLMs [39], [40], often overlooking security—an aspect that is crucial, especially in security-sensitive domains.”
Reimagining feedback through generative AI in engineering education
Pecuchova, J., Benko, Ľ., & Drlík, M. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100574 · Cited by 5
Chen, Z., Chen, X., Li, Z. B., Xue, J., & Gao, C. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112855 · Cited by 5
Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement
Jin, H., & Chen, H. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00638-5 · Cited by 13
“For instance, recent studies have leveraged models such as GPT-4o to evaluate code submissions and determine whether they meet quality standards or require revisions [5].”
Automated Refactoring of Non-Idiomatic Python Code With Pythonic Idioms
Zhang, Z., Xing, Z., Zhao, D., Xu, X., Zhu, L., & Lu, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3420886 · Cited by 3
“These models are proficient in code translation, generation, and bug fixing [51], [52], [53], [54], [55], [56], [57], [58], making them suitable for identifying and refactoring non-idiomatic code patterns that our approach might overlook.”
BackgroundFine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats
Setak, M., & Madani, P. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00043 · Cited by 5
“Table 1 presents an evaluation of some of the code synthe-sizer LLMs [1], [25].”
BackgroundPrivCore: Multiplication-activation co-reduction for efficient private inference
Pang, Z., Wang, L., Yu, F., Zhao, K., Zeng, B., & Xu, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107307 · Cited by 3
Evaluating LLMs for Arabic Code Summarization: Challenges and Insights from GPT-4
Aljohani, A., Alharbi, R., Alkhaldi, A., & Aljedaani, W. (2025) · International Conference on Data Science and Machine Learning Applications (CDMA) · doi:10.1109/cdma61895.2025.00017 · Cited by 3
“GPT-4 has outperformed other LLMs in the HumanEvalPlus benchmark [14] and has been widely adopted in research and applications [6], [15], [22].”
Can Large Language Models Discover Metamorphic Relations? A Large-Scale Empirical Study
Zhang, J., Sun, C., Liu, H., & Dong, S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00011 · Cited by 5
“LLMs are pre-trained using billions of texts available on the Internet, and they can be applied to a wide range of applications, including text generation [41], code completion [42], and program repair [43].”
A Code Analysis Tool to Help Students in the Age of Generative AI
Dosaru, D.-F., Simion, D.-M., Ignat, A.-H., Negreanu, L., & Olteanu, A.-C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72312-4_31 · Cited by 3
Ha, T. (2024) · IEEE Access · doi:10.1109/access.2024.3488903 · Cited by 3
“Prominent application domains include finance [9], medicine [10], writing [11], and code generation [12].”
BackgroundUsing Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values
He, R., Zhang, L., Lyu, M., Lyu, L., & Xue, C. (2025) · Aerospace · doi:10.3390/aerospace12060498 · Cited by 3
Self-citationCoding Malware in Fancy Programming Languages for Fun and Profit
Apostolopoulos, T., Koutsokostas, V., Totosis, N., Patsakis, C., & Smaragdakis, G. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726506 · Cited by 3
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?Preprint
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · Qeios · doi:10.32388/sbl97o · Cited by 3
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3
“To rigorously benchmark the correctness of LLM-generated code, EvalPlus [Liu et al., 2023] extends the test cases of the original HumanEval by 80 × to build HumanEval + .”
InfluentialSelf-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3
NoviCode: Generating Programs from Natural Language Utterances by Novices
Mordechai, A. A., Goldberg, Y., & Tsarfaty, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00694 · Cited by 4
“Some works (Liu et al., 2023) mitigate this using an automatic test input generation engine.”
MethodEvaluating Correct-Consistency and Robustness in Code-Generating LLMs
Honarvar, S. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988971 · Cited by 2
“Although these models have become increasingly effective, they generate incorrect code [3]–[6], which poses significant challenges to their safe and reliable integration into mainstream software engineering, particularly in safety-critical environments where correctness and robustness are essential.”
Metadata
- Authors
- Liu, J., Xia, C. S., Wang, Y., & Zhang, L.
- ORCID
- Xia, Chunqiu Steven
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 21558-21572
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133198585
- Cited by
- 384
- References
- 0
- Citations by year
- 2026: 143 · 2025: 116 · 2024: 102 · 2023: 23
- FWCI
- 98.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2131943, 2141474)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, ChatGPT, program correctness, program verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
