The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Open accessAbstract
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Similarly, MBPP+ offers 35 times more test cases than the original MBPP.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the second case, a finite test suite necessarily under-approximates an executable's full specification (Liu et al., 2023; Le Goues et al., 2015).”
From ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026) · cites this paper as Liu et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“To further ensure accurate evaluation, EvalPlus (Liu et al., 2023) extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions in HumanEval.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 61
- T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-5, 2022.
- L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. · arXiv
- S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. In Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II, pages 386-407. Springer, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209-224, 2008.
- F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.
- S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725-741. IEEE, 2015.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023.
- Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE), 2024.
- U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM (JACM), 45(4):634-652, 1998.
- D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence, pages 202-222. Elsevier, 1981.
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not., 46(1):317-330, jan 2011.
- S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1-119, 2017.
- T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 21st USENIX Security Sym- posium (USENIX Security 12), pages 445-458, Bellevue, WA, Aug. 2012. USENIX Association.
- M. Ivanković, G. Petrović, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 955-963, 2019.
- S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, 2018.
- N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020, 2023. · arXiv
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. · arXiv
- A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In International Conference on Learning Representations, 2018.
- J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385-394, 1976.
- M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511, 2020. · arXiv
- K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pages 348-370. Springer, 2010.
- Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, 2022.
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages, 6(OOPSLA1):1-26, Apr. 2022.
- Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355-1367. IEEE, 2023.
- N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 65-79, 2021.
- S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. · arXiv
- Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. · arXiv
- Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
- W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
- B. Meyer. Applying'design by contract'. Computer, 25(10):40-51, 1992.
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32-44, 1990.
- G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83-94, 2000.
- E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint, 2023.
- E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2):58-62, 2005.
- OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
- G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice, pages 163-171, 2018.
- G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
- B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773, 2021. · arXiv
- K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev), pages 157-157. IEEE, 2016.
- D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In IJCAI, volume 75, pages 260-267, 1975.
- A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pages 246-256, 2014.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 241-252, 1969.
- Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023. · arXiv
- Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint arXiv:2309.00608, 2023. · arXiv
- D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1-25, 2020.
- C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE), 2024.
- C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
- C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program re- pair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-971, 2022.
- F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.
- T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
- L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 170-179. IEEE, 2011.
- Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, 2023. · arXiv
Cited by
Sorted by citations
Items 201–300 of 425
CoCoNUT: Structural Code Understanding does not fall out of a tree
Beger, C., & Dutta, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00021 · Cited by 3
“Additionally, Liu et al. introduced EvalPlus [10], which expands the number of tests for HumanEval tasks, revealing that model performance was previously overestimated.”
Alter, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94569-4_2 · Cited by 2
The fools are certain; the wise are doubtful: Exploring LLM confidence in code completion
Kotti, Z., Dritsa, K., Spinellis, D. D., & Λουρίδας, Π. (2026) · Automated Software Engineering · doi:10.21203/rs.3.rs-7302683/v1 · Cited by 3
Song, A.-Y., & Azman, A. (2025) · Symmetry · doi:10.3390/sym17101660 · Cited by 2
Let’s Take Esoteric Programming Languages Seriously
Singer, J., & DRAPER, S. W. (2025) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3759429.3762632 · Cited by 2
“In his HOPL presentation of this paper, Peyton Jones explains how Haskell was intended to ‘avoid success at all costs’ which means the relatively small user-base are happy for the language to be ‘nimble’ in terms of feature updates.”
Evaluating the effectiveness of LLM-based interoperability
Falcão, R., Schweitzer, S., Siebert, J., Calvet, E., & Elberzhager, F. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773251 · Cited by 2
Hallucination detection in LLM code generation: A sampling-based consensus verification approach
Huang, T., Ren, Z., Huang, Y., Chen, X., Liu, Y., & Zheng, Z. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00605-0 · Cited by 4
“6 and 7, the average time cost of HalluCodeDetector-METEOR, HalluCodeDetector-BLEU and HalluCodeDetector-CodeBLEU for hallucination detection is 0.008 seconds, and the time cost of HalluCodeDetector-MRCM is 0.016 seconds, which is longer than the former three.”
InfluentialAutomated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithms
Taherkhani, H., Sepidband, M., Pham, H. V., Wang, S., & Hemmati, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3805704 · Cited by 15
“The goal of the EPiC framework is to optimize the fitness function F based on the provided test cases T by identifying the optimal input 𝑥 within the prompt space X .”
Impact of GPU Architecture and VRAM on Quantized LLM Inference for Code Deobfuscation
dos Santos, I. T. F., Barboza, R. D. S., Almeida, F. P. D., de Araújo, C. S., & Silva, J. D. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11468042 · Cited by 2
“As a result, the adoption of LLMs is expanding rapidly, driven by diverse applications including classification [4], code generation [5], translation [6], healthcare [7], and cybersecurity tasks such as code deobfuscation [8].”
SymLoc: Symbolic Localization of Hallucination across HaluEval and TruthfulQA
Lamba, N., Tiwari, S., & Gaur, M. (2025) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3799830.3799850 · Cited by 2
“To ensure consistency and eliminate sampling variability, we use each model’s recommended decoding settings (typically low or zero temperature) [16].”
AutoEmbed: LLM-driven Automated Software Development for Generic Embedded IoT Systems
Yang, H., Li, M., Han, M., Li, Z., & Xu, W. (2026) · doi:10.1145/3774906.3800466 · Cited by 2
Zhang, W., Jiang, B., Yuhong, F., Koziolek, A., Hebig, R., & Strüber, D. (2026) · Software & Systems Modeling · doi:10.1007/s10270-026-01402-9 · Cited by 3
Edge AI Inference in Heterogeneous Constrained Computing: Feasibility and Opportunities
Morabito, R., Tatipamula, M., Tarkoma, S., & Chiang, M. (2023) · IEEE International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD) · doi:10.1109/camad59638.2023.10478416 · Cited by 2
“Ensuring the reliability of dynamically generated code [21], maintaining security and integrity during real-time adaptations, and verifying the consistency and correctness of such on-the-fly solutions emerge as critical research avenues.”
BackgroundExploring Advanced Methodologies in Security Evaluation for Large Language Models
Huang, J., Zhang, J., Wang, Q., Han, W., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-4519-7_10 · Cited by 2
“No Free Lunch” when using Large Language Models to Verify Self-Generated Programs
Zilberman, S., & Cheng, H. C. B. (2024) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw60967.2024.00018 · Cited by 2
“Development teams seeking to use LLMs in their workflow may wish to explore implementing more sophisticated frameworks (e.g., EvalPlus [35], TeCO [34]) that can refine, constrain, and specialize LLMs for software testing.”
MethodJacinto, M. V. G., Oliveira, L. H. L. D., Rodrigues, T. C., de Medeiros, G. C., Medeiros, D. R., Silva, M. A., de Montalvão, L. C., González, M., & de Almeida, R. V. (2024) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/221864-ms · Cited by 4
Improving Retrieval-Augmented Code Comment Generation by Retrieving for Generation
Lu, H., & Liu, Z. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00040 · Cited by 3
“To demonstrate the feasibility of our proposed framework on large language models, we evaluate CodeLlama-7b-Instruct [45], which is widely used by previous work for code-related tasks [46], on the PCSD dataset with in-context learning.”
Question Selection for Multimodal Code Search Synthesis Using Probabilistic Version Spaces
Wu, J., Jiang, Y., Wei, L., Xu, C., Cheung, S.-C., & Xu, C. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3565387 · Cited by 2
Studying How Configurations Impact Code Generation in LLMs: The Case of ChatGPT
Donato, B., Mariani, L., Micucci, D., & Riganelli, O. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00055 · Cited by 15
“Similar results have been reported in Liu et al. [16], where again low values of temperature and top-p are suggested to be beneficial for the correctness of the responses.”
Artemis AI: Multi-LLM Framework for Code Optimisation
Giavrimis, R., Basios, M., Wu, F., Kanthan, L., & Bauer, R. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00057 · Cited by 2
“However, the code generated by these models often suffers from quality issues [12].”
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“Coding : LiveCodeBench (Jain et al., 2024), HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2023), MBPP (Austin et al., 2021), MBPP+ (Liu et al., 2023), CRUXEval (Gu et al., 2024).”
Self-citationDoes Few-Shot Learning Help LLM Performance in Code Synthesis?
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.02906 · Cited by 3
“However, because the H UMAN E VAL + benchmark (Liu et al., 2024b) already uses GPT to generate its test cases, to prevent data leakage, the large pool of examples is obtained from H UMAN E VAL + benchmark.”
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 73
“Many benchmarks (Yu et al., 2024; Yin et al., 2023; Khan et al., 2023; Orlanski et al., 2023) have been woven to accurately assess code quality, functionality, and efficiency, such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), their upgraded version EvalPlus (Liu et al., 2023b).”
BackgroundSelf-citationLongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
RewardBench 2: Advancing Reward Model Evaluation
Malik, S., Pyatkin, V., Land, S., Morrison, J., Smith, N. A., Hajishirzi, H., & Lambert, N. (2025) · arXiv · doi:10.48550/arxiv.2506.01937 · Cited by 1
“Experimental Setup We evaluated 113 RMs, with a wide range of scores on R EWARD B ENCH 2, on BoN sampling over evaluations covering several domains: GSM8K [43], MATH [44], IFEval [45], AlpacaEval 2 [46], BigBenchHard (BBH) [47], PopQA [48], and HumanEval+[49].”
Enhancing Security Awareness and Education for LLMs
Panichella, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_18 · Cited by 1
Conducting Final Programming Exams with Auto-Grading and Code Evaluation Tools
Subirats, J. L., Luque-Polo, G., & Luque‐Baena, R. M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-75016-8_25 · Cited by 1
Specification-driven code translation powered by Large Language Models: How far are we?
Saha, S. K., Rabbi, F., Pham, T. M. T., Wang, S., & Yang, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112964 · Cited by 4
Zhang, Z., Mao, X., Wang, S., Yang, K., Zhang, T., Gao, F., & Zhang, X. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00027 · Cited by 1
Evidence selection via multi-aspect query diversification for cross-document relation extraction
Wang, X., Zhu, X., & Hu, W. P. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00952-6 · Cited by 1
“Therapidadvancementoflargelanguagemodels(LLMs)(Kamallooetal.,2023;Jakesch etal.,2023;Liuetal.,2023;Dingetal.,2025)exhibitsasubstantialexpansionoftheir intrinsicknowledge,alongsidenotableimprovementsinlogicalreasoningandjudgment capabilities.”
FuseApplyBench: Multilingual Benchmark for Trustworthy Code Edit Applying Task
Liang, M., Zhang, Q., Zuo, Z., Zheng, S., Chen, D., Jiang, W., & Li, Y. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732929 · Cited by 1
Revisiting SWE-Bench: On the Importance of Data Quality for LLM-Based Code Models
Aleithan, R. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00075 · Cited by 1
“EvalPlus [1] further improved benchmarks through automated and mutation-based testing, emphasizing the necessity of rigorous and comprehensive evaluations.”
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Random Perturbation Attack on LLMs for Code Generation
Peng, Q., Zhang, C., Mangal, R., Păsăreanu, C. S., & Jia, L. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00052 · Cited by 1
“In evaluating LLMs for code generation, HumanEval[2] and MBPP (Mostly Basic Python Problems[1]) are widely used benchmark datasets[10].”
Self-Regeneration: Step-Level Regeneration Improves the LLM's Causal Reasoning Ability
Jiang, W., Zhao, S., & Dai, H.-W. (2025) · International Conference on Computer Supported Cooperative Work in Design · doi:10.1109/cscwd64889.2025.11033548
Code like an economist: Analyzing LLMs’ code generation capabilities in economics and finance
Song, Z., Wu, S., & Zhou, Y. (2025) · Finance research letters · doi:10.1016/j.frl.2025.108540 · Cited by 1
Evaluation and Improvement of Test Selection for Large Language Models
Quan, L., Jin, W., Hu, Q., Cordy, M., Huang, Y., Ma, L., & Li, X. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70057 · Cited by 4
“Liu et al. [32] propose a new code synthesis evaluation framework EvalPlus to measure the correctness of code generated by LLMs. Based on this framework, they found that the existing benchmark HumanEval is not rigorous enough.”
Ethical AI prompt recommendations in large language models using collaborative filtering
Nelson, J., Baimagambetov, A., Avgerinakis, K., & Polatidis, N. (2025) · International Journal of Parallel Emergent and Distributed Systems · doi:10.1080/17445760.2025.2573086 · Cited by 2
“Meanwhile, other research [13, 14] has gone beyond analysing the code itself, focusing instead on the evaluation conditions and metrics used to assess the performance of LLM-generated code.”
He, Y., Liu, J., Wen, W., Guo, Y., Chen, H., Ma, Y., Ke, W., & Wang, P. (2026) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2026.3654446 · Cited by 1
“W ITH the rise of large language models (LLMs), remarkable progress has been made across various downstream tasks, including open-domain question answering [1], code generation [2], and machine translation [3].”
Self-citationRa, D., Lee, D., & Lee, S. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.118157 · Cited by 1
Regression Accumulation in Multi-turn LLM Programming Conversations
Huang, A., Ma, L., Tahir, A., Zhang, Q., Xiao, L., & Xiao, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834357 · Cited by 1
Alfonseca, M., & Colino, A. M. (2026) · Academia AI and Applications · doi:10.20935/acadai8228 · Cited by 1
Wang, Q., Shen, A., Zeng, Y., Feng, D., Zhang, X., Ding, B., & Wang, H. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132870 · Cited by 1
Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localization
Caumartin, G., & Melo, G. (2026) · doi:10.1145/3786161.3788460 · Cited by 1
Shen, J., Peng, Z., & Owen, L. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00085 · Cited by 1
“Instead, the remaining errors clustered around high-level functional omissions [15], along four dimensions: • Local Logic Failures: Issues solvable through code review (CRUD, validation rules) • External Integration Failures: Requiring API docs, service configs • Domain Knowledge Failures:…”
An Automated Methodology for Generating Labeled Datasets of Semantic Errors in Code
Kassem, M., Ribeiro, F., & Nadi, S. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788730 · Cited by 1
Setup AGent (SAG): A Dual-Model LLM Agent for Autonomous End-to-End Java Project Configuration
Wei, C., Zhao, G., Li, X., Ye, B., & Xiao, L. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786818 · Cited by 1
“LLMs have demonstrated repository-level capabilities in issue resolution [9], code understanding [14], and documentation generation [6, 11, 12, 15].”
Yu, L., Wang, P., Zhang, J., Wang, X., Xu, J., Yang, L., Deng, C., Ma, J., & Zhang, F. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794803 · Cited by 3
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Al-Kaswan, A., Spieß, C. K., Dévanbu, P., Arie Van Deursen, A., & Izadi, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793341 · Cited by 1
“Of the 50 top performing models on the EvalPlus benchmark [29], only two have open training data, and both are instruction tuned versions of StarCoder2.”
ATeam: Governance-Aware LLM-Assisted Software Sustaining Engineering for Enterprise Systems
Vella, S., Ferworn, A., & Sharieh, M. (2026) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet65726.2026.11633274 · Cited by 1
“There is now an evolution toward agentic software engineering, in which AI agents, acting autonomously, plan and execute complex tasks [14] [15].”
Write Your Own Code Checker: An Automated Test-Driven Checker Development Approach with LLMs
Liu, J., Xie, Y., Yan, J., Huang, J., Yan, J., & Zhang, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764560 · Cited by 7
“The smallest model, Llama3.”
Self-citationEvaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing
Li, Z., Li, Z., Xiao, K., & Li, X. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429345 · Cited by 2
“Besides, EvalPlus [32] integrated existing benchmarks and added automatically generated test inputs, along with LLM-based then used for differential tests.”
MethodGrahn, D., Chen, L., & Zhang, J. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00219 · Cited by 1
“Despite the recent advances in LLMs, studies have indicated that they still struggle to produce accurate, let alone safe, code [15].”
BackgroundParaCkpt: Heterogeneous Multi-Path Checkpointing Mechanism for Training Deep Learning Models
Wang, S., Cao, Q., Zhou, K., Xu, J., Guo, Z., & Guo, J. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00036 · Cited by 1
“Large deep learning models have been extensively adopted in many applications including language processing [1], [2], computer vision [3] and code generation [4], [5].”
Towards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Framework
Wang, C., Chen, Z., Li, T., Zhang, Y., & Liu, Y. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00017 · Cited by 8
CODEMORPH: Mitigating Data Leakage in Large Language Model Assessment
Rao, H., Zhao, Y., Zhu, W., Xiao, L., Wang, M., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00081 · Cited by 1
“Liu et al. [25] expanded the test cases of HumanEval, increasing the difficulty for LLMs to pass the tests.”
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v1
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2024) · Preprints.org · doi:10.20944/preprints202306.0363.v5 · Cited by 1
Atkinson, C. F. (2024) · Research Square · doi:10.21203/rs.3.rs-4007725/v1 · Cited by 1
Prompt-Based Building Code Information Transformation Method for Automated Compliance CheckingPreprint
Yang, F., & Zhang, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4819834 · Cited by 1
A Systematic Literature Review and Multi-Stakeholder Ethical Framework for Social ChatbotsPreprint
Piispanen, J.-R., Myllyviita, T., Vakkuri, V., & Rousi, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5220728 · Cited by 1
Wang, T. D., Murphy, S. N., Castro, V. M., & Klann, J. G. (2025) · medRxiv · doi:10.1101/2025.04.17.25325962 · Cited by 1
“[11 – 13] Microsoft’s GitHub Copilot, for instance, is a tool designed to support programmers by leveraging OpenAI’s Codex, which was trained on a vast collection of publicly available code, including GitHub repositories.”
PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Woo, S., Kim, H., Shim, S., Jo, M., Jeong, H., Lee, J., Kim, J., Lee, S., Ryu, J., Park, B., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12029
“Agentic AI enables complex task solving via structured reasoning, planning, and tool interaction (Yao et al., 2022; Schick et al., 2023; Shinn et al., 2023; Wei et al., 2022; Wang et al., 2023).”
Jiang, J., Shen, J., Kim, S., Yoo, K. M., Kim, J., & Kim, S. (2026) · arXiv · doi:10.48550/arxiv.2603.05863 · Cited by 8
“…benchmarks, ranging from foundational programming challenges such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and EvalPlus (Liu et al., 2023), to significantly more complex and competitive programming problems found in BigCodeBench (Zhuo et al., 2024), Live-CodeBench (Jain…”
ProgramBench: Can Language Models Rebuild Programs From Scratch?
Yang, J., Lieret, K., Ma, J., Thakkar, P., Pedchenko, D., Sootla, S., McMilin, E., Yin, P., Hou, R., Synnaeve, G., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.03546 · Cited by 84
“For the second case, a finite test suite necessarily under-approximates an executable’s full specification (Liu et al., 2023; Le Goues Rust(107) Other (2) C/C++(45) Figure 3 Distribution of programming languages across ProgramBench task instances.”
Life After Benchmark Saturation: A Case Study of CORE-Bench
Nadgir, N., Kapoor, S., Liu, K., Kirgis, P., Orona, M., Rabanser, S., Bayer, T., Shetty, A., Ling, Y., Chan-Sew, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.26158
“The prevailing response has been to retire these benchmarks in favor of more difficult successors; for example, ARC-AGI 1 progressing to ARC-AGI 2 and 3, MMLU to MMLU-Pro, HumanEval to HumanEval+, and SWE-bench to SWE-bench Pro [12, 21, 53, 35, 15].”
Zhou, C., Zhang, X., Song, D., Chen, X., Gu, W., Ma, H., Tian, Y., Zhang, M., & Hu, L. (2025) · arXiv · doi:10.48550/arxiv.2502.09183 · Cited by 18
“7B and RefineCoder-QW-7B achieve average pass@1 improvements of 2.4 and 3.0, respectively, on the HumanEval (+), MBPP (+), LiveCodeBench, and the BigCodeBench-hard benchmark (Chen et al., 2021; Austin et al., 2021; Liu et al., 2023; Jain et al., 2024; Zhuo et al., 2024).”
InfluentialHow to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…math (Tang et al., 2024); (e) code (Hu-manEval (Chen et al., 2021), LiveCodeBench (Jain et al., 2025), MBPP (Tao et al., 2024), HumanEval plus (Liu et al., 2023), MBPP plus (Liu et al., 2023), HumanEval cn (Peng et al., 2024), HumanEval fim (Bavarian et al., 2022) and CruxEval (Gu et al.,…”
CLEVER: A Curated Benchmark for Formally Verified Code Generation
Thakur, A., Lee, J., Tsoukalas, G., Sistla, M., Zhao, M., Zetzsche, S., Durrett, G., Yue, Y., & Chaudhuri, S. (2025) · arXiv · doi:10.48550/arxiv.2505.13938
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric—since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
Foster, D., & Germonprez, M. (2026) · Information Systems Journal · doi:10.1111/isj.70060
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
“HumanEval [8], HumanEval+ [34], MBPP [1], MBPP+ [34] Using strong LLMs have become a tool for augmenting existing benchmarks with more difficult questions or answer choices [64] and re-evaluating benchmark quality [62], and may provide a cheap method for improving signal .”
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v2
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v3
Withdrawn: Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v4
Developing Time Series Forecasting Models with Generative Artificial IntelligencePreprint
Morales-García, J., Llanes, A., Arcas-Túnez, F., & Terroso-Sáenz, F. (2023) · Research Square · doi:10.21203/rs.3.rs-3626057/v1
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2024) · Preprints.org · doi:10.20944/preprints202306.0363.v6
Bekhouche, S. E., & Hadid, A. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595920
“…and been deployed in numerous downstream applications, encompassing diverse fields such as biology and medicine [2], [18], [20], [21], [27]–[29], education [11], [22], [31], [34], legislation [6], [10], [25], [40], computer programming [17], [19], [23], [24], [32], [39], and finance [3], [30], [37].”
BackgroundCodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation
Aggarwal, P., Chatterjee, O., Dai, T., Mohapatra, P., Paulovicks, B. D., Blancett, B., & Magalhaes, A. D. (2024) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud62652.2024.00052
“Restrictions apply. programming problems while MBPP+, which is a refined subset of the original MBPP dataset [8] by Evalplus [26], contains 399 python problems.”
BackgroundCommons-Based Memories: Programming Practices and Large Language Models
Depaz, P. (2024) · Memory studies review. · doi:10.1163/29498902-202400018
Inoshita, K. (2024) · SHS Web of Conferences · doi:10.1051/shsconf/202420403012
Processo de Migração de Aplicação Mainframe Cobol para Microsserviços Java na Nuvem
Florêncio, J. R., Bentes, C. B., & Castro, M. C. (2024) · doi:10.5753/eradrj.2024.4528
Predicting ChatGPT’s Ability to Solve Complex Programming Challenges
Ho, N., May, J., Ngo, B., Formato, J., Ngo, L., Ho, V. L., & Bui, H. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825139
Enhancing Code LLMs with Reinforcement Learning in Code Generation: A SurveyPreprint
He, Y., Shi, T., Li, Y., & Chen, Q. (2025) · Qeios · doi:10.32388/8g8tb2
Investigating LLMs Potential in Software Requirements Evaluation
Alsaedi, N., Alsaedi, A., Almaghathawi, A., Alshanqiti, M., & Siddiqi, A. A. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-80334-5_18
HinglishEval: Evaluating the Effectiveness of Code-Generation Models on Hinglish Prompts
Pawagi, M., Gupta, A., Rolla, S. R., & Saha, K. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-84391-4_2
“In general, several benchmarks have been proposed to evaluate the programming capabilities of LLMs in various aspects of software engineering [9, 6, 36, 5, 22, 14, 20].”
Enhancing Code Completion with Position Embedding of Abstract Syntax Tree
Li, J., Lu, J., & Zhu, L. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913308
“Despite significant advancements in code generation research, which can provide programmers with entire code segments as suggestions [7][8], token-level code completion remains crucial for users.”
Leveraging Generative AI for Accelerating Enterprise Application Development: Insights from ChatGPT
Rajbhoj, A., Sant, T., Somase, A., & Kulkarni, V. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00052
“Many researchers have discussed the significant impact of systematic interactions with ChatGPT and other LLMs on their effectiveness and outcomes [4, 5].”
Integrating GenAI in Advancing Game Product Management and Development
Pilaniwala, P. (2024) · International Conference on Parallel, Distributed and Grid Computing (PDGC) · doi:10.1109/pdgc64653.2024.10984420
“Lin et al. have published their findings on the evaluation of LLMs for code generation [45].”
Hybrid Equivalence/Non-Equivalence Testing
Sarker, L., & Bultan, T. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988990
Hort, M., & Moonen, L. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988963
“However, it has recently been shown that these test suites do not always cover all cases, allowing false negative submissions to slip through [7], [8].”
LCache: Log-Structured SSD Caching for Training Deep Learning Models
Wang, S., Xu, Z., Guo, Z., Sheng, J. L., Zhou, K., & Cao, Q. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992907
Assessing Code Synthesis In LLMS: CodeGeex and Codeium
Sheoran, S., & Kumar, D. (2025) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v12i5.563406
Evaluating Language Models for Computer Graphics Code Completion
Kels, J., Dahou, A. H., & Mathiak, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00017
“Similar analysis has been done [33] to give a solve rate per problem [6].”
Automated Repair of Cyber-Physical Systems
Valle, P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00060 · Cited by 1
“In addition, this algorithm includes a patch validation mechanism due to the inherent threats of the LLMs [19] (e.g., hallucination).”
Augmenting the Generality and Performance of Large Language Models for Software Engineering
Peña, F. C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00059 · Cited by 1
Advancing Large Language Models in Code Generation: Usaco Benchmark and Bug Mitigation Insights
Trentini, J., Liu, V., Peng, Y., & Zong, Z. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00057 · Cited by 1
“HumanEvalX [61] and MultiPL-E [62] extended the amount of available languages along with HumanEval+ to increase problem difficulty [63].”
AVALIAÇÃO DA QUALIDADE DE CÓDIGO UTILIZANDO GRANDES MODELOS DE LINGUAGEM
Martins, D. Q. R., & de Sá, S. L. (2025) · Revista Contemporânea · doi:10.56083/rcv5n7-003
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
Metadata
- Authors
- Liu, J., Xia, C. S., Wang, Y., & Zhang, L.
- ORCID
- Xia, Chunqiu Steven
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 21558-21572
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133198585
- Cited by
- 384
- References
- 0
- Citations by year
- 2026: 143 · 2025: 116 · 2024: 102 · 2023: 23
- FWCI
- 98.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2131943, 2141474)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, ChatGPT, program correctness, program verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
