CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
Abstract
Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks (e.g., HumanEval and AiXBench) are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios (i.e., code generation for real settings of open source or proprietary code).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT is the most widely adopted LLM in code generation in the literature.
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
From An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2025) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
Self-citationGPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
What is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47 · see citing works
AixBench: A Code Generation Benchmark Dataset
Hao, Y., Li, G., Liu, Y., Miao, X., Zong, H., Jiang, S., Liu, Y., & Wei, H. (2022) · arXiv · doi:10.48550/arxiv.2206.13179 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251 · see citing works
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
Self-citationSelf-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256 · see citing works
Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Liu, M., Pinckney, N., Khailany, B., & Ren, H. M. (2023) · doi:10.1109/iccad57390.2023.10323812 · Cited by 253 · see citing works
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120 · see citing works
LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
Zhang, Z., Wang, C., Wang, Y., Shi, E., Ma, Y., Zhong, W., Chen, J., Mao, M., & Zheng, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728894 · Cited by 131 · see citing works
Self-citationClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification
Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., & Wang, Q. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660810 · Cited by 74 · see citing works
Self-citationBugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70 · see citing works
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 99 · see citing works
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43 · see citing works
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 47 · see citing works
Wickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 48 · see citing works
Nazzal, M., Khalil, I. M., Khreishah, A. A., & Phan, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690298 · Cited by 33 · see citing works
SALLM: Security Assessment of Generated Code
Siddiq, M. L., Santos, J. C. S., Devareddy, S., & Muller, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694934 · Cited by 49 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
On the Effectiveness of LLM-as-a-Judge for Code Generation and Summarization
Crupi, G., Tufano, R., Velasco, A., Mastropaolo, A., Poshyvanyk, D., & Bavota, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586082 · Cited by 33 · see citing works
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23 · see citing works
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 34 · see citing works
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40 · see citing works
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43 · see citing works
Self-citationArtificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 32 · see citing works
Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?
Wang, G., Sun, Z., Ye, S., Gong, Z., Chen, Y., Zhao, Y., Liang, Q., & Hao, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771933 · Cited by 26 · see citing works
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 22 · see citing works
The Impact of Prompt Programming on Function-Level Code Generation
Khojah, R., de Oliveira, F. D., Mohamad, M., & Leitner, P. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3587794 · Cited by 48 · see citing works
Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
Cotroneo, D., Improta, C., & Liguori, P. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00035 · Cited by 20 · see citing works
An Empirical Study on the Code Refactoring Capability of Large Language Models
Cordeiro, J., Noei, S., & Zou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3801158 · Cited by 17 · see citing works
Development in times of hype: How freelancers explore Generative AI?
Dolata, M., Lange, N., & Schwabe, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639111 · Cited by 18 · see citing works
AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code
Solovyeva, L., Weidmann, S., & Castor, F. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00012 · Cited by 15 · see citing works
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3657432 · Cited by 188 · see citing works
Self-citationFranc: A Lightweight Framework for High-Quality Code Generation
Siddiq, M. L., Casey, B., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00020 · Cited by 27 · see citing works
Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models
Wang, Y., Jiang, T., Liu, M., Chen, J., Mao, M., Liu, X., Ma, Y., & Zheng, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715749 · Cited by 21 · see citing works
Self-citationMa, L., Yang, W., Li, Y., Fei, B., Zhou, M., Li, S., Jiang, S., Xu, B., & Xiao, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3749840 · Cited by 17 · see citing works
Self-citationRMCBench: Benchmarking Large Language Models' Resistance to Malicious Code
Chen, J., Zhong, Q., Wang, Y., Ning, K., Liu, Y., Xu, Z., Zhao, Z., Chen, T., & Zheng, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695480 · Cited by 12 · see citing works
Weyssow, M., Kamanda, A., Zhou, X., & Sahraoui, H. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3736407 · Cited by 31 · see citing works
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Improta, C., Tufano, R., Liguori, P., Cotroneo, D., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00056 · Cited by 11 · see citing works
ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
Xue, P., Wu, L., Yang, Z., Wang, C., Li, X., Zhang, Y., Li, J., Jin, R., Pei, Y., Shen, Z., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728940 · Cited by 15 · see citing works
User Centric Evaluation of Code Generation Tools (Invited Paper)
Miah, T., & Zhu, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00022 · Cited by 9 · see citing works
ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
Feng, J., Liu, J., Gao, C., Chong, C. Y., Wang, C., Gao, S., & Xia, X. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695552 · Cited by 9 · see citing works
One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code
Xu, J., Zhang, H., Yang, Y., Yang, L., Cheng, Z., Lyu, J., Liu, B., Zhou, X., Bacchelli, A., Chiam, Y. K., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748506 · Cited by 14 · see citing works
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far are We?
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00064 · Cited by 11 · see citing works
An Overview on Large Language Models
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_1 · Cited by 8 · see citing works
The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
Siddiq, M. L., Dristi, S., Saha, J., & Santos, J. C. S. (2024) · IEEE Working Conference on Source Code Analysis and Manipulation · doi:10.1109/scam63643.2024.00028 · Cited by 26 · see citing works
Νικολακόπουλος, Α., Λίτκε, Α., Psychas, A., Veroni, E., & Varvarigou, T. A. (2025) · IEEE Access · doi:10.1109/access.2025.3556973 · Cited by 10 · see citing works
Xiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7 · see citing works
Benchmarking large language models for automated labeling: The case of issue report classification
Colavito, G., Lanubile, F., & Novielli, N. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107758 · Cited by 8 · see citing works
Chen, Z., & Jiang, L. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00068 · Cited by 7 · see citing works
Do Code LLMs Understand Design Patterns?
Pan, Z., Song, X., Wang, Y., Cao, R., Li, B., Li, Y., & Liu, H. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00031 · Cited by 10 · see citing works
Self-citationUnseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
Zhang, Y., Xie, Y., Lit, S., Liu, P., Wang, C., Jia, Z., Huang, X., Song, J., Luo, C., Zheng, Z., et al. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00082 · Cited by 7 · see citing works
Tihanyi, N., Bisztray, T., Dubniczky, R. A., Tóth, R., Borsos, B., Cherif, B., Jain, R., Muzsai, L., Ferrag, M. A., Marinelli, R. A., et al. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825051 · Cited by 7 · see citing works
Enhancing Project-Specific Code Completion by Inferring Internal API Information
Deng, L., Ren, X., Ni, C., Liang, M., Lo, D., & Liu, Z.-X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3592823 · Cited by 13 · see citing works
Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs
Ngassom, S. K., Dakhel, A. M., Tambon, F., & Khomh, F. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664772 · Cited by 8 · see citing works
Addressing Data Leakage in HumanEval Using Combinatorial Test Design
Bradbury, J. S., & More, R. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989022 · Cited by 6 · see citing works
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11 · see citing works
Self-citationStatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis
Song, X., Lee, L., Xie, K., Liu, X., Deng, X., & Hong, Y. (2026) · Scientific Data · doi:10.1038/s41597-026-06731-4 · Cited by 6 · see citing works
Ashraf, H., Danish, S. M., Leivadeas, A., Otoum, Y., & Sattar, Z. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391072 · Cited by 5 · see citing works
BePilot: An AI Programming Assistant for Compiler Backend Development
Zhong, M., Sun, X., Lv, F., Wang, L., Geng, H., Qiu, L., Cui, H., & Feng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764585 · Cited by 5 · see citing works
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
Self-citationOmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution
Guo, L., Tao, W., Jiang, R., Wang, Y., Chen, J., Liu, X., Ma, Y., Mao, M., Zhang, H., & Zheng, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728871 · Cited by 4 · see citing works
Self-citationEvaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4 · see citing works
IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
Nekrasov, R., Fossati, S., Kumara, I., Tamburri, D. A., & van den Heuvel, W. J. A. M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3817608 · Cited by 6 · see citing works
Evaluating LLMs for Multimodal GUI Test Generation in Android Applications
Fagundes, N., & Teixeira, L. (2025) · Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST) · doi:10.5753/sast.2025.13852 · Cited by 4 · see citing works
ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00015 · Cited by 4 · see citing works
A State-of-the-Art of Intelligent Problem-Oriented Low-Code Systems
Sá, D., Lobo, A., Cunha, J. C., Duarte, R. J. D. S., Guimarães, T. A. S., & Santos, M. F. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.03.148 · Cited by 4 · see citing works
Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?
Vitale, A., Mastropaolo, A., Oliveto, R., Penta, M. D., & Scalabrino, S. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00033 · Cited by 8 · see citing works
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3 · see citing works
Less Is More: Failing Test Generation with Large Language Models
Li, T.-O., Yan, X., Zong, W., Wang, Y., Tian, H., Xu, C., Wang, Y., Cheung, S.-C., & Kramer, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3793675 · Cited by 3 · see citing works
Starace, L. L. L. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112989 · Cited by 3 · see citing works
CodeS : Natural Language to Code Repository via Multi-Layer Sketch
Zan, D., Yu, A., Liu, W., Chen, D., Shen, B., Yao, Y., Li, W., Chen, X., Gong, Y., Guan, B., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3768577 · Cited by 15 · see citing works
Self-citationWhere Is Self-admitted Code Generated by Large Language Models on GitHub?
Yu, X., Liu, L., Hu, X., Liu, J., & Xia, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00047 · Cited by 3 · see citing works
Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values
He, R., Zhang, L., Lyu, M., Lyu, L., & Xue, C. (2025) · Aerospace · doi:10.3390/aerospace12060498 · Cited by 3 · see citing works
Investigating Execution-Aware Language Models for Code Optimization
Menna, F. D., Traini, L., Bavota, G., & Cortellessa, V. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00029 · Cited by 3 · see citing works
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y.-F., & Zhang, T.-Y. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00165 · Cited by 11 · see citing works
An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?Preprint
Suh, H., Tafreshipour, M., Li, J., Bhattiprolu, A., & Ahmed, I. (2025) · Qeios · doi:10.32388/sbl97o · Cited by 3 · see citing works
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
Di, Y., & Zhang, T. (2025) · arXiv · doi:10.48550/arxiv.2505.07768 · Cited by 3 · see citing works
Improving FIM Code Completions via Context & Curriculum Based Learning
Sagtani, H., Mehrotra, R., & Liu, B. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703563 · Cited by 2 · see citing works
Collaboration with Generative AI to improve Requirements Change
Kong, Y., Zhang, N., Duan, Z., & Yu, B. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104013 · Cited by 3 · see citing works
CG-Bench: Can Language Models Assist Call Graph Construction in the Real World?
Yuan, T., Zhang, W., Chen, D., & Wang, J. (2025) · ACM SIGPLAN International Workshop on Language Models and Programming Languages (LMPL) · doi:10.1145/3759425.3763379 · Cited by 2 · see citing works
CatCoder : Repository-Level Code Generation with Relevant Code and Type Context
Pan, Z., Hu, X., Xia, X., & Yang, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3779217 · Cited by 9 · see citing works
SolContractEval: A Benchmark for Evaluating Contract-Level Solidity Code Generation
Ye, Z., Chen, T., SHAO, Z., Bao, L., Yang, X., & Liu, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00235 · Cited by 2 · see citing works
PrefGen: A Preference-Driven Methodology for Secure Yet Gas-Efficient Smart Contract Generation
Peng, Z., Yin, X., Zhou, Z., Ying, C., Ni, C., & Luo, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00034 · Cited by 6 · see citing works
Li, J., Deng, H., Zhang, Y., Zhang, K., Shao, T., Zhao, T., Wang, W., Jin, Z. G., Li, G., Liu, Y., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797106 · Cited by 2 · see citing works
“No Free Lunch” when using Large Language Models to Verify Self-Generated Programs
Zilberman, S., & Cheng, H. C. B. (2024) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw60967.2024.00018 · Cited by 2 · see citing works
RepoSim: Evaluating Prompt Strategies for Code Completion via User Behavior Simulation
Peng, C., Wu, Q., Liu, J., Liu, J., Jiang, B., Xu, M., Wang, Y., Liu, X., & Yang, P. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695299 · Cited by 2 · see citing works
YABLoCo: Yet Another Benchmark for Long Context Code Generation
Valeev, A., Garaev, R., Lomshakov, V., Piontkovskaya, I., Ivanov, V. V., & Adewuyi, I. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00016 · Cited by 2 · see citing works
Empowering AI to Generate Better AI Code: Guided Generation of Deep Learning Projects with LLMs
Chen, X., Jiao, M., Gu, X., & Shen, B. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00175 · Cited by 2 · see citing works
Self-citationCan Language Models Pretend Solvers? Logic Code Simulation with LLMs
Chen, M., Li, G., Wu, L.-I., Liu, R., Su, Y., Chang, X., & Xue, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0602-3_6 · Cited by 3 · see citing works
Yu, H., Shen, B., Zhang, J. Y., Lin, S., Li, L., Liang, G., Li, Y., Wang, Q., & Xie, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725213 · Cited by 1 · see citing works
Self-citationRethinking Software Development Considering Collaboration with AI Assistants
Donato, B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00045 · Cited by 1 · see citing works
Contextual Augmented Multi-Model Programming (CAMP): A Local-Cloud Copilot Solution
Wang, Y., Guo, S., & Tan, C. W. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00122 · Cited by 1 · see citing works
Toward Non-Expert Customized Congestion Control
Zhang, M., Bagheri, H., & Xu, L. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11160790 · Cited by 1 · see citing works
CFCEval: Evaluating Security Aspects in Code Generated by Large Language Models
Cheng, C., & Yang, J. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00021 · Cited by 1 · see citing works
FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification
Zhao, Q., Zhang, L., Liu, F., Lian, X., Meng, Q., Jiao, Z., Zhou, Z., Li, J., & Shi, L. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00190 · Cited by 1 · see citing works
Mundhra, Y., Valk, M., & Izadi, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00287 · Cited by 1 · see citing works
RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation
Yu, X., Chen, H., Liu, L., Hu, X., Keung, J., & Xia, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00248 · Cited by 1 · see citing works
HAFix: history-augmented large language models for bug fixing
Shi, Y., Bangash, A. A., Fallahzadeh, E., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10833-3 · Cited by 2 · see citing works
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 1 · see citing works
SWE-Bench+: Enhanced LLM Coding Benchmark
Xue, H., Aleithan, R., Enan, N., Uddin, G., & Wang, S. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814924 · Cited by 1 · see citing works
Revolutionizing Software Development: Autonomous Software Evolution
Mlinarić, D., Dončević, J., Brčić, M., & Botički, I. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569871 · Cited by 1 · see citing works
- Authors
- Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., & Xie, T.
- Affiliations
- Peking University (CN)
- Huawei Technologies (China) (CN)
- ORCID
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-02-06
- Pages
- 1-12
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4391558462
- Cited by
- 140
- References
- 7
- Citations by year
- 2026: 42 · 2025: 69 · 2024: 28 · 2023: 1
- FWCI
- 52.93
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- Natural Science Foundation of China (No.62161146003)
- National key research and development program Project (2021YFF0704202)
- Tencent Foundation/XPLORER PRIZE
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code generation benchmarks, HumanEval, codices, CodeGen, open-source code
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
