The DOI matches its DataCite record, with a small difference in title or year; 1 of the 6 references it cites were found in a registry or an index, and 5 are not checked yet.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Open accessAbstract
Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering.”
From Code as Agent Harness (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“HumanEval [7]-developed by OpenAI for evaluating Codex-and other benchmarks [2,13,20] only consist of programming problems in a single language”
From CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“In code generation tasks, the smaller LLMs trained for the tasks are competitive in performance, and CodeGen-16B [141] is comparable in performance to ChatGPT using a larger parameter setting, reaching about a 78% match [125].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [141]
The cited source says what the claim attributes to it.
In the citing paper
“Previous studies (Chen et al., 2021;Nijkamp et al., 2022;Chen et al., 2022) have demonstrated the effectiveness of utilizing large pre-trained language models for code generation in a zero-shot completion manner, conditioned on the provided context.”
From RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023) · cites this paper as Nijkamp et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 39
- Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. Juice: A large scale distantly supervised dataset for open domain context-based code generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 5436-5446, 2019.
- Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics , 8:556-571, 2020.
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021. · arXiv
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397 , 2022. · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. · arXiv
- Alvin Cheung, Armando Solar-Lezama, and Samuel Madden. Optimizing database-backed applications with query synthesis. ACM SIGPLAN Notices , 48(6):3-14, 2013.
- Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 9052-9065, 2020.
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .
- Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp. 1536-1547, 2020.
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 , 2022. · arXiv
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. · arXiv
- Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices , 46(1):317-330, 2011.
- Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trends® in Programming Languages , 4(1-2):1-119, 2017.
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. URL https://openreview.net/forum?id= sD93GOzH3i5 .
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR , 2020. URL https://openreview.net/forum?id=rygGQyrFvH .
- Rajeev Joshi, Greg Nelson, and Keith Randall. Denali: A goal-directed superoptimizer. ACM SIGPLAN Notices , 37(5):304-314, 2002.
- John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning , pp. 5110-5121. PMLR, 2020.
- Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015. URL http://arxiv.org/abs/1412.6980 . · arXiv
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems , 32, 2019.
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 7871-7880, 2020.
- Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM , 14(3):151-165, 1971.
- Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021. · arXiv
- Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. · arXiv
- Pavel Panchekha, Alex Sanchez-Stern, James R Wilcox, and Zachary Tatlock. Automatically improving accuracy for floating point expressions. ACM SIGPLAN Notices , 50(6):1-11, 2015.
- Emilio Parisotto, Abdel rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. In ICLR (Poster) , 2017. URL https://openreview. net/forum?id=rJ0JwFcex .
- Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pp. 1310-1318. PMLR, 2013.
- Oleksandr Polozov and Sumit Gulwani. Flashmeta: A framework for inductive program synthesis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications , pp. 107-126, 2015.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:1-67, 2020.
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1-16. IEEE, 2020.
- Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices , 51(10):731-747, 2016.
- Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. ACM SIGARCH Computer Architecture News , 41(1):305-316, 2013.
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. · arXiv
- Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts , pp. 1-7, 2022.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pp. 5998-6008, 2017.
- Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. Codet5: Identifier-aware unified pretrained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 , 2021.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. · arXiv
Cited by
Sorted by citations
Items 101–158 of 158
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49
A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT
Yan, D., Gao, Z., & Liu, Z. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00096 · Cited by 47
From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining
Ren, X., Ye, X., Zhao, D., Xing, Z., & Yang, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00143 · Cited by 44
SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3543187 · Cited by 32
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 37
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
Fine-tuning and Utilization Methods of Domain-specific LLMsPreprint
Jeong, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02981 · Cited by 31
Optimizing Secure AI Lifecycle Model Management With Innovative Generative AI Strategies
Almagrabi, A. O., & Khan, R. A. (2024) · IEEE Access · doi:10.1109/access.2024.3491373 · Cited by 33
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
Peng, J., Cui, L., Huang, K., Yang, J., & Ray, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00009 · Cited by 84
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29
“More recently, pre-trained base code models [41, 34, 36, 52] have been built to perform accurate code completion.”
MethodLearning-based Widget Matching for Migrating GUI Test Cases
Zhang, Y., Zhang, W., Ran, D., Zhu, Q., Dou, C., Hao, D., Xie, T., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623322 · Cited by 42
Jin, Y., & Ma, J. (2024) · Sustainable Cities and Society · doi:10.1016/j.scs.2024.105940 · Cited by 33
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
Oh, S., Lee, K., Park, S., Kim, D., & Kim, H. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00046 · Cited by 27
Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
Paul, D. G., Zhu, H., & Bayley, I. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00019 · Cited by 34
A Multi-Expert Large Language Model Architecture for Verilog Code Generation
Nadimi, B., & Zheng, H. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691683 · Cited by 30
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43
LLM-based Processor Verification: A Case Study for Neuromorphic Processor
Xiao, C., Deng, Y., Yang, Z., Chen, R., Wang, H., Zhao, J., Dai, H., Wang, L., Tang, Y., & Xu, W. (2024) · Design, Automation, and Test in Europe · doi:10.23919/date58400.2024.10546707 · Cited by 26
Self-citationMOSS: An Open Conversational Large Language Model
Sun, T., Zhang, X., He, Z., Li, P., Cheng, Q., Liu, X., Yan, H., Shao, Y., Tang, Q., Zhang, S., et al. (2024) · Machine Intelligence Research · doi:10.1007/s11633-024-1502-8 · Cited by 48
Self-citationDelving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
Liu, S., Keung, J., Yang, Z., Liu, F., Zhou, Q., & Liao, Y. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00055 · Cited by 48
Search-Based LLMs for Code Optimization
Gao, S., Gao, C., Gu, W., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00021 · Cited by 64
Exploring Automated Assertion Generation via Large Language Models
Zhang, Q., Sun, W., Fang, C., Yu, B., Li, H., Yan, M. L., Zhou, J., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699598 · Cited by 24
Automated Test Case Repair Using Language Models
Yaraghi, A. S., Holden, D., Kahani, N., & Briand, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3541166 · Cited by 25
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“In contrast, we focus on adapting pre-trained LLMs (Chen et al., 2021b; Nijkamp et al., 2022; Tunstall et al., 2022; Xu et al., 2022; Fried et al., 2022) to performance optimization.”
BackgroundMethodInstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Pan, X., Li, E., Li, Q., Liang, S., Shan, Y., Zhou, K., Luo, Y., Wang, X., & Zhang, J. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00113 · Cited by 74
HLSPilot: LLM-based High-Level Synthesis
Xiong, C., Liu, C., Li, H., & Li, X. (2024) · doi:10.1145/3676536.3676781 · Cited by 50
Self-citationEspejel, J. L., Alassan, M. S. Y., Chouham, E. M., Dahhane, W., & Ettifouri, E. H. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100013 · Cited by 26
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
Non-Autoregressive Line-Level Code Completion
Liu, F., Fu, Z., Li, G., Jin, Z., Liu, H., Hao, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649594 · Cited by 22
Mostajabdaveh, M., Yu, T. T., Ramamonjison, R., Carenini, G., Zhou, Z., & Zhang, Y. (2024) · INFOR Information Systems and Operational Research · doi:10.1080/03155986.2024.2381306 · Cited by 22
Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715004 · Cited by 53
Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 87
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“Through large-scale pretraining more generally applicable code models have been developed (Nijkamp et al., 2022; 2023; Xu et al., 2022a; Christopoulou et al., 2022; Gunasekar et al., 2023; Li et al., 2023b; Bui et al., 2023; Scao et al., 2022a;b).”
MethodBenchmarking Causal Study to Interpret Large Language Models for Source Code
Rodriguez-Cardenas, D., Palacio, D. N., Khati, D., Burke, H., & Poshyvanyk, D. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00040 · Cited by 14
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“Additionally, we are connected to the line of code execution since we also leverage it as an important retrieval source. generation (Li et al., 2022; Nijkamp et al., 2022; Li et al., 2023b; Roziere et al., 2023; Wei et al., 2023).”
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3645056 · Cited by 12
Miceli Barone, A. V., Barez, F., Cohen, S. B., & Konstas, I. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.19 · Cited by 12
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2024) · arXiv · doi:10.48550/arxiv.2410.22339 · Cited by 8
A Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Miserendino, S., Wang, M., Patwardhan, T., & Heidecke, J. (2025) · arXiv · doi:10.48550/arxiv.2502.12115 · Cited by 5
“…et al., 2024), APPS, LiveCodeBench (Jain et al., 2024), NaturalCodeBench (Zhang et al., 2024), Big-CodeBench (Zhuo et al., 2024), MBPP, MathQA-Python (Austin et al., 2021), CodeXGlue (Lu et al., 2021), Al-phaCode (Li et al., 2022), CODEGEN (Nijkamp et al., 2023), and DS-1000 (Lai et al., 2022).”
Peng, Q., Chai, Y., & Li, X. (2024) · arXiv · doi:10.48550/arxiv.2402.16694 · Cited by 88
“…Prior efforts have predominantly concentrated on English-centric Python generation, evident in benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DSP (Chandel et al., 2022), MTPB (Nijkamp et al., 2023b), and DS-1000 (Lai et al., 2023).”
BackgroundLongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
On the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
Full Line Code Completion: Bringing AI to Desktop
Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.08704 · Cited by 8
“In addition to recent advancements in single-token code completion, there is a niche of multi-token completion that is being actively studied in academia [26]–[29] and employed in products [3]–[6].”
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“Despite all their success and remarkable capabilities in diverse sets of tasks [15, 40, 41], LLMs are largely static after their initial deployment phase, meaning that they successfully perform tasks learned during pre-or post-training, but are unable to continually acquire new capabilities beyond…”
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 73
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7
“Similarly, Code-Gen [51] adopts a decoder-only architecture, with a focus on multi-turn program synthesis and open-source availability.”
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. T. (2023) · arXiv · Cited by 112
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Ivanova, M., Zadorozhny, P., Levichev, R., Petrov, I., Pavel, A., Lopatin, I., Kutalev, A., & Babaev, D. (2026) · arXiv · doi:10.48550/arxiv.2606.20517 · Cited by 1
“…language models (LLMs) have recently demonstrated impressive capabilities in code-related tasks (Ridnik et al., 2024; Lozhkov et al., 2024; Roziere et al., 2023; Li et al., 2022; Nijkamp et al., 2022), powering applications such as AI-assisted programming, automated debugging, and code translation.”
Measuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · Cited by 21
“, 2022), and other works (Nijkamp et al., 2022; Fried et al., 2022; Allal et al., 2023; Christopoulou et al., 2022) have shown that decoder-only code language models achieve exceptional performance on a wide range of tasks.”
BackgroundNing, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Chen, Z., Sun, Z., Shi, Y., Lo, D., & Jiang, L. (2026) · arXiv · doi:10.48550/arxiv.2607.01211 · Cited by 9
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · arXiv · doi:10.48550/arxiv.2404.10100
What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · arXiv
Metadata
- Authors
- Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C.
- Affiliations
- Salesforce (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2203.13474
- OpenAlex
- W4226485558
- Cited by
- 330
- References
- 0
- Citations by year
- 2026: 11 · 2025: 94 · 2024: 130 · 2023: 90 · 2022: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- program synthesis, large language models, code generation, open-source models, HumanEval, natural language programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
