A systematic evaluation of large language models of code
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
From Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design (2025) · cites this paper as [14]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Research works printed but not linked: 13
- Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. 245–256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. 245–256.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , and Daxin Jiang . 2020 . Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. · arXiv
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. · arXiv
- Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural Language Processing with Transformers. " O’Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. " O’Reilly Media, Inc.".
- Daniel Zügner , Tobias Kirschstein , Michele Catasta , Jure Leskovec , and Stephan Günnemann . 2021 . Language-Agnostic Representation Learning of Source Code from Structure and Context . In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF
Cited by
Items 301–400 of 567
PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage
Chu, B., Feng, Y., Liu, K., Shi, H., Nan, Z., Guo, Z., & Xu, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00223 · Cited by 12
An empirical study of testing practices in open source AI agent frameworks and agentic applications
Hasan, M. M., Li, H., Fallahzadeh, E., Rajbahadur, G. K., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10857-9 · Cited by 40
Ramamonjison, R., Yu, T. T., Xing, L., Mostajabdaveh, M., Li, X., Fu, X., Han, X., Chen, Y., Li, R., Mao, K., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.45 · Cited by 8
Concrete Type Inference for Code Optimization using Machine Learning with SMT Solving
Ye, F., Zhao, J., Shirako, J., & Sarkar, V. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622825 · Cited by 5
Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorization
Chen, Z., & Jiang, L. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695021 · Cited by 12
Alsofyani, M., & Wang, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07237-3 · Cited by 6
An empirical study on the effectiveness of large language models for binary code understanding
Shang, X., Fu, Z., Cheng, S., Chen, G., Li, G., Hu, L. H., Zhang, W., & Yu, N. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10748-5 · Cited by 10
Optimized Tokenization Process for Open-Vocabulary Code Completion: An Empirical Study
Hussain, Y., Huang, Z., Zhou, Y., Khan, I. A., Khan, N., & Abbas, M. Z. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3594236 · Cited by 8
Context-Aware Name Recommendation for Field Renaming
Dong, C., Jiang, Y., Niu, N., Zhang, Y., & Liu, H. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639195 · Cited by 8
Attribution-guided Adversarial Code Prompt Generation for Code Completion Models
Li, X., Meng, G., Liu, S., Lu, X., Sun, K., Chen, K., Luo, X., & Liu, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695517 · Cited by 17
Robustness evaluation of code generation systems via concretizing instructions
Yan, M. X., Chen, J., Zhang, J. M., Cao, X., Yang, C., & Harman, M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107645 · Cited by 22
Extracting Abstractive Summaries Through Generative AI Models
Khan, R., Sharma, S., & Upadhyay, D. (2025) · International Conference on Disruptive Technologies (ICDT) · doi:10.1109/icdt63985.2025.10986332 · Cited by 5
Automated Software Testing through Large Language Models: Opportunities and Challenges
Fuente, J. A. D. (2022) · Qubahan Techno Journal · doi:10.48161/qtj.v1n3a15 · Cited by 5
Cheok, A. D., & Zhang, E. Y. (2023) · Qeios · doi:10.32388/3ntolq.2 · Cited by 8
Metamorphic-Based Many-Objective Distillation of LLMs for Code-Related Tasks
Panichella, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00230 · Cited by 14
Evaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 9
GraphCodeBERT-Augmented Graph Attention Networks for Code Vulnerability Detection
Rangapuram, D. S., & Ratha, N. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00161 · Cited by 5
On the Evaluation of Large Language Models in Multilingual Vulnerability Repair
Wang, D., Yu, J., Shu, H., Fu, M. C., Tantithamthavorn, C. K., Kamei, Y., & Chen, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771930 · Cited by 9
Digital Game Development Using Large Language Models (LLMs): An Exploratory Study
Serra, C. B., Serra, G. M. B., & de Classe, T. M. (2025) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames.2025.9992 · Cited by 4
LLM-Based Multi-Artifact Consistency Verification for Programming Exercise Quality Assurance
Dietrich, F. T. J., Zhou, Y., Wasner, T., Krusche, S., & Acosta, M. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770042 · Cited by 5
Takahara, W., Yamaguchi, Y., Ogano, M., Kakami, F., Harashima, Y., Takayama, T., Takasuka, S., Kudo, A., & Fujii, M. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01941 · Cited by 8
Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 13
A Pilot Study on AI-Assisted Code Generation with Large Language Models for Software Engineering
Liu, H.-C., Tsai, C.-T., & Day, M.-Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1711-8_12 · Cited by 6
CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking
Su, Z., Xu, X., Huang, Z., Zhang, Z., Ye, Y., Huang, J., & Zhang, X. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643752 · Cited by 28
Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis
Chen, L., Xie, H., Qin, Z., Guo, Y., Rohde, J. A., & Zhang, Y. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328374 · Cited by 9
Towards LLM-Based System Migration in Language-Driven Engineering
Busch, D., Bainczyk, A., & Steffen, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49252-5_14 · Cited by 5
bjEnet: a fast and accurate software bug localization method in natural language semantic space
Han, J., Huang, C., & Liu, J. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09693-1 · Cited by 6
Measuring code efficiency optimization capabilities with ACEOB
Pan, Y., Shao, X., & Lyu, C. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112250 · Cited by 7
ExCS: accelerating code search with code expansion
Huang, S., Cai, B., Yu, Y., & Luo, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-73907-6 · Cited by 5
Attributing ChatGPT-Generated Source Codes
Choi, S., & Mohaisen, A. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3535218 · Cited by 12
Combining Contexts from Multiple Sources for Documentation-Specific Code Example Generation
Khan, J. Y., & Uddin, G. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00071 · Cited by 11
Python Code Generation by Asking Clarification Questions
Li, H.-S., Mesgar, M., Martins, A. F. T., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.799 · Cited by 35
Eichstädt, S., Gruber, M., & Vedurmudi, A. P. (2023) · tm - Technisches Messen · doi:10.1515/teme-2023-0103 · Cited by 4
Zheng, Z., Florit, F., Jin, B., Wu, H., Li, S., Nandiwale, K. Y., Salazar, C. A., Mustakis, J. G., Green, W. H., & Jensen, K. F. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-pk105-v2 · Cited by 3
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Semantic Similarity Search for Source Code Plagiarism Detection: An Exploratory Study
Ebrahim, F., & Joy, M. S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653622 · Cited by 13
Salvi, R. C., & Bosch, N. (2025) · ACM Journal on Responsible Computing · doi:10.1145/3737882 · Cited by 4
Is Quantization a Deal-Breaker? Empirical Insights From Large Code Models
Afrin, S., Xu, B., & Mastropaolo, A. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00049 · Cited by 18
CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMs
Pan, J., Jacobson, I., Zhao, Z., Chen, T.-C., Zhou, G.-S., Chang, C.-C., Rashingkar, V., & Chen, Y. (2025) · doi:10.1109/iccad66269.2025.11240643 · Cited by 8
Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement
Jin, H., & Chen, H. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00638-5 · Cited by 16
Modelling Engineering Processes in Natural Language: A Case Study
Kogler, P., Chen, W., Falkner, A. A., Haselböck, A., & Wallner, S. (2024) · International Software Product Line Conference (SPLC) · doi:10.1145/3646548.3672584 · Cited by 7
Effectiveness of symmetric metamorphic relations on validating the stability of code generation LLM
Chan, P. Y. P., Keung, J., & Yang, Z. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112330 · Cited by 8
Fine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats
Setak, M., & Madani, P. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00043 · Cited by 12
Large-scale Evaluation of Notebook Checkpointing with AI Agents
Fang, H., Chockchowwat, S., Sundaram, H., & Park, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719928 · Cited by 4
Tsamis, G., Evangelos, G., Papakostas, A., Vassiliou, G., Grafanakis, M., Garefalakis, A. E., Vassalos, M., Mylona, A., & Papadakis, N. (2025) · Algorithms · doi:10.3390/a18060339 · Cited by 5
Khan, H. A., Jiang, Y., Umer, Q., Zhang, Y., Akram, W., & Liu, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715765 · Cited by 4
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
Petrukha, I., Kurliak, Y., & Stulova, N. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00015 · Cited by 6
Pre-trained Model-based Actionable Warning Identification: A Feasibility Study
Ge, X., Fang, C., Zhang, Q., Wu, D., Yu, B., Zheng, Q., Guo, A., Lin, S., Zhao, Z., Liu, Y., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3777369 · Cited by 6
Chen, N., Sun, Q., Wang, J., Gao, M. Y., Li, X., & Li, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.991 · Cited by 17
Exploring the Impact of Vocabulary Techniques on Code Completion: A Comparative Approach
Hussain, Y., Huang, Z., Zhou, Y., & Khan, I. A. (2023) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194023500687 · Cited by 3
Extending Jupyter with Multi-Paradigm Editors
Weber, T., Ehe, J., & Mayer, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3660247 · Cited by 4
Couler: Unified Machine Learning Workflow Optimization in Cloud
Wang, X., Tang, Y., Guo, T., Sang, B., Wu, J., Sha, J., Zhang, K., Qian, J., & Tang, M. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00393 · Cited by 4
Dong, J., Luo, X., & Zhu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5501-1_23 · Cited by 3
E-code: Mastering efficient code generation through pretrained models and expert encoder group
Pan, Y., Lyu, C., Yang, Z., Li, L., Liu, Q., & Shao, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107602 · Cited by 8
Do LLMs Provide Links to Code Similar to What They Generate? A Study with Gemini and Bing CoPilot
Bifolco, D., Cassieri, P., Scanniello, G., Penta, M. D., & Zampetti, F. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00042 · Cited by 7
Can LLMs evaluate items measuring collaborative problem-solving?Preprint
Anghel, E., Wang, Y., Gopalakrishnan, M., Mansukhani, P., & Bergner, Y. (2024) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/tuvks · Cited by 4
MVD: A Multi-Lingual Software Vulnerability Detection FrameworkPreprint
Zhang, B., Le, T. H. M., & Babar, M. A. (2025) · Qeios · doi:10.32388/4ahqy3 · Cited by 3
Assessing Small Language Models for Code Generation: An Empirical Study with BenchmarksPreprint
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5472927 · Cited by 3
Full Line Code Completion: Bringing AI to Desktop
Semenkin, A., Bibaev, V., Sokolov, Y., Krylov, K., Kalina, A., Khannanova, A., Savenkov, D., Rovdo, D., Davidenko, I., Karnaukhov, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.08704 · Cited by 8
“The most famous examples of such models are InCoder [28], [40], StarCoder [29], [41], CodeLLaMA [27], [42], CodeGen and CodeGen-2 [26], [43], [44], CodeParrot [45], and PolyCoder [46], [47].”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4
SCC-GPT: Source Code Classification Based on Generative Pre-Trained Transformers
Alahmadi, M. D., Alshangiti, M., & Alsubhi, J. (2024) · Mathematics · doi:10.3390/math12132128 · Cited by 3
Lenz, S., Ustjanzew, A., Jeray, M., Ressing, M., & Panholzer, T. (2025) · BioData Mining · doi:10.1186/s13040-025-00463-8 · Cited by 8
Evaluating Correct-Consistency and Robustness in Code-Generating LLMs
Honarvar, S. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988971 · Cited by 3
Adding Context to LLM-Guided Verilog Repair
Elnaggar, A., & Tan, B. (2025) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed65160.2025.11014472 · Cited by 4
Teaching Things To Think: Bootstrapping Local Reasoning for Smart(er) Devices
King, E., Yu, H., Vartak, S., Jacob, J., Lee, S., & Julien, C. (2025) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom64205.2025.00027 · Cited by 3
Trustworthiness of Large Language Models for Code
Khati, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00063 · Cited by 3
DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation
Li, J., Hyun, S., & Babar, M. A. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00030 · Cited by 6
StreamLLM: Enhancing Constraint Programming with Large Language Model-Generated Streamliners
Voboril, F., Ramaswamy, V. P., & Szeider, S. (2025) · doi:10.1109/nse66660.2025.00010 · Cited by 10
Advancing research software engineering with AI: a research framework
Farshidi, S., Bennin, K. E., Babur, Ö., Sallou, J., Kassahun, A., & Tekinerdoğan, B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00621-0 · Cited by 9
Prompt2DAG: A Modular Prompting Approach for Democratizing Data Pipeline Generation
Alidu, A., Ciavotta, M., & Paoli, F. D. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00010 · Cited by 2
The fools are certain; the wise are doubtful: Exploring LLM confidence in code completion
Kotti, Z., Dritsa, K., Spinellis, D. D., & Λουρίδας, Π. (2026) · Automated Software Engineering · doi:10.21203/rs.3.rs-7302683/v1 · Cited by 3
RefModel: Detecting Refactorings using Foundation Models
Simões, P., Gheyi, R., Melo, R., Oliveira, J., Ribeiro, M., & Assunção, W. K. G. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11582 · Cited by 3
Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?
Kochnev, R., Goodarzi, A. T., Bentyn, Z. A., Ignatov, D., & Timofte, R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00598 · Cited by 30
HPC-Coder-v2: Studying Code LLMs Across Low-Resource Parallel Languages
Chaturvedi, A., Nichols, D., Singh, S., & Bhatele, A. S. (2025) · ISC High Performance (ISC) · doi:10.23919/isc.2025.11017585 · Cited by 20
When Domains Collide: An Activity Theory Exploration of Cross-Disciplinary Collaboration
Feng, Z., Zimmermann, T., Pisani, L., Gooley, C., Wander, J., & Sarma, A. (2025) · 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem64174.2025.00017 · Cited by 4
Unit Test Update through LLM-Driven Context Collection and Error-Type-Aware Refinement
Zhang, Y., Yang, Z., Pan, S., & Liu, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00206 · Cited by 10
Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications
Jin, H., & Chen, H. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00323 · Cited by 16
Hierarchical Syntax-Aware Modeling for Code Generation and Program Repair
Qiu, C. (2025) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc68035.2025.11385172 · Cited by 4
Prompting the Priorities: Evaluating LLMs for Organization-Specific Vulnerability Prioritization
Haddad, O. A., Ikram, M., Ahmed, M. E., & Lee, Y. C. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00044 · Cited by 2
Automated Repair of Alloy Specifications in the Era of Large Language Models
Hasan, M. R., Li, J., Ahmed, I., & Bagheri, H. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3682711 · Cited by 2
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 3
How Well Can Masked Language Models Spot Identifiers That Violate Naming Guidelines?
Villmow, J., Campos, V., Petry, J., Andaloussi, A. A., Ulges, A., & Weber, B. (2023) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam59687.2023.00023 · Cited by 5
Digital Rubber Duck: Leveraging Large Language Models for Extreme Programming
Elvira, T., Procko, T., Couder, J. O., & Ochoa, O. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00051 · Cited by 9
Martins, G. F., Firmino, E. C. M., & Mello, V. P. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_6 · Cited by 4
Meta-requirements for LLM-Based Knowledge Exploration Tools in Information Systems Research
Sjöström, J., & Cronholm, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61175-9_29 · Cited by 7
Gou, Q., Dong, Y., & Ke, Q. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112140 · Cited by 2
Exploring Advanced Methodologies in Security Evaluation for Large Language Models
Huang, J., Zhang, J., Wang, Q., Han, W., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-4519-7_10 · Cited by 2
Liu, H., Zhou, S., Chen, Z., Perl, Y., & Wang, J. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00040 · Cited by 7
BugOut: Automated Test Generation and Bug Detection for Low-Code
Coutinho, J., Lemos, A., Terra-Neves, M., Ribeiro, A., Manquinho, V. M., Quintino, R., & Matejczyk, B. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00041 · Cited by 3
Tackling Students' Coding Assignments with LLMs
Dingle, A., & Kruliš, M. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648389 · Cited by 9
Large Language Models in Automated Repair of Haskell Type Errors
Santos, S. L., Saraiva, J., & Ribeiro, F. (2024) · doi:10.1145/3643788.3648012 · Cited by 5
kNN Classification of Malware Data Dependency Graph Features
Musgrave, J., & Ralescu, A. (2024) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon61878.2024.10670673 · Cited by 2
Self-Evolving Programs: A Novel Approach Leveraging LLMs and Quine Programs
Saghiri, A. M., & Wang, N. (2024) · International Conference on Computing, Internet of Things and Microwave Systems (ICCIMS) · doi:10.1109/iccims61672.2024.10690672 · Cited by 2
PACGBI: A Pipeline for Automated Code Generation from Backlog Items
Sarschar, M., Zhang, G., & Nowak, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695346 · Cited by 6
Jacinto, M. V. G., Oliveira, L. H. L. D., Rodrigues, T. C., de Medeiros, G. C., Medeiros, D. R., Silva, M. A., de Montalvão, L. C., González, M., & de Almeida, R. V. (2024) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/221864-ms · Cited by 4
LLAMA-2 Acceleration Using the ARM Scalable Vector Extension
Rossi, F., Cococcioni, M., & Saponara, S. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-3-031-84100-2_31 · Cited by 2
Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
AI support for data scientists: An empirical study on workflow and alternative code recommendations
Ramasamy, D., Sarasua, C., & Bernstein, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10622-4 · Cited by 3
Metadata
- Authors
- Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 1-10
- OpenAlex
- W4281763794
- Cited by
- 540
- References
- 19
- Citations by year
- 2026: 66 · 2025: 166 · 2024: 209 · 2023: 91 · 2022: 8
- FWCI
- 106.16
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- code generation, codices, GPT-J, GPT-Neo, programming languages, C programming language, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.