A systematic evaluation of large language models of code
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
From Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design (2025) · cites this paper as [14]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Research works printed but not linked: 13
- Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. 245–256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. 245–256.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , and Daxin Jiang . 2020 . Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. · arXiv
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. · arXiv
- Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural Language Processing with Transformers. " O’Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. " O’Reilly Media, Inc.".
- Daniel Zügner , Tobias Kirschstein , Michele Catasta , Jure Leskovec , and Stephan Günnemann . 2021 . Language-Agnostic Representation Learning of Source Code from Structure and Context . In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF
Cited by
Items 401–500 of 567
New Deep Learning-Based Approach for Source Code Generation: Application to Computer Vision Systems
Alshehri, W. S., Jarraya, S. K., & Allinjawi, A. (2025) · AI · doi:10.3390/ai6070162 · Cited by 2
Generalized Adversarial Code-Suggestions: Exploiting Contexts of LLM-based Code-Completion
Rubel, K., Noppel, M., & Wressnegger, C. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3733870 · Cited by 3
Evaluation of the Code Generated By Large Language Models: The State of the Art
Ying, Z., Towey, D., & Zhang, Y. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00064 · Cited by 3
Machine Learning Powered Code Smell Detection as a Business Improvement Tool
Siksna, M., Berzina, I., & Romānovs, A. (2023) · International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS) · doi:10.1109/itms59786.2023.10317724 · Cited by 2
Search and Retrieval in Semantic-Structural Representations of Novel Malware
Musgrave, J. A., Câmpan, A., Messay-Kebede, T., Kapp, D., & Wang, B. (2024) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2024.41117 · Cited by 2
Learning AI Coding Style for Software Plagiarism Detection
Ambati, S. H., Stakhanova, N., & Branca, E. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-64954-7_24 · Cited by 3
LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine
Isaku, E., Laaber, C., Sartaj, H., Ali, S., Schwitalla, T., & Nygård, J. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989025 · Cited by 7
ISiCell: involving biologists in the design process of agent-based models in cell biologyPreprint
Cogoni, F., Bernard, D., Kazhen, R., Valitutti, S., Lobjois, V., & Cussat‐Blanc, S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.30.547165 · Cited by 2
Transforming Ideas into Code: Visual Sketching for ML Development
Gomes, L. (2023) · ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH) · doi:10.1145/3618305.3623588 · Cited by 2
Improving Source Code Pre-Training via Type-Specific Masking
Zou, W., Li, Q., Li, C., Ge, J., Chen, X., Huang, L., & Luo, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699599 · Cited by 3
Analysis of the Effectiveness of Large Language Model Feature in Source Code Defect Detection
He, T., Yang, M., Hu, W. P., & Chen, Y. (2024) · International Conference on Artificial Intelligence and Computer Information Technology (AICIT) · doi:10.1109/aicit62434.2024.10730232 · Cited by 1
DL-VulBERT: A Deep Learning Classifier for the Identification of Software Vulnerabilities
Gupta, A., Tomar, D. S., & Shekhar, R. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724230 · Cited by 5
Yadav, J., Arora, S., & Kashyap, N. (2025) · International Conference on Computational Intelligence, Communication Technology and Networking (CICTN) · doi:10.1109/cictn64563.2025.10932635 · Cited by 1
Li, Y., Zhong, W., Shen, Z., Li, C., Chen, X., Ge, J., & Luo, B. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00502-y · Cited by 3
On the Applicability of Code Language Models to Scientific Computing Programs
Zhao, Q., Liu, F., Long, X., Wu, C., & Zhang, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3564599 · Cited by 4
Enhancing Code Generation Through Retrieval of Cross-Lingual Semantic Graphs
Jiang, Z., Shi, Z., Gao, X., & Xiong, Y. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00026 · Cited by 4
Is More or Less Automation Better? An Investigation into the LLM4TDD Process
Piya, S., Samadi, A., & Sullivan, A. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00025 · Cited by 2
Mix-of-Language-Experts Architecture for Multilingual Programming
Zong, Y., Deng, Y., & Nie, P. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00030 · Cited by 4
Code Ranking with Structure Awareness Contrastive Learning
Huang, H., Cao, L., Wang, J., Yu, T., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00053 · Cited by 1
Liang, Y., Ying, R., Taniguchi, T., Gan, C., & Cui, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_43 · Cited by 2
LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code
Jararweh, A., Adams, M., Sahu, A. D., Mueen, A., & Anwar, A. (2025) · Intelligent Cybersecurity Conference (ICSC) · doi:10.1109/icsc65596.2025.11140501 · Cited by 2
The “Question Neighbourhood” Approach for Systematic Evaluation of Code-Generating LLMs
Honarvar, S., Rei, M., & Donaldson, A. F. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3612251 · Cited by 6
Evaluation and Improvement of Test Selection for Large Language Models
Quan, L., Jin, W., Hu, Q., Cordy, M., Huang, Y., Ma, L., & Li, X. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70057 · Cited by 5
Wu, S., Xiao, J., & Sang, J. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3620146 · Cited by 3
Unveiling the Centralized Security Risks in Decentralized Ecosystems
Yan, K., Zhang, J., Liu, X., & Diao, W. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3622304 · Cited by 2
RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
Liu, A. Q., Li, H., Wang, B., Yang, A., & Li, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228706 · Cited by 7
Synthetic Malware at Scale: Malicious Code Generation With Code Transplanting
Wang, G., Chen, D., Gu, X., Chen, Y., & Shen, B. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3633280 · Cited by 1
Duan, Z., Bian, C., & Yang, S. (2025) · Renewable Energy · doi:10.1016/j.renene.2025.124808 · Cited by 4
Fu, A., Xu, P., Li, J., Kuang, B., & Gao, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3618407 · Cited by 10
A Comparative Study on Large Language Models with an Overview of Their Issues and Challenges
Kumar, A., & Sood, M. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-5969-2_13 · Cited by 1
Securing AI Code Generation - A Prompt Rectification Approach for Mitigating Cyber Risks
Dong, J., Ni, Z., Sun, N., Jha, S., Zhang, Y., Bertino, E., Nepal, S., & Ma, S. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00106 · Cited by 2
LLM-Based Explainable Detection of LLM-Generated Code in Python Programming Courses
Baek, J., Yamazaki, T., Morihata, A., Mori, J., Yamakata, Y., Taura, K., & Chiba, S. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772605 · Cited by 3
APILOT: Improving the Security and Usability of LLM Code Suggestions via Outdated API Mitigation
Bai, W., Ke, X., Huang, P., Wu, Q., Wen, J., Wu, J., & Lu, K. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00094 · Cited by 17
Towards defect-type-aware adaptive program repair: A stage-wise approach with large language models
Zhang, R., Zhang, S., & Xie, L. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09751-w · Cited by 5
Enhancing Serverless Function Resilience Against Vulnerabilities with MoFaaS
Escaleira, P., Cunha, V. A., Barraca, J. P., Gomes, D., & Aguiar, R. L. (2026) · Journal of Network and Systems Management · doi:10.1007/s10922-026-10054-5 · Cited by 1
Fun2spec: Code Contract Synthesis At Scale
Ugare, S., Suresh, T., Misailovíc, S., & Vanegue, J. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805262 · Cited by 5
Foundation Models for Text Generation
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_6 · Cited by 5
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Payan, J., Mishra, S. R., Singh, M. P., Negreanu, C., Poelitz, C., Baral, C. R., Roy, S., Chakravarthy, R., Durme, B. V., & Nouri, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.265 · Cited by 13
The JavaScript Package Selection Task: A Comparative Experiment Using ChatGPT
Vázquez, H. C., Diaz-Pace, J. A., & Tommasel, A. (2023) · Latin American Computer Conference (CLEI) · doi:10.1109/clei60451.2023.10345786 · Cited by 1
Towards Safe, Secure, and Usable LLMs4Code
Al-Kaswan, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639803 · Cited by 2
Semantic similarity loss for neural source code summarization
Su, C., & McMillan, C. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2706 · Cited by 2
Aligning Crowd-sourced Human Feedback for Code Generation with Bayesian Inference
Wong, M. F., & Tan, C. W. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00037 · Cited by 8
A Study of Coding Framework Generation by ChatGPT
Zhu, H. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.18277 · Cited by 1
Memes in the Machine: Ideological Propagation in Large Language Models
Farlow, H., Ceruti, C., Garratt, M., Mount, G., & Lynar, T. (2024) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom63007.2024.10894714 · Cited by 1
Prompt Learning for Source Code Summarization
Xu, T., Fang, C., Qian, H., Feng, X., & Sun, W. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00028 · Cited by 2
Bhutani, V., Toosi, F. G., & Buckley, J. (2025) · Applied Computer Systems · doi:10.2478/acss-2025-0013 · Cited by 1
Can Mamba Be Better? An Experimental Evaluation of Mamba in Code Intelligence
Liu, S., Keung, J., Yang, Z., Mao, Z., & Sun, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00155 · Cited by 3
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.04106 · Cited by 1
Semantic Similarity Loss for Neural Source CodeSummarizationPreprint
Su, C., & McMillan, C. (2023) · Research Square · doi:10.21203/rs.3.rs-3240092/v1 · Cited by 6
Predicting typeScript type annotations and definitions with machine learning
Yee, M. (2024) · Digital Repository Service (Northeastern University) · doi:10.17760/d20653005 · Cited by 2
Zheng, Z., Florit, F., Jin, B., Wu, H., Li, S., Nandiwale, K. Y., Salazar, C. A., Mustakis, J. G., Green, W. H., & Jensen, K. F. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-pk105
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 5
Document Understanding with Deep Learning Techniques
Nguyen, K.-A. L. (2024) · theses.fr (ABES) · doi:10.70675/b83ce709z5fc9z4a33za0bezd7aaad779ef1 · Cited by 4
Peer Review #3 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/3
Peer Review #2 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/2
Peer Review #1 of "Code4ML: a large-scale dataset of annotated Machine Learning code (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1230v0.1/reviews/1
Reconstructing Word Representations from Pre-trained Subword Embeddings
Lauren, P. (2022) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci58124.2022.00013 · Cited by 2
The Proof is in the Pudding: Using Automated Theorem Proving to Generate Cooking Recipes
Mahon, L., & Vogel, C. M. (2023) · LDV-Forum/Journal for language technology and computational linguistics · doi:10.21248/jlcl.36.2023.233 · Cited by 2
An Approach to Generating API Test Scripts Using GPT
Nguyen, C., Bui, H., Nguyen, V., & Nguyen, T. N. (2023) · International Symposium on Information and Communication Technology · doi:10.1145/3628797.3628947
Gu, J., Nan, Z., Peng, Z., Shen, X., & Xu, D. (2023) · doi:10.18653/v1/2023.pandl-1.7
Natural Language Annotations for Reasoning about Program Semantics
Zocca, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.601
Secure Inference on Layered Spiking Neural P Systems Using Homomorphic EncryptionPreprint
Pleṣa, M.-I., Gheorghe, P., & Ipate, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4685965
Bridging the Gap in Code Reviews: A Large Language Model-based Proposal for Global Development Teams
Martins, G. F., & Firmino, E. C. M. (2024) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2024.237740
A Survey on Code Pre-Trained Models for Neural Program RepairPreprint
Zhan, S., Wang, X., Wei, D., & Cao, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4831589
Programming Language Models in Multilingual Settings
Katzy, J. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639787
Road to AGI: The Importance of Clear Objectives and Reward FunctionsPreprint
Rasoolinejad, M. (2024) · Preprints.org · doi:10.20944/preprints202407.0687.v1
Bekhouche, S. E., & Hadid, A. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595920
The JavaScript Package Selection Task: A Comparative Experiment Using an LLM-based Approach
Pace, A. D., Tommasel, A., & Vazquez, H. C. (2024) · CLEI electronic journal · doi:10.19153/cleiej.27.2.4 · Cited by 6
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Lightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
Steynberg, J., van Biljon, J. A., & van der Merwe, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71713-0_15 · Cited by 1
Automatic Pseudocode Extraction at Scale
Toksoz, L., Tan, G., & Giles, C. L. (2024) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri62200.2024.00061 · Cited by 1
Use Site Checking Considered Harmful
Racordon, D., & Chung, B. (2024) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3689492.3689814
DataRecipe --- How to Cook the Data for CodeLLM?
Kim, K., Kim, J., Park, B., Kim, D., Chong, C. Y., Wang, Y., Sun, T., Tang, X., Klein, J., & Bissyandé, T. F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695593 · Cited by 8
Assessing the Potential of Large Language Models for Chemical Engineering Applications
Xuan, L., Haoxiang, Z., Baozheng, J., & You, L. (2024) · doi:10.1145/3698385.3699878
Hyla, T., & Wawrzyniak, N. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3691417
Beyond Copy–Pasting—Contextualizing LLMs for Secure Code Generation
Arjun, S., Majumdar, D., Nabin, A. Z., Sharma, A. K., & Raheman, S. M. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-97-5791-6_15
Liu, X., Yin, D., Zhang, C., Zhao, D., & Feng, Y. (2025) · Computational Linguistics · doi:10.1162/coli_a_00548 · Cited by 10
Assessing GPT's Legal Knowledge in Japanese Real Estate Transactions Exam
Inoshita, K. (2024) · International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT) · doi:10.1109/3ict64318.2024.10824669 · Cited by 1
Zhu, S., Xia, B., Duan, F., Zhenyang, Z., Zhao, Z., Xiao, T., Liu, Z., & Liu, X. (2024) · International Conference on Electronics, Computers and Communication Technology (CECCT) · doi:10.1145/3705754.3705790
Research on Compressed Input Sequences Based on Compiler Tokenization
Li, Z., & Lu, X. (2025) · Information · doi:10.3390/info16020073 · Cited by 1
Leveraging Generative AI for Accelerating Enterprise Application Development: Insights from ChatGPT
Rajbhoj, A., Sant, T., Somase, A., & Kulkarni, V. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00052
Sokol, O. (2025) · Technologies and Engineering · doi:10.30857/2786-5371.2025.1.3 · Cited by 2
Assessing Code Synthesis In LLMS: CodeGeex and Codeium
Sheoran, S., & Kumar, D. (2025) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v12i5.563406
Training a Transformer Model for OpenMP Pragma Generation
Bhattacharjee, A., & Jannesari, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90203-1_52
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
Desarkar, A., & Raman, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-3287-9_42
Leveraging Large Language Models for MATLAB to Python Translation
Pietrini, R. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-3-031-87154-2_49 · Cited by 1
Yan, L., Lyu, C., Xing, R., Li, W., Samih, Y., Ji, T., Wang, L., & Jiang, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_12
Yonathan, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5367078 · Cited by 1
Exploiting Contexts of LLM-based Code-Completion
Noppel, M., Rubel, K., & Wressnegger, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02813-6_26
Syntactic multilingual probing of pre-trained language models of code
López, J. A. H., Weyssow, M., Cuadrado, J. S., & Sahraoui, H. A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112604
A Two-Stage Framework Integrating Prompt Learning and Fine-Tuning for Code Summarization
Sun, X., Lv, S., Wan, W., Qin, Y., & Hu, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_24 · Cited by 1
LLMs: A Driving Force in Next Generation Digital Design Automation
Moreira, M. T., Markosyan, A. H., Cummins, C., Hunt, W. A., Synnaeve, G., & Beigné, É. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132759
Method Name Recommendation Based on Large Language Model
Chen, Q., Liu, K., & Liu, Z. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00020 · Cited by 1
Tell-Tale Watermarks for Explanatory Reasoning in Synthetic Media Forensics
Chang, C., & Echizen, I. (2026) · IEEE Access · doi:10.1109/access.2026.3660000 · Cited by 1
Law, K.-M., Ho, S.-B., & Tan, C.-H. (2025) · AIP conference proceedings · doi:10.1063/5.0299254
A Survey of Learning-based Method Name Prediction
Qian, H., Liu, W., Xu, T., Yin, J., Feng, X., Sun, W., Ding, Z., Chen, Y., Miao, Y., Li, J., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771999
Secure Code Generation with Parameter-Efficient Fine-Tuning of LLMs
Hasan, M. A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07992-3_19
Metadata
- Authors
- Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 1-10
- OpenAlex
- W4281763794
- Cited by
- 540
- References
- 19
- Citations by year
- 2026: 66 · 2025: 166 · 2024: 209 · 2023: 91 · 2022: 8
- FWCI
- 106.16
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- code generation, codices, GPT-J, GPT-Neo, programming languages, C programming language, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.