A systematic evaluation of large language models of code
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
From Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design (2025) · cites this paper as [14]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Research works printed but not linked: 13
- Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. 245–256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. 245–256.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , and Daxin Jiang . 2020 . Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. · arXiv
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. · arXiv
- Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural Language Processing with Transformers. " O’Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. " O’Reilly Media, Inc.".
- Daniel Zügner , Tobias Kirschstein , Michele Catasta , Jure Leskovec , and Stephan Günnemann . 2021 . Language-Agnostic Representation Learning of Source Code from Structure and Context . In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF
Cited by
Items 101–200 of 567
KernelGPT: Enhanced Kernel Fuzzing via Large Language Models
Yang, C., Zhao, Z., & Zhang, L. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716022 · Cited by 131
CAT-LM Training Language Models on Aligned Code And Tests
Rao, N., Jain, K., Alon, U., Goues, C. L., & Hellendoorn, V. J. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00193 · Cited by 45
Self-citationPeykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 84
Automating the Development of Task-oriented LLM-based Chatbots
Sánchez Cuadrado, J., Pérez-Soler, S., Guerra, E., & De Lara, J. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665538 · Cited by 46
A unified industrial large knowledge model framework in Industry 4.0 and smart manufacturing
Lee, J., & Su, H. (2024) · International Journal of AI for Materials and Design · doi:10.36922/ijamd.3681 · Cited by 82
Federated and edge learning for large language models
Piccialli, F., Chiaro, D., Qi, P., Bellandi, V., & Damiani, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102840 · Cited by 69
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
Large Language Models to generate meaningful feature model instances
Galindo, J. Á., Dominguez, A. J., White, J., & Benavides, D. (2023) · International Software Product Line Conference (SPLC) · doi:10.1145/3579027.3608973 · Cited by 49
Better Patching Using LLM Prompting, via Self-Consistency
Ahmed, T., & Devanbu, P. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00065 · Cited by 41
Abdollahi, M., Yeganli, S. F., Baharloo, M. M., & Baniasadi, A. (2024) · Electronics · doi:10.3390/electronics14010120 · Cited by 87
The Stack: 3 TB of permissively licensed source code
Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.15533 · Cited by 42
The Plastic Surgery Hypothesis in the Era of Large Language Models
Xia, C. S., Ding, Y., & Zhang, L. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00047 · Cited by 67
Hajipour, H., Hassler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 40
Zhou, Z., Li, J., Zhang, Z., Yu, J., & Duh, H. B. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2370635 · Cited by 55
Shi, H. H., Fang, L., Chen, X., Gu, C., Ma, K., Zhang, X., Zhang, Z., Gu, J., & Lim, E. G. (2024) · IET Smart Grid · doi:10.1049/stg2.12191 · Cited by 63
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 76
The Impact and Opportunities of Generative AI in Fact-Checking
Wolfe, R., & Mitra, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658987 · Cited by 40
Under the Influence: A Survey of Large Language Models in Fake News Detection
Kuntur, S., Wróblewska, A., Paprzycki, M., & Ganzha, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3471735 · Cited by 74
CYCLE: Learning to Self-Refine the Code Generation
Ding, Y., Min, M. J., Kaiser, G. E., & Ray, B. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3649825 · Cited by 94
The next generation of machine learning for tracking adaptation texts
Sietsma, A. J., Ford, J. D., & Minx, J. C. (2023) · Nature Climate Change · doi:10.1038/s41558-023-01890-3 · Cited by 55
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Ding, Y., Wang, Z., Ahmad, W. U., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11248 · Cited by 37
Wickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 89
TOGLL: Correct and Strong Test Oracle Generation with LLMS
Hossain, S. B., & Dwyer, M. B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00098 · Cited by 110
Can Large Language Models be Anomaly Detectors for Time Series?
Alnegheimish, S., Nguyen, L., Berti‐Équille, L., & Veeramachaneni, K. K. (2024) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa61799.2024.10722786 · Cited by 44
Traces of Memorisation in Large Language Models for Code
Al-Kaswan, A., Izadi, M., & van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639133 · Cited by 87
From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining
Ren, X., Ye, X., Zhao, D., Xing, Z., & Yang, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00143 · Cited by 66
Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets
Chakraborty, P., Arumugam, K. K., Alfadel, M., Nagappan, M., & McIntosh, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3423712 · Cited by 53
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 91
Teng, D., Ye, C., & Martinez, V. (2025) · Technovation · doi:10.1016/j.technovation.2025.103191 · Cited by 55
Large Language Models for manufacturing
Li, Y., Zhao, H., Jiang, H., Pan, Y., Liu, Z., Wu, Z., Shu, P., Tian, J., Yang, T., Xu, S., et al. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.02.014 · Cited by 117
(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Advanced Smart Contract Vulnerability Detection via LLM-Powered Multi-Agent Systems
Wei, Z., Sun, J., Sun, Y., Liu, Y., Wu, D., Zhang, Z., Zhang, X., Li, M., Liu, Y., Li, C., et al. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3597319 · Cited by 31
Investigating the Efficacy of Large Language Models for Code Clone Detection
Khajezade, M., Wu, J. J., Fard, F. H., Rodríguez-Pérez, G., & Shehata, M. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3645030 · Cited by 66
Prompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 109
Chen, L., Lei, Y., Jin, S., Zhang, Y., & Zhang, L. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3400189 · Cited by 54
Leveraging Large Language Models for Enhancing Safety in Maritime Operations
Miller, T. I., Durlik, I., Kostecka, E., Łobodzińska, A., Łazuga, K., & Kozlovska, P. (2025) · Applied Sciences · doi:10.3390/app15031666 · Cited by 45
Memorization and Generalization in Neural Code Intelligence Models
Rabin, M. R. I., Hussain, A., Alipour, M. A., & Hellendoorn, V. J. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107066 · Cited by 48
Self-citationGenerative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions
Xiao, T., Hata, H., Treude, C., & Matsumoto, K. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643773 · Cited by 29
Leveraging GPT-like LLMs to Automate Issue Labeling
Colavito, G., Lanubile, F., Novielli, N., & Quaranta, L. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644903 · Cited by 58
X-Lifecycle Learning for Cloud Incident Management using LLMs
Goel, D., Husain, F., Singh, A., Ghosh, S., Parayil, A., Bansal, C., Zhang, X., & Rajmohan, S. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663861 · Cited by 68
Gao, W., Mei, Y., Duh, H. B., & Zhou, Z. (2024) · The Design Journal · doi:10.1080/14606925.2024.2435703 · Cited by 47
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 70
Exploring and Characterizing Large Language Models for Embedded System Development and Debugging
Englhardt, Z., Li, R. C., Nissanka, D., Zhang, Z., Narayanswamy, G., Breda, J., Liu, X., Patel, S., & Iyer, V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650764 · Cited by 63
Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
Huang, Y., Li, Y., Wu, W., Zhang, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660818 · Cited by 67
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Fine Tuning Large Language Model for Secure Code Generation
Li, J., Sangalay, A., Cheng, C., Tian, Y., & Yang, J. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652299 · Cited by 53
Exploring the application of LLM-based AI in UX design: an empirical case study of ChatGPT
Zhou, Z., Li, Y., & Yu, J. (2024) · Human-Computer Interaction · doi:10.1080/07370024.2024.2420991 · Cited by 45
ChatAssert: LLM-Based Test Oracle Generation With External Tools Assistance
Hayet, I., Scott, A., & d’Amorim, M. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519159 · Cited by 47
Singh, V., Hughes, L., Albashrawi, M. A., Jeon, I., & Dwivedi, Y. K. (2025) · Journal of Systems and Information Technology · doi:10.1108/jsit-03-2025-0139 · Cited by 42
OMPGPT: A Generative Pre-trained Transformer Model for OpenMP
Chen, L., Bhattacharjee, A., Ahmed, N. K., Hasabnis, N., Oren, G., Vo, V. A., & Jannesari, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-69577-3_9 · Cited by 48
Nashaat, M., & Miller, J. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428324 · Cited by 47
Language models can identify enzymatic binding sites in protein sequences
Teukam, Y. G. N., Dassi, L. K., Manica, M., Probst, D., Schwaller, P., & Laino, T. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.012 · Cited by 36
Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design
Liu, Z., Chai, Y., & Li, J. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01653 · Cited by 24
Automated Test Case Repair Using Language Models
Yaraghi, A. S., Holden, D., Kahani, N., & Briand, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3541166 · Cited by 57
Bai, X., & Yang, L. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1589318 · Cited by 34
Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation
Jin, K., Wang, C.-Y., Pham, H. V., & Hemmati, H. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3645074 · Cited by 24
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“In contrast, we focus on adapting pre-trained LLMs (Chen et al., 2021b; Nijkamp et al., 2022; Tunstall et al., 2022; Xu et al., 2022; Fried et al., 2022) to performance optimization.”
BackgroundMethodSelf-citationHands-on analysis of using large language models for the auto evaluation of programming assignments
Mohamed, K. M., Yousef, M., Medhat, W., Mohamed, E. H., Khoriba, G. A., & Arafa, T. (2024) · Information Systems · doi:10.1016/j.is.2024.102473 · Cited by 36
Distinguishing LLM-Generated from Human-Written Code by Contrastive Learning
Xu, X., Ni, C., Guo, X., Liu, S., Wang, X., Liu, K., & Yang, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705300 · Cited by 57
Analyzing Automatic Code Generation for Learning Models in Generative AI
Jain, A., William, P., Ayasrah, F. T., Lakshmi, G. P., & Diwan, T. D. (2024) · International Conference on Software Defined Systems (SDS) · doi:10.1109/sds64317.2024.10883930 · Cited by 24
Espejel, J. L., Alassan, M. S. Y., Chouham, E. M., Dahhane, W., & Ettifouri, E. H. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100013 · Cited by 28
Detecting Code Comment Inconsistencies using LLM and Program Analysis
Zhang, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3664458 · Cited by 27
Zhou, W., Prater, L., Goldstein, E. V., & Mooney, S. J. (2023) · JMIR Mental Health · doi:10.2196/49359 · Cited by 27
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 41
Wong, S. M., Leung, H.-F., & Wong, K. Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4063228/v1 · Cited by 67
Towards Standardized benchmarks of LLMs in software modeling tasks: a conceptual framework
Cámara, J., Burgueño, L., & Troya, J. (2024) · Software & Systems Modeling · doi:10.1007/s10270-024-01206-9 · Cited by 34
What Makes a High-Quality Training Dataset for Large Language Models: A Practitioners' Perspective
Yu, X., Zhang, Z., Niu, F., Hu, X., Xia, X., & Grundy, J. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695061 · Cited by 72
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 98
Large Language Models: The Next Frontier for Variable Discovery within Metamorphic Testing?
Tsigkanos, C., Rani, P., Müller, S., & Kehrer, T. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00070 · Cited by 35
Graph embedding approaches for social media sentiment analysis with model explanation
Anoop, V. S., Krishna, C. S., & Govindarajan, U. H. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100221 · Cited by 35
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11912 · Cited by 21
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 31
AI-Related Risk: An Epistemological Approach
Zanotti, G., Chiffi, D., & Schiaffonati, V. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00755-7 · Cited by 34
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 31
A Study of Vulnerability Repair in JavaScript Programs with Large Language Models
Le, T. K., Alimadadi, S., & Ko, S. Y. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651463 · Cited by 39
Sarda, K., Namrud, Z., Litoiu, M., Shwartz, L., & Watts, I. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663855 · Cited by 19
Automating Zero-Shot Patch Porting for Hard Forks
Pan, S., Wang, Y., Liu, Z., Hu, X., Xia, X., & Li, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652134 · Cited by 37
CodeFusion: A Pre-trained Diffusion Model for Code Generation
Singh, M. P., Cambronero, J., Gulwani, S., Le, V., Negreanu, C., & Verbruggen, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.716 · Cited by 80
Quantifying OpenMP: Statistical Insights into Usage and Adoption
Kadosh, T., Hasabnis, N., Mattson, T. G., Pinter, Y., & Oren, G. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363459 · Cited by 19
Can GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
Open Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning
Lin, Z., Ma, W., Lin, T., Zheng, Y., Ge, J., Wang, J., Klein, J., Bissyandé, T. F., Liu, Y., & Li, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708529 · Cited by 33
Lv, S., Peng, L., Wu, W., Yao, Y., Jiao, S., & Hu, W. (2025) · Materials Genome Engineering Advances · doi:10.1002/mgea.70013 · Cited by 18
Shi, Z., Landrum, E., O'Connell, A., Kian, M., Pinto-Alva, L., Shrestha, K., Zhu, X., & Matarić, M. J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31245 · Cited by 34
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 157
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
“Through large-scale pretraining more generally applicable code models have been developed (Nijkamp et al., 2022; 2023; Xu et al., 2022a; Christopoulou et al., 2022; Gunasekar et al., 2023; Li et al., 2023b; Bui et al., 2023; Scao et al., 2022a;b).”
MethodDiaVio: LLM-Empowered Diagnosis of Safety Violations in ADS Simulation Testing
Lu, Y. J., Tian, Y., Bi, Y., Chen, B., & Peng, X. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652135 · Cited by 31
An Empirical Study on the Code Refactoring Capability of Large Language Models
Cordeiro, J., Noei, S., & Zou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3801158 · Cited by 95
A User-centered Security Evaluation of Copilot
Asare, O., Nagappan, M., & Asokan, N. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639154 · Cited by 32
Multibody Models Generated from Natural Language
Gerstmayr, J., Manzl, P., & Pieber, M. (2024) · Multibody System Dynamics · doi:10.1007/s11044-023-09962-0 · Cited by 31
Isolating Compiler Bugs by Generating Effective Witness Programs With Large Language Models
Tu, H., Zhou, Z., Jiang, H., Yusuf, I. N. B., Li, Y., & Jiang, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3397822 · Cited by 35
Deep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 27
PreparedLLM: effective pre -pretraining framework for domain-specific large language models
Chen, Z., Lin, M., Wang, Z., Zang, M., & Bai, Y. (2024) · Big Earth Data · doi:10.1080/20964471.2024.2396159 · Cited by 24
Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models
Yang, Z., Zhao, Z., Wang, C., Shi, J., Kim, D., Han, D., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3482719 · Cited by 42
Generating vulnerability security fixes with Code Language Models
Bhandari, G. P., Gavrić, N., & Shalaginov, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107786 · Cited by 28
Li, J., Zhang, Y., Karas, Z., McMillan, C., Leach, K., & Huang, Y. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644434 · Cited by 37
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 21
LLM4TDD: Best Practices for Test Driven Development Using Large Language Models
Piya, S., & Sullivan, A. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648382 · Cited by 42
Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models
Wu, J., Lu, C., Arrieta, A., Yue, T., & Ali, S. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652296 · Cited by 23
Black, G., Rimal, B. P., & Vaidyan, V. M. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3446695 · Cited by 27
Metadata
- Authors
- Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 1-10
- OpenAlex
- W4281763794
- Cited by
- 540
- References
- 19
- Citations by year
- 2026: 66 · 2025: 166 · 2024: 209 · 2023: 91 · 2022: 8
- FWCI
- 106.16
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- code generation, codices, GPT-J, GPT-Neo, programming languages, C programming language, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.