A systematic evaluation of large language models of code
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
From Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design (2025) · cites this paper as [14]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Research works printed but not linked: 13
- Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. 245–256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. 245–256.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , and Daxin Jiang . 2020 . Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. · arXiv
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. · arXiv
- Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural Language Processing with Transformers. " O’Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. " O’Reilly Media, Inc.".
- Daniel Zügner , Tobias Kirschstein , Michele Catasta , Jure Leskovec , and Stephan Günnemann . 2021 . Language-Agnostic Representation Learning of Source Code from Structure and Context . In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF
Cited by
Items 201–300 of 567
Zhao, Y. D., Gong, L., Huang, Z., Wang, Y., Wei, M., & Wu, F. Q. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695539 · Cited by 18
Black, G., Vaidyan, V. M., & Comert, G. (2024) · IEEE Access · doi:10.1109/access.2024.3484947 · Cited by 23
Benchmarking Causal Study to Interpret Large Language Models for Source Code
Rodriguez-Cardenas, D., Palacio, D. N., Khati, D., Burke, H., & Poshyvanyk, D. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00040 · Cited by 14
Hong, E., Kazmir, S., Dylik, B., Auerbach, M. A., Rosati, M., Athanasopoulou, S., Himmelstein, R., Whitfill, T. M., Johnston, L., Wolbrink, T. A., et al. (2025) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000861 · Cited by 19
Online dispute resolution: can we leave the initial decision to Large Language Models (LLM)?
Ferrer-Benítez, M. (2022) · Metaverse Basic and Applied Research · doi:10.56294/mr202223 · Cited by 15
Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?
Kou, B., Chen, S., Wang, Z., Ma, L., & Zhang, T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660807 · Cited by 53
Yang, Y., Li, J., Guo, J., Pang, P. C., Wang, Y., Yang, X., & Im, S. (2025) · IEEE Access · doi:10.1109/access.2025.3549733 · Cited by 18
Bull, C., & Kharrufa, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.13936 · Cited by 14
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
Liang, J., Hou, S., Zhao, A., Xu, Q., Xiang, L., Li, R., & Wu, H. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2483884 · Cited by 22
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 35
Zhao, S., Tian, J., Fu, J., Chen, J., & Wen, J. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3395531 · Cited by 17
Franc: A Lightweight Framework for High-Quality Code Generation
Siddiq, M. L., Casey, B., & Santos, J. C. S. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00020 · Cited by 48
Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey
She, X., Liu, Y., Zhao, Y., He, Y., Li, L., Tantithamthavorn, C. K., Qin, Z., & Wang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748647 · Cited by 40
Watermarking techniques for large language models: a survey
Liang, Y., Xiao, J., Gan, W., & Yu, P. S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11474-6 · Cited by 65
Machine Learning for Translating Pseudocode to Python: A Comprehensive Review
Tiwari, S. P., Prasad, S., & Thushara, M. G. (2023) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs56967.2023.10142254 · Cited by 13
Variable Discovery with Large Language Models for Metamorphic Testing of Scientific Software
Tsigkanos, C., Rani, P., Müller, S., & Kehrer, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35995-8_23 · Cited by 23
Alahmadi, M. D., & Alshangiti, M. (2024) · Mathematics · doi:10.3390/math12071036 · Cited by 17
KubePlaybook: A Repository of Ansible Playbooks for Kubernetes Auto-Remediation with LLMs
Namrud, Z., Sarda, K., Litoiu, M., Shwartz, L., & Watts, I. (2024) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3629527.3653665 · Cited by 22
Evaluation of Large Language Model Generated Dialogues for an AI Based VR Nurse Training Simulator
Kapadia, N., Gokhale, S., Nepomuceno, A., Cheng, W., Bothwell, S., Mathews, M. A., Shallat, J. S., Schultz, C. M., & Gupta, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61041-7_13 · Cited by 20
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 82
LLMs and Stack Overflow discussions: Reliability, impact, and challenges
Silva, L. D., Samhi, J., & Khomh, F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112541 · Cited by 45
MPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers
Schneider, N., Kadosh, T., Hasabnis, N., Mattson, T. G., Pinter, Y., & Oren, G. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3624062.3624063 · Cited by 17
ToolCoder: Teach Code Generation Models to use API search tools
Zhang, K., Zhang, H., Li, G., Li, J., Li, Z., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.04032 · Cited by 12
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
“biology [42, 43], computer programming [64, 65], law [44– 46, 66], or robotics [47, 67, 68] among others.”
BackgroundStoryPal: Supporting Young Children's Dialogic Reading with Large Language Models
He, K., Gastón-Panthaki, A., Zhuo, D., Munsey, J., Zhang, M., & Warschauer, M. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728867 · Cited by 18
Correctness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 48
Diverse title generation for Stack Overflow posts with multiple-sampling-enhanced transformer
Zhang, F., Liu, J., Wan, Y., Yu, X., Liu, X., & Keung, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111672 · Cited by 14
Toward Automatic Tutoring of Math Word Problems in Intelligent Tutoring Systems
Arnau‐González, P., Serrano-Mamolar, A., Katsigiannis, S., Althobaiti, T., & Arevalillo‐Herráez, M. (2023) · IEEE Access · doi:10.1109/access.2023.3290478 · Cited by 11
Dong, X., Wang, Z., & Han, S. (2025) · Informatics · doi:10.3390/informatics12020051 · Cited by 23
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 33
Deploying Stateful Network Functions Efficiently using Large Language Models
Ghasemirahni, H., Farshin, A., Scazzariello, M., Chiesa, M., & Kostić, D. (2024) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3642970.3655836 · Cited by 14
Chain-of-Thoughts Prompting with Language Models for Accurate Math Problem-Solving
Fung, S. C. E., Wong, M.-F., & Tan, C.-W. (2023) · 2023 IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc60662.2023.10534945 · Cited by 9
Liu, Y., Wen, Z., Weng, L., Woodman, O., Yang, Y., & Chen, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3410523 · Cited by 18
CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning
Liu, Y., Tao, S., Zhao, X., Zhu, M. L., Ma, W., Zhu, J., Su, C., Hou, Y., Zhang, M., Zhang, M., et al. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00390 · Cited by 46
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J. H., & Omar, C. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689728 · Cited by 27
Fix the Tests: Augmenting LLMs to Repair Test Cases with Static Collector and Neural Reranker
Liu, J., Yan, J., Xie, Y., Yan, J., & Zhang, J. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00043 · Cited by 29
Context-aware prompting for LLM-based program repair
Li, Y., Cai, M., Chen, J., Xu, Y., Huang, L., & Li, J. K. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00512-w · Cited by 25
Less Is More: On the Importance of Data Quality for Unit Test Generation
Zhang, J., Hu, X., Gao, S., Xia, X., Lo, D., & Li, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715778 · Cited by 19
Jiang, P., Han, J., & Ahmed‐Kristensen, S. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101094 · Cited by 18
Tang, Z., Agarwal, M., Shypula, A., Wang, B., Wijaya, D. T., Chen, J., & Kim, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.119 · Cited by 35
Bibi, N., Maqbool, A., & Rana, T. A. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101865 · Cited by 16
Adapting Standard Retrieval Benchmarks to Evaluate Generated Answers
Arabzadeh, N., Bigdeli, A., & Clarke, C. L. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_26 · Cited by 27
Large Language Models are Catalyzing Chemistry EducationPreprint
Du, Y., Duan, C., Bran, A. M., Sotnikova, A., Qu, Y., Kulik, H. J., Bosselut, A., Xu, J., & Schwaller, P. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-h722v · Cited by 14
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616244 · Cited by 20
User Centric Evaluation of Code Generation Tools (Invited Paper)
Miah, T., & Zhu, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00022 · Cited by 25
MAGECODE: Machine-Generated Code Detection Method Using Large Language Models
Pham, H. N., Ha, H., Tong, V., Hoang, D., Tran, D., & Le, N. T. (2024) · IEEE Access · doi:10.1109/access.2024.3509987 · Cited by 18
Vulnerability Handling of AI-Generated Code - Existing Solutions and Open Challenges
Kaniewski, S., Holstein, D., Schmidt, F., & Heer, T. (2024) · Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset62544.2024.00026 · Cited by 12
Carreira-Munich, T., Paz-Marcolla, V., & Castro, R. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838994 · Cited by 12
LLM4TDG: test-driven generation of large language models based on enhanced constraint reasoning
Liu, J., Liang, R., Zhu, X., Zhang, Y., Liu, Y., & Liu, Q. (2025) · Cybersecurity · doi:10.1186/s42400-024-00335-4 · Cited by 20
LLM Assistance for Memory Safety
Mohammed, N., Lal, A., Rastogi, A., Sharma, R., & Roy, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00023 · Cited by 22
Analyzing the dependability of Large Language Models for code clone generation
Eagal, A., Stolee, K. T., & Ore, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112548 · Cited by 14
Prompt Alchemy: Automatic Prompt Refinement for Enhancing Code Generation
Ye, S., Sun, Z., Wang, G., Guo, L., Liang, Q., Li, Z., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3589634 · Cited by 28
Large Language Models for Computer-Aided Design: A Survey
Zhang, L., Le, B., Akhtar, N., Lam, S.-K., & Ngo, D. (2026) · ACM Computing Surveys · doi:10.1145/3787499 · Cited by 75
A MISLEADING GALLERY OF FLUID MOTION BY GENERATIVE ARTIFICIAL INTELLIGENCE
Kashefi, A. (2024) · Journal of Machine Learning for Modeling and Computing · doi:10.1615/jmachlearnmodelcomput.2024054191 · Cited by 12
The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code
Liu, X., Yin, D., Zhang, C., Feng, Y., & Zhao, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.574 · Cited by 37
Open AI Codex: An Inevitable Future?
Kumar, A., & Sharma, P. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.49048 · Cited by 8
RefactorScore: Evaluating Refactor Prone Code
Jesse, K., Kuhmünch, C., & Sawant, A. A. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3324613 · Cited by 10
On the Role of Large Language Models in Crowdsourcing Misinformation Assessment
Xu, J., Han, L., Sadiq, S. W., & Demartini, G. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31417 · Cited by 14
Alamsyah, N., Saparudin, S., & Kurniati, A. P. (2024) · IEEE Access · doi:10.1109/access.2024.3466270 · Cited by 8
HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 55
Evaluating Large Language Models for Code Generation: Assessing Accuracy, Quality, and Performance
Shehab, M. A., Wardat, M., Omari, S., & Jararweh, Y. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852439 · Cited by 19
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
Tinnes, C., Welter, A., & Apel, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00112 · Cited by 20
Securebert 2.0: Advanced Language Model for Cybersecurity Intelligence
Aghaei, E., Jain, S., Arun, P., & Sambamoorthy, A. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00071 · Cited by 21
Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test
Cai, X., Bai, X., Cui, Z., Xie, D., Fu, D., Yu, H., & Ren, Y. Q. (2026) · Automotive Innovation · doi:10.1007/s42154-025-00374-8 · Cited by 50
On the Impact of Language Selection for Training and Evaluating Programming Language Models
Katzy, J., Izadi, M., & van Deursen, A. (2023) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam59687.2023.00038 · Cited by 12
Automated Code Editing With Search-Generate-Modify
Liu, C., Cetin, P., Patodia, Y., Ray, B., Chakraborty, S., & Ding, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3376387 · Cited by 18
Honarvar, S., van der Wilk, M., & Donaldson, A. F. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989005 · Cited by 25
Do large language models know chemistry?Preprint
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2022) · ChemRxiv · doi:10.26434/chemrxiv-2022-3md3n · Cited by 8
Language Models Can Identify Enzymatic Active Sites in Protein SequencesPreprint
Teukam, Y. G. N., Dassi, L. K., Manica, M., Schwaller, P., Probst, D., & Laino, T. (2023) · SSRN Electronic Journal · doi:10.26434/chemrxiv-2021-m20gg-v3 · Cited by 8
Frankford, E., Höhn, I., Sauerwein, C., & Breu, R. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10662990 · Cited by 17
Can a Llama Be a Watchdog? Exploring Llama 3 and Code Llama for Static Application Security Testing
Curto, C., Giordano, D., Indelicato, D. G., & Patatu, V. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679444 · Cited by 18
Benchmarking large language models for automated labeling: The case of issue report classification
Colavito, G., Lanubile, F., & Novielli, N. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107758 · Cited by 18
Durán, F., Martínez, M., Lago, P., & Martínez‐Fernández, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112574 · Cited by 18
Leveraging Large Language Models for Auto-remediation in Microservices Architecture
Sarda, K. (2023) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C) · doi:10.1109/acsos-c58168.2023.00025 · Cited by 17
Prompt Fix: Vulnerability Automatic Repair Technology Based on Prompt Engineering
Liu, P., Wang, H., Zheng, C., & Zhang, Y. (2024) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc59896.2024.10556123 · Cited by 7
Large Language Models for Software Engineering: A Systematic Mapping Study
Görmez, M. K., Yılmaz, M., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_5 · Cited by 18
Top of the Heap: Efficient Memory Error Protection of Safe Heap Objects
Huang, K., Payer, M., Qian, Z., Sampson, J., Tan, G., & Jaeger, T. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690310 · Cited by 7
A Catalog of Data Smells for Coding Tasks
Vitale, A., Oliveto, R., & Scalabrino, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3707457 · Cited by 11
Automated Program Repair: Emerging Trends Pose and Expose Problems for Benchmarks
Renzullo, J., Reiter, P., Weimer, W. R., & Forrest, S. (2025) · ACM Computing Surveys · doi:10.1145/3704997 · Cited by 25
Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 18
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 31
Decoding Secret Memorization in Code LLMs Through Token-Level Characterization
Nie, Y., Wang, C., Wang, K., Xu, G., Xu, G., & Wang, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00229 · Cited by 26
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 29
Leveraging Large Language Model to Assist Detecting Rust Code Comment Inconsistency
Zhang, Y., Liu, Z., Feng, Y., & Xu, B. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695010 · Cited by 10
Code Red! On the Harmfulness of Applying Off-the-Shelf Large Language Models to Programming Tasks
Al-Kaswan, A., Deatc, S., Koç, B., van Deursen, A., & Izadi, M. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729380 · Cited by 11
Bouaziz, S., Hafiane, A., Canals, R., & Nedjaï, R. (2026) · Remote Sensing · doi:10.3390/rs18020289 · Cited by 17
Bridging Eras: Transforming Fortran Legacies into Python with the Power of Large Language Models
Pietrini, R., Paolanti, M., & Frontoni, E. (2024) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi60790.2024.10586058 · Cited by 12
Test-based and metric-based evaluation of code generation models for practical question answering
Kovalchuk, S. V., Fedrushkov, D., Lomshakov, V., & Aliev, A. (2023) · International Conference on Code Quality (ICCQ) · doi:10.1109/iccq57276.2023.10114665 · Cited by 7
A Composable Just-In-Time Programming Framework with LLMs and FBP
Vidan, A., & Fiedler, L. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363587 · Cited by 7
FLAME: A Small Language Model for Spreadsheet Formulas
Joshi, H., Ebenezer, A., Cambronero, J., Gulwani, S., Kanade, A., Le, V., Radiček, I., & Verbruggen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29197 · Cited by 27
Human perceiving behavior modeling in evaluation of code generation models
Kovalchuk, S. V., Lomshakov, V., & Aliev, A. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.24 · Cited by 12
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 10
Fuzzy-Assisted Contrastive Decoding Improving Code Generation of Large Language Models
Wang, S., Ding, L., Zhan, Y., Luo, Y., Liu, S., & Ding, W. (2025) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3575060 · Cited by 5
Model Editing for LLMs4Code: How Far are we?
Li, X., Wang, S., Li, S., Ma, J., Yu, J., Liu, X., Wang, J., Ji, B., & Zhang, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00049 · Cited by 23
Sooda, G., Hariharan, A., Patil, A., Vinod, A., Chandana, S., Manvith, L. B., Mandahas, N., Siri, H., Shobha, G., Deepamala, N., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3587056 · Cited by 5
Challenging Bug Prediction and Repair Models with Synthetic Bugs
Ibrahimzada, A. R., Chen, Y., Rong, R., & Jabbarvand, R. (2025) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam67354.2025.00021 · Cited by 32
Exploring the Mathematical Foundations of Generative AI Models: VAEs, GANs, Transformers, and DMs
Tera, S. P., Chinthaginjala, R., Natha, P., & Pau, G. (2025) · IEEE Access · doi:10.1109/access.2025.3647861 · Cited by 5
An Overview of Recent Advances in Natural Language Processing for Information Systems
O’Shaughnessy, D. D. (2026) · Applied Sciences · doi:10.3390/app16021122 · Cited by 14
Metadata
- Authors
- Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 1-10
- OpenAlex
- W4281763794
- Cited by
- 540
- References
- 19
- Citations by year
- 2026: 66 · 2025: 166 · 2024: 209 · 2023: 91 · 2022: 8
- FWCI
- 106.16
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- code generation, codices, GPT-J, GPT-Neo, programming languages, C programming language, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.