A systematic evaluation of large language models of code
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
From Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design (2025) · cites this paper as [14]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Research works printed but not linked: 13
- Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. 245–256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. 245–256.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
- Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853. · arXiv
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harri Edwards Yura Burda Nicholas Joseph and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
- Zhangyin Feng , Daya Guo , Duyu Tang , Nan Duan , Xiaocheng Feng , Ming Gong , Linjun Shou , Bing Qin , Ting Liu , and Daxin Jiang . 2020 . Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. · arXiv
- Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. · arXiv
- Aditya Kanade , Petros Maniatis , Gogul Balakrishnan , and Kensen Shi . 2020 . Learning and evaluating contextual embedding of source code . In International Conference on Machine Learning. 5110–5121 . Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning. 5110–5121.
- Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293. · arXiv
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9.
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. · arXiv
- Lewis Tunstall , Leandro von Werra , and Thomas Wolf . 2022. Natural Language Processing with Transformers. " O’Reilly Media , Inc .". Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. " O’Reilly Media, Inc.".
- Daniel Zügner , Tobias Kirschstein , Michele Catasta , Jure Leskovec , and Stephan Günnemann . 2021 . Language-Agnostic Representation Learning of Source Code from Structure and Context . In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. https://openreview.net/forum?id=Xh5eMZVONGF
Cited by
Items 501–567 of 567
Bugs in AI-Generated Code - Understanding Bug Patterns and Possible Fix Strategies
Gao, R. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00098 · Cited by 1
Automated Update of Android Deprecated API Usages With Large Language Models
Mahmud, T., Duan, B., Che, M., Yasmin, A., Ngu, A. H. H., & Yang, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3627897 · Cited by 9
Basu, R., Das, A., Gupta, P., Sen, S., Sadhukhan, B., & Das, N. (2025) · International Conference on Intelligent Algorithms for Computational Intelligence Systems (IACIS) · doi:10.1109/iacis65746.2025.11211373
Zhu, Z., Peng, L., Wang, Y., Li, Y., & Long, X. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761420
T 3 : Multi-level Tree-based Automatic Program Repair with Large Language Models
Liu, Q., Bu, X., Yan, Z., & Li, R. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228000 · Cited by 1
Very-Long-Distance Dependency Capturing Evaluation via Language Modeling Based on Gender Consistency
Xu, H., Liang, Z., van Genabith, J., Xiong, D., Zan, H., Liu, Q., & Zhang, T. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3352-7_3
Taracatac, A. C., & Fajardo, A. C. (2025) · International Conference on Control, Robotics and Informatics (ICCRI) · doi:10.1109/iccri67201.2025.11262379
The price of precision: the cost of preprocessing for automated code revision in code review
Pirouzkhah, S., Rani, P., Sovrano, F., Hellendoorn, V. J., & Bacchelli, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10781-4 · Cited by 1
Self-citationGenerating Test Cases with Large Language Model: Frameworks and Challenges
Hosseini, M. R., Janjua, N. K., Mohsin, A., & Rahman, S. U. (2025) · IEEE International Conference on e-Business Engineering (ICEBE) · doi:10.1109/icebe68123.2025.00041 · Cited by 1
Large language model-based code assistance
Wang, M., Liu, J., Sun, X., Liu, Z., Zhang, Y., & Wang, S. (2025) · International Conference on Intelligent Computing and System Simulation (ICSS) · doi:10.1117/12.3092066
Billah, M. M., Rahman, M. S., & Roy, B. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756977 · Cited by 2
Enhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
Context-Aware CodeLLM Eviction for AI-assisted Coding
Thangarajah, K., Chen, B., Chang, S., & Hassan, A. E. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00271 · Cited by 4
Enhancing Code Quality Using Pre-trained Language Models: A Fine-Tuning Approach for Code Refinement
Magar, A., Joshi, K., Shah, P., & Mishra, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-95-3495-1_21
Vanmala, & Dubey, A. K. (2025) · International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN) · doi:10.1109/icrcicn68210.2025.11364831
Fine-tuning LLMs for answer set programming
Coppolillo, E., Calimeri, F., Manco, G., Perri, S., & Ricca, F. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-01017-4 · Cited by 2
Leveraging Large Language Models (LLMs) for Service Optimization in Beyond 5G (B5G) Networks
Muralidharan, S., Woo, W., & Lee, N. (2025) · Springer eBooks · doi:10.1007/978-3-032-06418-9_15
CodeClarity: A Framework for Monitoring Code Quality Erosion with LLMs
Bhogavalli, S., Sonani, R., Yalamancheli, H., & Brahmandam, B. A. (2025) · IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) · doi:10.1109/iemcon67450.2025.11381048
Challenges of deploying code embeddings: an industrial case study on method name generation
Fein, B., Jungwirth, M., Fraser, G., & Kandlinger, F. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00592-2 · Cited by 3
R$$^2$$ComSync: improving code-comment synchronization with in-context learning and reranking
Yang, Z., Lin, H., Yu, X., Keung, J., Liu, S., Chan, P. Y. P., Sun, Y., & Zhang, F. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10800-4 · Cited by 2
Explainable Retrieval and Graph Augmented Code Vulnerability Repair Recommendations
Amoah, A. A., & Liu, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402509 · Cited by 1
Assessing AI-Based Code Generation using Code Comments
Sekar, T., & ElKharboutly, R. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418477
The dual-encoder detector for AI-generated go code
Liao, G., Hou, Y., Chai, L., Yue, Q., Zhang, Q., Han, W., & Ou, W. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00626-5 · Cited by 1
Investigating Transformer-Based Model for Long Document Summarization
Khan, R., & Sharma, S. (2025) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet67282.2025.11430300
Omari, S., Kathet, P., Wardat, M., & Shehab, M. A. (2026) · Cluster Computing · doi:10.1007/s10586-026-05947-8
Xu, X. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20268403009 · Cited by 1
Culture Clashes: LLM Support in the Engineering of Safety-Critical Systems
Thomas, C., & Wagner, M. (2026) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2026-01-0110
Integrating AI Literacy, Secure Coding, and Critical Evaluation
Atul Kumar Dnyaan Prasad Global University, Namazov, G., Nadira, A., Rizamat, S., Bobur, M., Mohichekhra, K., & Ozoda, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8450-4.ch008 · Cited by 1
PACGBI: A CI/CD Pipeline for LLM-assisted Code Generation of Web Front End
Sarschar, M., Zhang, G., & Nowak, A. (2025) · World Symposium on Software Engineering (WSSE) · doi:10.1145/3779657.3779672
Contreras, A., Guerra, E., & de Lara, J. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19050-5_7 · Cited by 1
Benchmarking AI Models for Automated Code Generation and Testing Using HumanEval
Manzoor, M., Rehman, M. U., Kazim, M., Deka, L., Khattak, A. A., & Usmani, I. A. (2026) · Global Conference on Wireless and Optical Technologies (GCWOT) · doi:10.1109/gcwot69191.2026.11499481
The Impact of LLMs on Software Development
Vasanthakumar, S., & Wickramarachchi, R. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499962 · Cited by 1
Bias Ahead: Sensitive Prompts as Early Warnings for Fairness in Large Language Models
Voria, G., de Lucía, M. E., Raia, A., Lucia, A. D., Catolino, G., & Palomba, F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00023
Taupe, R., & Falkner, A. (2026) · SpringerBriefs in computer science · doi:10.1007/978-3-032-17163-4_6
VulInject: Multi-Type Samples Generation for Learning-based Vulnerability Detection
Wu, Y. H., Li, Z., Li, Z. L., Yang, Z., Wu, Y., Zou, D., & Jin, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3816249
Exploring and improving knowledge distillation for pre-trained code models
Sun, W., Wu, R., Li, H., Fu, Y., Yu, M., & Yan, M. L. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10880-w · Cited by 1
Li, Q., Jiang, G., He, P., Liu, W., Dong, J., Sun, Y., & Wu, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10886-4 · Cited by 2
LiMO: Line-level Machine-generated Code Detection for Diverse Programming Languages
Kim, T., Jang, H., Kim, S., Jin, Z., Kim, S., & Bak, J. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779718
Shehab, M. A., Omari, S., Wardat, M., & Jararweh, Y. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09766-3
Using Adaptive Activation Functions in Pre-Trained Artificial Neural Network Models
Bodyanskiy, Y. V., & Kostiuk, S. (2023) · International Conference on Information Control Systems & Technologies (ICST) · doi:10.55056/ceur-ws.org/vol-3513/paper8.pdf · Cited by 2
Compiling Code LLMs into Lightweight Executables
Shi, J., He, J., Yang, Z., Yang, C., Klymenko, M. V., Hoang, T., Xu, X., Xing, Z., & Lo, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808196 · Cited by 1
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 7
SOSecure: The Wisdom of the Crowd for Safer AI-Generated Code
MUKHERJEE, M., & Hellendoorn, V. J. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814903 · Cited by 1
Self-citationBecker, N., Mammadov, T., & Zeller, A. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814888 · Cited by 3
Do LLMs Dream of Energy-Efficient Code?
Bernardo, A. D., Capozzi, G., Rosa, P. D., D’Elia, D. C., Querzoni, L., Luna, G. A. D., & Schiavoni, V. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788704
Anh, L. H. N., Luong, H. V. D., & Vũ, V. A. (2026) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3795095.3805199 · Cited by 1
Large Language Model-Based Data Querying and Analysis for Manufacturing Industry
Radhakrishnan, R., Beden, S., Shahid, Z., Giannetti, C., & Beckmann, A. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2864-5_29
Reza, S. M. I., Moon, I. T., Fahim, M. F. S., & Sheikh, M. T. (2026) · Journal of Cyber Security Technology · doi:10.1080/23742917.2026.2700412
GONets: A First-Look Into GitHub Organisation Networks
Dutta, H. S., Khan, B., Shah, P. M., & Nanavati, A. A. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808608
Leveraging Large Language Models for Trustworthiness Assessment of Web Applications
Yarotskyi, O., Pereira, J. D., & Campos, J. R. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00054
Understanding Vibe Coding: A Phenomenological Investigation of Flow-Like Human-AI Collaboration
Guriță, A.-E., & Vatavu, R.-D. (2026) · International Conference on Human-AI Interaction and Experience Design (HAXD) · doi:10.1109/haxd70072.2026.11620988
D3Fuzz: Dynamic, directed and decaying LLM-driven fuzzing for deep learning libraries
Sun, J., Dong, J., Zhang, H., Tian, Z., & Zhang, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108289
RL–LLMfuzzer: reinforcement learning–guided dual–model differential fuzzing for compiler testing
Li, D., Li, Y., Zhao, B., Xu, J., Yao, J., Wang, N., & Liu, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01115-5
Foundation models as oracles for refactoring correctness detection
Gheyi, R., Melo, R., Oliveira, J., Ribeiro, M., & Fonseca, B. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10951-y · Cited by 2
TAILOR: Token-Aware Partitioning and Routing for Edge–Cloud Transformer Inference
Huang, X., Liu, R. R., & Wu, J. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661040
Lim, S., Kwon, H., & Kim, M. (2026) · IEEE Access · doi:10.1109/access.2026.3730474 · Cited by 2
Improving code completion interpretability with concept-level intervention
Fan, K., Shen, Z., Li, Y.-X., Chen, X., Zhuang, Y., Ge, J., & Luo, B. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108309
Cipriano, B. P., & Ferreira, L. S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39106-3_17
Software maintainability prediction using static code metrics and machine learning
Şengül, F., & Yürüm, O. R. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09789-w
English or Spanish Prompts: Effect of Language Prompt on LLM Code Generation Quality
Mora-Monge, J. A., Delgado-Mora, M., Meléndez-Aguilar, D., Gonzalez-Villanueva, R., & Hidalgo‐Céspedes, J. (2026) · IEEE Jornadas Costarricenses de Investigación en Computación e Informática (JoCICI) · doi:10.1109/jocici69279.2026.11701728
LLMCloc: Compiler bug localization via large language model-based root cause analysis
Shi, B., Lu, W., Tang, J., Huang, Z., & Li, W. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108352
A Survey of Code-to-Code Translation: From Statistical Rule-Based Systems to LLM Agents
Zhang, Z., Wen, S., Yan, Z., Zhu, B., Shen, C., & Guo, H. (2026) · Algorithms · doi:10.3390/a19100857
Lightweight Language Models for Encoding ASP from Natural Language
Kareem, I., Borroto, M., Ricca, F., & Vitale, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38320-4_18
Towards Tracing the Quality of Code Generated by LLM Back to the Training Code
Wang, C., & Huang, Y. F. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28454-9_33
Serra, C. B., de Classe, T. M., & Sousa, H. P. (2026) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames.2026.25182
Repairing Code Defects Through Multiple LLM Collaboration and Failure Information Capturing
Han, J., Zhou, Z., Cao, G., Ban, Q., Cheng, Y., & Yan, B. (2026) · Software · doi:10.3390/software5040047
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · arXiv · doi:10.48550/arxiv.2404.10100
“There has been a surge in research around training LLMs over programming language artifacts in just the last couple of years [1], [2], [3], [4], [5].”
Background
Metadata
- Authors
- Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
- Publisher
- ACM
- Published
- 2022-06-13
- Pages
- 1-10
- OpenAlex
- W4281763794
- Cited by
- 540
- References
- 19
- Citations by year
- 2026: 66 · 2025: 166 · 2024: 209 · 2023: 91 · 2022: 8
- FWCI
- 106.16
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- code generation, codices, GPT-J, GPT-Neo, programming languages, C programming language, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.