The DOI matches its DataCite record; 76 of the 92 references it cites were found in a registry or an index, and 1 are not checked yet.
Code Llama: Open Foundation Models for Code
Open accessAbstract
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given a new task, current LLMs consume and learn from the task data 'as-is' via finetuning or in-context learning [9, 10, 11, 12].”
From Self-Adapting Language Models (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code-based LLMs (e.g., CodeLlama(Roziere etal. 2023) and StarCoder(Li etal. 2023d; Lozhkov et al. 2024)) are particularly well suited to address this demand.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the time of writing, only the CodeLlama models (Rozi` ere et al., 2023) are able to handle the very long contexts necessary.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code Llama family are based upon Llama 2. It is pre-trained on a 500B token corpus with both code and natural language texts using the infilling objective.”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which is a 7B decoder-only model trained on code synthesis.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have achieved remarkable proficiency at generating code in mainstream programming languages such as Python and JavaScript, yet this proficiency does not transfer to logic programming.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Rozière et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CodeLLaMA-34B [42] is an aligned version of LLaMA-234B, which is fine-tuned with 500B tokens of code data.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [42]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“At the same time, different LLMs possess unique strengths and specialize in various tasks aspects.”
From Mixture-of-Agents Enhances Large Language Model Capabilities (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“There have been many previous works that have confirmed that LLMs do have good code comprehension capabilities (He et al. 2024; Luo et al. 2024; Roziere et al. 2023; Li et al. 2023d).”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, while numerous LMs have been implemented with a primary focus on code generation, most tend to underperform the best generalist LMs (Li et al., 2022; Le et al., 2022; Roziere et al., 2023; Austin et al., 2021; Xu et al., 2022).”
From Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation (2023) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Utilizing SWE-bench-train, we release two fine-tuned models, SWE-Llama 7 b and 13 b, based on the CodeLlama (Rozi` ere et al., 2023) model.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Prior studies showed that within the same LLM family, larger models are associated with better performance [59, 67, 73].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“The main experiments are performed on CODELLAMA (Roziere et al., 2023).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore CODELLAMA is more specialized in coding tasks thus yielding slightly lower PERPLEX-ITY (TARGET) score compared to MISTRAL despite their comparable sizes.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Among the LLMs tested, CODELLAMA (Roziere et al., 2023) benefits the most from good test case selection, hence we focus this work around CODELLAMA.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Moreover, both of these tools aim to maximise the acceptance rate of the shown completions”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Specifically, optimising only acceptance rate results in worse completions [32, 49].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Consistent with earlier research [32, 49], we acknowledge that these proxy metrics might not fully capture the usability of the interaction without potentially compromising it in some other way.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“and contains most of the significant features found in previous work [32].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Effective handling of long sequences is a major topic of research in transformer-based language modeling (Vaswani et al., 2017).”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“( b) Difference in BLEU scores for single line code completion of long context models compared to their respective base models before fine-tuning.”
MethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“To measure the toxicity of the generated outputs from each of the LLMs, we employ the default ToxiGen classifier, tuned on RoBERTa (Liu et al., 2019).”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 2×
“The VADER produces sentiment scores between -1 and 1, where a positive (negative) score indicates a positive (negative) sentiment towards the population mentioned in the prompt.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 11×
“ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
BackgroundMethodInfluentialSelf-citationKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“� 500B 20B 20B”
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1×
“Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 3×
“Collecting supervised data from human annotators or training from human feedback (Ouyang et al., 2022) is expensive for coding tasks as it requires input from professional developers.”
BackgroundMethodModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Research works printed but not linked: 6
- Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 Technical Report. arXiv:abs/2305.10403, 2023.
- Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: From low-quality model to high-quality dataset & model for summarization and paraphrasing. arXiv:abs/2305.16635, 2023.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
- Han Peng, Ge Li, Wenhan Wang, Yunfei Zhao, and Zhi Jin. Integrating tree path in transformer for code representation. In NeurIPS, pp. 9343-9354, 2021.
- Vighnesh Leonardo Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In NeurIPS, pp. 12058-12068, 2019.
- Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. arXiv:abs/2009.05617, 2020.
Cited by
Sorted by citations
Items 101–200 of 609
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 31
“They can provide general-purpose information on a variety of topics, including computer programming (Rozière et al. 2023), writing assistance (Touvron et al. 2023; OpenAI 2023c), cooking instructions (OpenAI 2023c) and many others.”
BackgroundQiskit Code Assistant: Training LLMs for generating Quantum Computing Code
Dupuis, N., Buratti, L., Vishwakarma, S., Forrat, A. V., Kremer, D., Faro, I., Puri, R., & Cruz-Benito, J. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691762 · Cited by 42
“For example, to cite just a few, StarCoder [6], Code Llama [7], and DeepSeek Coder [8] all show impressive performances on various code benchmarks.”
BackgroundLeveraging Local LLMs for Secure In-System Task Automation With Prompt-Based Agent Classification
Sriram, S., Karthikeya, C. S., Kumar, K. K. R. S., Vijayaraj, N., & Thangavel, M. (2024) · IEEE Access · doi:10.1109/access.2024.3505298 · Cited by 24
“Three models were examined – qwen2 [27], Mistral 7B [28], codeLlama 7B [29].”
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
“This methodology aligns with self-play and self-instruct approachs in neural program synthesis (Haluptzok et al., 2022; Rozière et al., 2023).”
MethodHands-on analysis of using large language models for the auto evaluation of programming assignments
Mohamed, K. M., Yousef, M., Medhat, W., Mohamed, E. H., Khoriba, G. A., & Arafa, T. (2024) · Information Systems · doi:10.1016/j.is.2024.102473 · Cited by 26
SECURE: Benchmarking Large Language Models for Cybersecurity
Bhusal, D., Alam, T., Nguyen, L., Mahara, A., Lightcap, Z., Frazier, R., Fieblinger, R., Torales, G. L., Blakely, B., & Rastogi, N. (2024) · Annual Computer Security Applications Conference · doi:10.1109/acsac63791.2024.00019 · Cited by 50
“The most notable language models in security are code-based LLMs such as CodeLlama [27], adapted to analyze and generate secure code.”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Steenhoek, B. J., Tufano, M., Sundaresan, N., & Svyatkovskiy, A. (2025) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest66595.2025.00011 · Cited by 72
“Foundational works in this direction, such as InstructGPT [34], RLAIF [6], LLAMA2, and Code LLama [38, 47], have proven the effectiveness of RLHF in refining LLM performance.”
Ni, B., Glaser, B., & Taheri-Mousavi, S. M. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01768-2 · Cited by 23
“…models 26,27 , trained on general language corpora have demonstrated intelligent capabilities in perceiving, understanding, and applying nat-ural languages 28,29 as well as coding languages 30,31 , thus assisting human researchers in accelerating and automating work fl ows in various tasks 32 – 34 .”
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
“Eventually, we select Alpaca-7b [157], Llama2-7b [158], and CodeLlama-13b-Instruct [91].”
BackgroundRLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
Self-citationHybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715004 · Cited by 53
“The first three models were downloaded from HuggingFace [60] and then deployed on our local machines while the last GPT-3.”
Enhancing large language models for text-to-testcase generation
Alagarsamy, S., Tantithamthavorn, C. K., Takerngsaksiri, W., Arora, C., & Aleti, A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112531 · Cited by 29
Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Wang, Z., Zhou, Z., Song, D., Huang, Y., Chen, S., Ma, L., & Zhang, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00180 · Cited by 87
“Recent work on code generation can then be roughly categorized into three groups: (1) developing high-quality training data [51]–[53], (2) developing better instruction fine-tuning techniques [54]–[56], and (3) developing better prompting strategies [57]–[65].”
Routerdc: Query-based router by dual contrastive learning for assembling large language models
Chen, S., Jiang, W., Lin, B., Kwok, J., & Zhang, Y. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2120 · Cited by 22
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00026 · Cited by 68
“Despite achieving good results with benchmark tasks, these models often struggle to generate real-world software that requires complex logic and detailed acceptance criteria, which are essential for practical applications [1], [2]. b) Software Engineering Benchmark: The rapid progress of Large…”
Macedo, M., Tian, Y., Cogo, F. R., & Adams, B. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652301 · Cited by 22
Resolving Code Review Comments with Machine Learning
Froemmgen, A., Austin, J., Choy, P., Ghelani, N., Kharatyan, L., Surita, G., Khrapko, E., Lamblin, P., Manzagol, P.-A., Revaj, M., et al. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3639477.3639746 · Cited by 21
SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidance
Vungarala, D., Alam, S., Ghosh, A., & Angizi, S. (2024) · IEEE International Conference on Rebooting Computing (ICRC) · doi:10.1109/icrc64395.2024.10937006 · Cited by 29
“We extensively evaluate the capability of LLMs in circuit design, including CodeLlama-70B-Instruct [21], Wizardcoder-33B-V1.”
Template-Guided Program Repair in the Era of Large Language Models
Huang, K., Zhang, J., Meng, X., & Liu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00030 · Cited by 30
“…role of patch generation in NTR, which demands advanced natural and programming language understanding for effective patch synthesis, we selected two of the foremost LLMs for code families currently available, CodeLlama [48] and StarCoder [49], as identified on the Big Code Models Leaderboard [50].”
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 27
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24
“The repository structure format provides the necessary file paths alongside the neighboring file names to maintain organizational information in the original codebase.”
BackgroundMethodL2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models
Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Ye, L., Yavuz, S., Xiong, C., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00705 · Cited by 49
FlexFL: Flexible and Effective Fault Localization With Open-Source Large Language Models
Xu, C., Liu, Z., Ren, X., Zhang, G., Liang, M., & Lo, D. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3553363 · Cited by 42
“Recently, due to the capability of large language models (LLMs) in language understanding, planning, and reasoning [13], [14], [15], [16], [17], [18], [19], [20], [21], LLMs have attracted significant attention from software debugging researchers with successful research done on patch generation…”
Rapid review: Growing usage of Multimodal Large Language Models in healthcare
Gupta, P., Zhang, Z., Song, M., Michalowski, M., Hu, X., Štiglic, G., & Topaz, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104875 · Cited by 24
Interleaving Static Analysis and LLM Prompting
Chapman, P. J., Rubio-González, C., & Thakur, A. (2024) · State Of the Art in Java Program Analysis · doi:10.1145/3652588.3663317 · Cited by 19
Generating Troubleshooting Trees for Industrial Equipment using Large Language Models (LLM)
Vidyaratne, L., Lee, X. Y., Kumar, A., Watanabe, T., Farahat, A., & Gupta, C. (2024) · IEEE Conference on Prognostics and Health Management · doi:10.1109/icphm61352.2024.10626823 · Cited by 21
“Moreover, the scope of LLMs extends beyond the previously mentioned tasks, as they demonstrate proficiency in solving mathematical problems [7] and crafting code [8].”
BackgroundChaudhuri, J., Thapar, D., Chaudhuri, A., Firouzi, F., & Chakrabarty, K. (2024) · IEEE Physical Assurance and Inspection of Electronics (PAINE) · doi:10.1109/paine62042.2024.10792717 · Cited by 19
Leveraging Large Language Model for Automatic Patch Correctness Assessment
Zhou, X., Xu, B., Kim, K., Han, D., Nguyen, H. H., Le-Cong, T., He, J., Le, X.-B. D., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3452252 · Cited by 29
“In addition to the Starcoder-7B model, we also implemented a version of LLM4PatchCorrect based on another popular 7B model, named Code LLamma-7B [47].”
MethodA Comprehensive Evaluation of Neural SPARQL Query Generation From Natural Language Questions
Diallo, P. A. K. K., Reyd, S., & Zouaq, A. (2024) · IEEE Access · doi:10.1109/access.2024.3453215 · Cited by 27
“We also conduct an empirical evaluation of four LLMs, namely Llama [11], Code Llama [12], Mistral 7B v0.”
MethodResultInfluentialChen, Y., Li, R., Yu, X., Zhao, Z., & Zhang, H. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2400468 · Cited by 28
“These models are highly proficient at generating human-like text (Brown et al., 2020; Wei et al., 2021; Bai et al., 2022; Le Scao et al., 2022; Touvron et al., 2023), catalyzing progress across various domains (Nijkamp et al., 2022; Rozière et al., 2023; Webb et al., 2023; Jin et al., 2024).”
Let’s Fix this Together: Conversational Debugging with GitHub Copilot
Bajpai, Y., Chopra, B., Biyani, P., Aslan, C., Coleman, D., Gulwani, S., Parnin, C., Radhakrishna, A., & Soares, G. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00011 · Cited by 18
Hou, S., Zhao, A., Liang, J., Shen, Z., & Wu, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113624 · Cited by 25
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17
Supersonic: Learning to Generate Source Code Optimizations in C/C++
Chen, Z., Fang, S., & Monperrus, M. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3423769 · Cited by 33
Code Gradients: Towards Automated Traceability of LLM-Generated Code
North, M., Atapour–Abarghouei, A., & Bencomo, N. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00038 · Cited by 18
“2) CodeLlama Instruct [33]: a variation of Meta’s general natural language Llama2 [34] model fine-tuned for coding.”
BackgroundRelax: Composable Abstractions for End-to-End Dynamic Machine Learning
Lai, R., Shao, J., Feng, S. M., Lyubomirsky, S., Hou, B., Lin, W., Ye, Z., Jin, H., Jin, Y., Liu, J., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716249 · Cited by 17
Self-citationGenerating vulnerability security fixes with Code Language Models
Bhandari, G. P., Gavrić, N., & Shalaginov, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107786 · Cited by 21
LLMEffiChecker : Understanding and Testing Efficiency Degradation of Large Language Models
Feng, X., Han, X., Chen, S. H., & Yang, W. M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664812 · Cited by 20
Distilling mathematical reasoning capabilities into Small Language Models
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106594 · Cited by 29
Chaudhuri, J., Thapar, D., Chaudhuri, A., Firouzi, F., & Chakrabarty, K. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3527382 · Cited by 16
“LLMs such as Llama-2 and Llama-3 are primarily trained on Python datasets [25].”
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 40
“Codegen Multi (6B) [18] 1 2022 CodeLlama (7B) [19] 2.5 2023 LlaMa 3.1 (8B / 70B) [20] 15.0 2024 StarCoder 2 (7B) [21] 3.5 2024 Gemma 2 (2B / 27B) [22] 2.0 / 13.0 2024 CodeGemma (7B) [23] 6.5 2024 Mistral (7B) [24] - -”
InfluentialLADAC: Large Language Model-driven Auto-Designer for Analog CircuitsPreprint
Liu, C., Liu, Y., Du, Y., & Du, L. (2024) · doi:10.36227/techrxiv.170473941.10097233/v1 · Cited by 17
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Si, C., Zhang, Y., Li, R., Yang, Z., Liu, R., & Yang, D. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.199 · Cited by 17
“LLMs trained on code, such as Codex (Chen et al., 2021), Star-Coder (Li et al., 2023), InCoder (Fried et al., 2022), CodeLlama (Rozière et al., 2023), and DeepSeek-Coder (Guo et al., 2024), enable a wave of programming support applications such as automatic code completion and infilling, and allowing users to chat wit…”
LTM: Scalable and Black-Box Similarity-Based Test Suite Minimization Based on Language Models
Pan, R., Ghaleb, T. A., & Briand, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3469582 · Cited by 19
“In this paper, we employ five pre-trained language models, namely CodeBERT [10], GraphCodeBERT [11], UniXcoder [12], StarEncoder [13], and CodeLlama [14].”
Can large language models generate geospatial code?
Hou, S., Shen, Z., Liang, J., Jiao, H., Zhao, A., Qing, Y., Peng, D., Gui, Z., Guan, X., Xiang, L., et al. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2535523 · Cited by 19
Human–robot collaborative visual inspection with Large Language Models
Tasneem, O., & Pieters, R. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103154 · Cited by 18
Hardware Phi-1.5B: A Large Language Model Encodes Hardware Domain Specific Knowledge
Fu, W., Li, S., Zhao, Y., Ma, H., Dutta, R., Zhang, X., Yang, K., Jin, Y., & Guo, X. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473927 · Cited by 19
“…that this approach will significantly enhance the model’s performance in comprehending and handling the aforementioned complex terminologies, surpassing the capabilities of existing general code language models (such as CodeLlama [22]) and natural language models (such as BERT [36] and GPT-2 [37]).”
BackgroundData preparation for Deep Learning based Code Smell Detection: A systematic literature review
Zhang, F., Zhang, Z., Keung, J., Tang, X., Yang, Z., Yu, X., & Hu, W. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112131 · Cited by 15
Sänger, M., Mecquenem, N. D., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., & Kosch, T. (2024) · GigaScience · doi:10.1093/gigascience/giae030 · Cited by 20
“However, there are many other general-purpose models available (e.g., PaLM-2 [44], BARD 15 or Llama-2 [19]) as well as models more specially designed for programming tasks (e.g., GitHub Copilot, Code Llama [87] or OpenAI Codex 16 ) publicly available and worth investigating.”
BackgroundAPIGen: Generative API Method Recommendation
Chen, Y., Gao, C., Zhu, M., Liao, Q., Wang, Y., & Xu, G. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00025 · Cited by 25
Kang, I., Woensel, W. V., & Seneviratne, O. W. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-63592-2_11 · Cited by 18
A Vision Check-up for Language Models
Sharma, P., Shaham, T. R., Baradad, M., Rodriíuez-Muñoz, A., Duggal, S., Isola, P., Torralba, A., & Fu, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01366 · Cited by 15
What If the TV was off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
Zhang, L., Zhai, X., Zhao, Z., Zong, Y., Wen, X., & Zhao, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02064 · Cited by 19
“CodeLlama [35] shows a relatively moderate performance decrease when counterfactuals are introduced, suggesting some resilience but still a noticeable drop.”
COFFE: A Code Efficiency Benchmark for Code Generation
Peng, Y., Wan, J., Li, Y., & Ren, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715727 · Cited by 15
“Recently, decoder-only models such as Codex [9], CodeGen [59, 60], InCoder [22], CodeGeeX [93], SantaCoder [2], StarCoder [42, 52], WizardCoder [53], CodeLlama [71], MagicCoder [84], DeepSeek-Coder [26] show superior performance than encoder-decoder models on code generation.”
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
BugWhisperer: Fine-Tuning LLMs for SoC Hardware Vulnerability Detection
Tarek, S., Saha, D., Saha, S. K., & Farahmandi, F. (2025) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts65138.2025.11022958 · Cited by 14
“Proprietary models, including well-known solutions such as Gemini and GPT [13], typically demonstrate superior detection accuracy compared to smaller open-source alternatives like LLama [14], Mistral, and CodeLlama [15].”
MetaIndux-PLC: A Control Logic-Guided LLM for PLC code generation in industrial control systems
Ren, L., Wang, H., Dong, J., Wang, H., Liu, S., Laili, Y., & Li, Z. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113673 · Cited by 14
On the Effectiveness of Large Language Models in Statement-level Code Summarization
Zhu, J., Miao, Y., Xu, T., Zhu, J., & Sun, X. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00030 · Cited by 14
“5 [20] that were developed by OpenAI, as well as the open-source LLMs CodeLlama [21] and StarChat [22].”
BackgroundMethodExploring Parameter-Efficient Fine-Tuning of Large Language Model on Automated Program Repair
Li, G., Zhi, C., Chen, J., Han, J., & Deng, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695066 · Cited by 29
Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design
Nakkab, A., Zhang, S. Q., Karri, R., & Garg, S. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740247 · Cited by 14
ChatVis: Automating Scientific Visualization with a Large Language Model
Mallick, T., Yildiz, O., Lenz, D., & Peterka, T. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00014 · Cited by 17
“8B [20], and other models specifically designed for code generation, such as Code LLaMA [17] and Codegemma [19].”
Pytester: Deep reinforcement learning for text-to-testcase generation
Takerngsaksiri, W., Charakorn, R., Tantithamthavorn, C. K., & Li, Y.-F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112381 · Cited by 27
INTERTRANS: Leveraging Transitive Intermediate Translations to Enhance LLM-Based Code Translation
Macedo, M., Tian, Y., Nie, P., Cogo, F. R., & Adams, B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00236 · Cited by 14
“To evaluate the effectiveness of I NTER T RANS , we conducted experiments using three code LLMs (Code Llama [33], Magicoder [38], and StarCoder2 [25]) on 4,926 translation problems sourced from three datasets, i.e., CodeNet [32], HumanEval-X [42], and TransCoder [34].”
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
Gim, I., Chen, G., Lee, S.-S., Sarda, N., Khandelwal, A., & Zhong, L. (2023) · arXiv · doi:10.48550/arxiv.2311.04934 · Cited by 18
Jin, X., & Lin, Z. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660769 · Cited by 17
“While the public datasets are centered on Java methods, we expand the scope to build our own evaluation dataset to include code in Python, JavaScript, Ruby, Go, and PHP ( i.e. , the five most popular programming languages [Roziere et al. 2023; Yang et al. 2022]).”
Structured knowledge-based causal discovery: Agentic streams of thought
Meier, S., Raut, P. N., Mahr, F., Thielen, N., Franke, J. E., & Risch, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104202 · Cited by 14
Generative AI for Self-Healing Systems
Khlaisamniang, P., Khomduean, P., Saetan, K., & Wonglapsuwan, S. (2023) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp60301.2023.10354608 · Cited by 13
“• Testing Agent : The Testing Agent leverages Code Llama [17] to automatically create unit tests for the code generated by GPT-4 [15].”
BackgroundMethodThe Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Leiter, C., Opitz, J., Deutsch, D., Gao, Y., Dror, R., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.10 · Cited by 19
How people prompt generative AI to create interactive VR scenes
Manesh, S. A., Zhang, T., Onishi, Y., Hara, K., Bateman, S., Li, J., & Tang, A. (2024) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 13
Towards LLM-Assisted System Testing for Microservices
Almutawa, M., Ghabrah, Q., & Canini, M. (2024) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw63686.2024.00011 · Cited by 19
On Large Language Models’ Resilience to Coercive Interrogation
Zhang, Z., Shen, G., Tao, G., Cheng, S., & Zhang, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00208 · Cited by 20
“Table 4 shows our evaluation of L INT on Codellama-13B [49], a suite of LLM tailored for coding tasks.”
MethodCoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios
Zeng, Z., Wang, Y., Xie, R., Ye, W., & Zhang, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652115 · Cited by 16
Generating PLC Code with Universal Large Language Models
Tran, K., Zhang, J., Pfeiffer, J., Wortmann, A., & Wiesmayr, B. (2024) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa61755.2024.10711113 · Cited by 13
“This success has led to specifically tailored LLMs for generating GPL code, such as Github Copilot [7] or Code Llama [8].”
MethodBhattarai, M., Santos, J. E., Jones, S., Biswas, A., Alexandrov, B., & O'Malley, D. (2024) · 2024 IEEE High Performance Extreme Computing Conference (HPEC) · doi:10.1109/hpec62836.2024.10938485 · Cited by 13
“We conducted extensive experiments using various open language models, including Starcoder [7], Llama3-70B In-struct [8], CodeLlama-34B Instruct [9], Granite-34B Code Instruct [10], Mistral 8x22B [11] and Codestral [12] and commercial models such as GPT-3.”
Unraveling the Potential of Large Language Models in Code Translation: How Far are We?
Tao, Q., Yu, T., Gu, X., & Shen, B. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00046 · Cited by 13
Type-Constrained Code Generation with Language Models
Mündler-Sasahara, N., He, J.-X., Wang, H., Sen, K., Song, D., & Vechev, M. T. (2025) · Proc. ACM Program. Lang. · doi:10.1145/3729274 · Cited by 77
What You See is What You Get: Attention-Based Self-Guided Automatic Unit Test Generation
Yin, X., Ni, C., Xu, X., & Yang, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00105 · Cited by 21
LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models
Ma, Z., Kim, D. J., & Chen, T.-H. P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00103 · Cited by 39
“Large Language Models (LLMs), primarily built on the transformer architecture [4, 7, 34], have significantly advanced the field of natural language processing (NLP).”
Fixing Large Language Models' Specification Misunderstanding for Better Code Generation
Tian, Z., Chen, J.-J., & Zhang, X.-Y. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00108
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12
“Additionally, we are connected to the line of code execution since we also leverage it as an important retrieval source. generation (Li et al., 2022; Nijkamp et al., 2022; Li et al., 2023b; Roziere et al., 2023; Wei et al., 2023).”
Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., AlShikh, W., & Salakhutdinov, R. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73113-6_10 · Cited by 12
Arithmetic Reasoning with LLM: Prolog Generation & Permutation
Yang, X., Chen, B., & Tam, Y.-C. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.61 · Cited by 12
Enhancing vulnerability detection by fusing code semantic features with LLM-generated explanations
Tian, Z., Li, M., Sun, J., Chen, Y., & Chen, L. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103450 · Cited by 15
Large Language Models for Combinatorial Optimization: A Systematic Review
Ros, F. D., Soprano, M., Gaspero, L. D., & Roitero, K. (2026) · ACM Computing Surveys · doi:10.1145/3801961 · Cited by 40
Zhang, J., Hu, X., Xia, X., Cheung, S.-C., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745765 · Cited by 13
“To evaluate the effectiveness of our approach, we compare it with six learning-based baselines (i.e., CodeBERT [18], AthenaTest [63], StarCoder [33], CodeLlama-7B [56]), GPT-3.”
InfluentialLykousas, N., & Patsakis, C. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103974 · Cited by 13
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 13
The Block Copolymer Phase Behavior Database
Rebello, N. J., Arora, A., Mochigase, H., Lin, T.-S., Shi, J., Audus, D. J., Muckley, E. S., Osmani, A., & Olsen, B. D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00242 · Cited by 12
Toward Hardware Security Benchmarking of LLMs
Afsharmazayejani, R., Shahmiri, M. M., Link, P., Pearce, H. A., & Tan, B. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691745 · Cited by 12
“• CodeLlama: a fine-tuned version of Llama2 [22].”
ECG: Augmenting Embedded Operating System Fuzzing via LLM-Based Corpus Generation
Zhang, Q., Shen, Y., Liu, J., Xu, Y., Shi, H., Jiang, Y., & Chang, W. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3447220 · Cited by 12
“LLMs [16], [17], [18] are artificial neural networks that have excellent abilities in understanding and generating human-like text.”
BackgroundSelf-citationSuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants
Moghani, M., Doorenbos, L., Panitch, W. C.-H., Huver, S. D., Azizian, M., Goldberg, K., & Garg, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802053 · Cited by 12
“CodeLlama [15] calls many undefined functions, such as get_end_effector_pose() , despite the end-effector pose being given in the prompt.”
Assessing and improving syntactic adversarial robustness of pre-trained models for code translation
Yang, G., Zhou, Y., Zhang, X., Chen, X., Han, T., & Chen, T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107699 · Cited by 12
Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry
Gurioli, A., Gabbrielli, M., & Zacchiroli, S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00044 · Cited by 18
“(Self-hosted open-weight LLMs, like Code Llama [39] and StarCoder [24] mitigate this issue.)”
InfluentialLLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 17
“These prompts were then inputted into seven advanced code LLMs, including CodeLlama [5] and GPT-3.”
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 16
Improving Retrieval-Augmented Deep Assertion Generation via Joint Training
Zhang, Q., Fang, C., Zheng, Y., Qian, R., Yu, S., Zhao, Y., Zhou, J., Yang, Y., Zheng, T., & Chen, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3545970 · Cited by 11
“We include AG-RAG against six prior AG approaches, i.e., ATLAS, IR ar , RA , RA NNadapt , Integration and E DIT AS, two benchmarks, i.e., Data old and Data new , and three evaluation metrics, i.e., accuracy, BLEU and CodeBLEU.”
Metadata
- Authors
- Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., & Synnaeve, G.
- Affiliations
- École nationale des ponts et chaussées (FR)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.12950
- OpenAlex
- W4386185625
- Cited by
- 531
- References
- 92
- Citations by year
- 2026: 65 · 2025: 232 · 2024: 216 · 2023: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- CodeLlama, code large language models, Llama 2, HumanEval, MBPP, code generation benchmarks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
