The DOI matches its DataCite record; 76 of the 92 references it cites were found in a registry or an index, and 1 are not checked yet.
Code Llama: Open Foundation Models for Code
Open accessAbstract
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given a new task, current LLMs consume and learn from the task data 'as-is' via finetuning or in-context learning [9, 10, 11, 12].”
From Self-Adapting Language Models (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code-based LLMs (e.g., CodeLlama(Roziere etal. 2023) and StarCoder(Li etal. 2023d; Lozhkov et al. 2024)) are particularly well suited to address this demand.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the time of writing, only the CodeLlama models (Rozi` ere et al., 2023) are able to handle the very long contexts necessary.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code Llama family are based upon Llama 2. It is pre-trained on a 500B token corpus with both code and natural language texts using the infilling objective.”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which is a 7B decoder-only model trained on code synthesis.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have achieved remarkable proficiency at generating code in mainstream programming languages such as Python and JavaScript, yet this proficiency does not transfer to logic programming.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Rozière et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CodeLLaMA-34B [42] is an aligned version of LLaMA-234B, which is fine-tuned with 500B tokens of code data.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [42]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“At the same time, different LLMs possess unique strengths and specialize in various tasks aspects.”
From Mixture-of-Agents Enhances Large Language Model Capabilities (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“There have been many previous works that have confirmed that LLMs do have good code comprehension capabilities (He et al. 2024; Luo et al. 2024; Roziere et al. 2023; Li et al. 2023d).”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, while numerous LMs have been implemented with a primary focus on code generation, most tend to underperform the best generalist LMs (Li et al., 2022; Le et al., 2022; Roziere et al., 2023; Austin et al., 2021; Xu et al., 2022).”
From Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation (2023) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Utilizing SWE-bench-train, we release two fine-tuned models, SWE-Llama 7 b and 13 b, based on the CodeLlama (Rozi` ere et al., 2023) model.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Prior studies showed that within the same LLM family, larger models are associated with better performance [59, 67, 73].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“The main experiments are performed on CODELLAMA (Roziere et al., 2023).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore CODELLAMA is more specialized in coding tasks thus yielding slightly lower PERPLEX-ITY (TARGET) score compared to MISTRAL despite their comparable sizes.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Among the LLMs tested, CODELLAMA (Roziere et al., 2023) benefits the most from good test case selection, hence we focus this work around CODELLAMA.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Moreover, both of these tools aim to maximise the acceptance rate of the shown completions”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Specifically, optimising only acceptance rate results in worse completions [32, 49].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Consistent with earlier research [32, 49], we acknowledge that these proxy metrics might not fully capture the usability of the interaction without potentially compromising it in some other way.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“and contains most of the significant features found in previous work [32].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Effective handling of long sequences is a major topic of research in transformer-based language modeling (Vaswani et al., 2017).”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3×
“( b) Difference in BLEU scores for single line code completion of long context models compared to their respective base models before fine-tuning.”
MethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“To measure the toxicity of the generated outputs from each of the LLMs, we employ the default ToxiGen classifier, tuned on RoBERTa (Liu et al., 2019).”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 2×
“The VADER produces sentiment scores between -1 and 1, where a positive (negative) score indicates a positive (negative) sentiment towards the population mentioned in the prompt.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 11×
“ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
BackgroundMethodInfluentialSelf-citationKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1×
“� 500B 20B 20B”
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1×
“Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 3×
“Collecting supervised data from human annotators or training from human feedback (Ouyang et al., 2022) is expensive for coding tasks as it requires input from professional developers.”
BackgroundMethodModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Research works printed but not linked: 6
- Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 Technical Report. arXiv:abs/2305.10403, 2023.
- Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: From low-quality model to high-quality dataset & model for summarization and paraphrasing. arXiv:abs/2305.16635, 2023.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
- Han Peng, Ge Li, Wenhan Wang, Yunfei Zhao, and Zhi Jin. Integrating tree path in transformer for code representation. In NeurIPS, pp. 9343-9354, 2021.
- Vighnesh Leonardo Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In NeurIPS, pp. 12058-12068, 2019.
- Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. arXiv:abs/2009.05617, 2020.
Cited by
Sorted by citations
Items 201–300 of 609
Correctness assessment of code generated by Large Language Models using internal representations
Bui, T.-D., Vu, T. T., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112570 · Cited by 39
A comprehensive evaluation of large language models in mining gene relations and pathway knowledge
Azam, M., Chen, Y., Arowolo, M. O., Liu, H., Popescu, M., & Xu, D. H. (2024) · Quantitative Biology · doi:10.1002/qub2.57 · Cited by 26
“In the open ‐ source LLMs, Codellama ‐ 34 is known for its efficiency in instruction ‐ based tasks [11], and the Wiz-ardLM series excels in knowledge retrieval and processing [12].”
Demo: Towards a Conversational LLM-Based Voice Assistant for Transportation Applications
Jafarnejad, S., Berthe-Pardo, A., & Frank, R. (2024) · IEEE Vehicular Networking Conference (VNC) · doi:10.1109/vnc61989.2024.10575993 · Cited by 11
“NexusRaven is instruction-tuned on Meta’s CodeLlama-13B-instruct [7].”
Generalizable clinical note section identification with large language models
Zhou, W., & Miller, T. A. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae075 · Cited by 11
LASSI: An LLM-Based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes
Dearing, M. T., Tao, Y., Wu, X., Lan, Z., & Taylor, V. (2024) · IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops) · doi:10.1109/clusterworkshops61563.2024.00029 · Cited by 23
“Many existing commercial and open-source LLMs, such as GPT-4 [3], Codestral [4], StarCoder [5], and Code Llama [6], are trained on massive collections of shared code primarily developed in widely-used programming languages like Python and JavaScript.”
MethodComparative analysis of generative LLMs for labeling entities in clinical notes
del Moral-González, R., Gómez-Adorno, H., & Ramos-Flores, O. (2025) · Genomics & Informatics · doi:10.1186/s44342-024-00036-x · Cited by 13
“There is also a code fine-tuned version of the Llama 2 model [20], which is called Code-Llama and has further training on a code-heavy corpus.”
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 21
Lu, H., Peng, H., Nan, G., Cui, J., Wang, C., Jin, W., Wang, S., Pan, S., & Tao, X. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3583552 · Cited by 12
“We conduct experiments using BLEURT-sum, BLEU, ROUGE, and METEOR (the usability of BLEURT-sum is substantiated in the referenced paper) to carry out experiments on M AL - SIGHT , WizardLM [56], and Code Llama [57], etc.”
Evaluating quantized Large Language Models for code generation on low-resource language benchmarks
Nyamsuren, E. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101351 · Cited by 18
Mitchell, M. (2024) · doi:10.21428/e2759450.2bb20e3c · Cited by 13
Zhang, Y., Zhao, X., Wang, Z. Z., Yang, C., Wei, J., & Wu, T. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.430 · Cited by 11
Learning to Generate Structured Code Summaries From Hybrid Code Context
Zhou, Z., Li, M., Yu, H., Fan, G., Yang, P., & Huang, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3439562 · Cited by 10
“Recently, large language models (LLMs) pre-trained on massive unlabelled corpora of code are also skilled in code summarization [18], [19], [20], [21], [22].”
BackgroundMethodInfluentialBridging the Gap: A Study of AI-based Vulnerability Management between Industry and Academia
Wan, S., Saxe, J., Gomes, C., Chennabasappa, S., Rath, A., Sun, K., & Wang, X. (2024) · IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · doi:10.1109/dsn-s60304.2024.00032 · Cited by 20
“Furthermore, constructing security Supervised Fine-Tuning (SFT) datasets also presents an appealing opportunity [31].”
ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models
Jin, H., Zhang, C., Shi, S., Lou, W., & Hou, Y. T. (2024) · IEEE Conference on Communications and Network Security · doi:10.1109/cns62487.2024.10735575 · Cited by 25
Evaluating the effectiveness of prompt engineering for knowledge graph question answering
Kosten, C., Nooralahzadeh, F., & Stockinger, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1454258 · Cited by 11
TypeFly: Low-Latency Drone Planning With Large Language Models
Chen, G., Yu, X., Ling, N., & Zhong, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3561282 · Cited by 31
“For example, according to [41], GPT4 shows an unprecedented 94% Pass@1 rate on the HumanEval [3] dataset, as a comparison, the best local deployable model Code Llama [28] only achieves a 62% Pass@1 rate.”
Survey on Modeling of Human‐made Articulated Objects
Liu, J., Savva, M., & Amiri, A. M. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70092 · Cited by 23
Self-citationHow Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
Velasco, A., Rodríguez-Cárdenas, D., Alif, L. R., Palacio, D. N., & Poshyvanyk, D. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00025 · Cited by 12
“RQ 1 [Propensity of Code Smells] : Both CodeLlama-7b-Instruct-hf [30] and Mistral-7B-v0.”
InfluentialHierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs
Dhulshette, N., Shah, S., & Kulkarni, V. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00023 · Cited by 12
“More recently, the emergence of code-specific LLMs, such as CodeLlama [13], StarCoder [14], and DeepSeek-Coder [15], has signifi-cantly improved summarization accuracy.”
Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy
Alizadeh, N., Belchev, B., Saurabh, N., Kelbert, P., & Castor, F. (2025) · Empirical Software Engineering · doi:10.1109/msr66628.2025.00109 · Cited by 24
Combining Large Language Models with Static Analyzers for Code Review Generation
Jaoua, I., Sghaier, O. B., & Sahraoui, H. A. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00038 · Cited by 13
“Code-specific LLMs, such as CodeGen [30], StarCoder [31] and CodeLlama [32] are optimized to excel in code comprehension, code generation, and other programming-related tasks.”
Less Is More: On the Importance of Data Quality for Unit Test Generation
Zhang, J., Hu, X., Gao, S., Xia, X., Lo, D., & Li, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715778 · Cited by 10
“• CodeLlama7B [51]: CodeLlama7B is a more recent SOTA model based on the Llama2 architecture, specifically fine-tuned for code-related tasks.”
InfluentialHuman-guided collective LLM intelligence for strategic planning via two-stage information retrieval
Kim, S., Ha, J., Lee, H., Park, S., & Cho, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104288 · Cited by 14
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
“…al., 2023; Laskar et al., 2023), as well as on specialized tasks such as mathematical and spatial reasoning, symbols manipulation, and code generation (Chen et al., 2019; Kojima et al., 2022; Cohn and Hernandez-Orallo, 2023; Frieder et al., 2023; Ray, 2023; Rozière et al., 2023; Shen et al., 2023).”
BackgroundWeaknesses in LLM-Generated Code for Embedded Systems Networking
Dunne, M., Schram, K., & Fischmeister, S. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00033 · Cited by 10
“5 [8], Code LLama [2], Mistral [4], and XGen [5].”
BackgroundMethodA Generalize Hardware Debugging Approach for Large Language Models Semi-Synthetic, Datasets
Fu, W., Li, S., Zhao, Y., Yang, K., Zhang, X., Jin, Y., & Guo, X. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3487486 · Cited by 11
“The intelligence of LLMs is regarded as a compression of the training datasets [38], making these datasets the most crucial component of LLMs. Fig.”
Eliminating Backdoors in Neural Code Models for Secure Code Understanding
Sun, W., Chen, Y., Fang, C., Feng, Y., Xiao, Y., Guo, A., Zhang, Q., Chen, Z., Xu, B., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715782 · Cited by 15
“ANR is computed as”
Analyzing the dependability of Large Language Models for code clone generation
Eagal, A., Stolee, K. T., & Ore, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112548 · Cited by 13
Context-aware code summary generation
Su, C., Bansal, A., Huang, Y., Li, T. J.-J., & McMillan, C. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112580 · Cited by 17
Typhoon: Thai Large Language Models
Pipatanakul, K., Jirabovonvisut, P., Manakul, P., Sripaisarnmongkol, S., Patomwong, R., Chokchainant, P., & Tharnpipitchai, K. (2023) · arXiv · doi:10.48550/arxiv.2312.13951 · Cited by 9
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26
“In the Sheet Environment, progress is evaluated using r match t , which uses a matching score between the cells of the current and golden table (Appendix L.10).”
InfluentialTabSAL: Synthesizing Tabular data with Small agent Assisted Language models
Li, J., Qian, R., Tan, Y., Li, Z., Chen, L., Liu, S., Wu, J., & Chai, H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112438 · Cited by 10
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 15
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 11
Exploring Explainable AI in Large Language Models: Enhancing Transparency and Trust
H., I., Jo, A. A., & Raj, E. D. (2024) · International Conference on Advances in Computing and Communications (ICACC) · doi:10.1109/icacc63692.2024.10845370 · Cited by 17
Interleaving static analysis and LLM prompting with applications to error specification inference
Chapman, P. J., Rubio-González, C., & Thakur, A. (2025) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-025-00780-7 · Cited by 8
“How nondeterminism from the LLM affects the results of the analysis.”
AutoPAC: Exploring LLMs for Automating Policy to Code Conversion in Business Organizations
Chowdhary, N., Dutta, T., Chattopadhyay, S., & Chakraborty, S. (2025) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets63942.2025.10885751 · Cited by 10
“Although there exist a few LLM-based natural language to program generators (like NL2Code [27], CodeX [28], Code Llama [29], StarCoder [30], Tabnine 8 , etc.), training domain-specific languages like REGO requires significant resources.”
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 11
“In Design & Governance , [41] introduces a framework for qualitatively analyzing chatbot dialogues to reine customer service, [137] proposes an interface for eicient literature search in medical archives, highlighting chatbot versatility and the importance of Deining roles for responsibility and…”
Code Comment Inconsistency Detection and Rectification Using a Large Language Model
Rong, G., Yu, Y., Liu, S., Tan, X., Zhang, T., Shen, H., & Hu, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00035 · Cited by 19
Self-citationZhang, Y., Lu, Q., Liu, K., Dou, W., Zhu, J., Li, Q., Zhang, C., Lin, Z., & Wei, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3763791 · Cited by 11
“First, each project has received more than 50 stars on GitHub and is actively maintained, ensuring ongoing community interest and updates.”
Context-Enhanced Vulnerability Detection Based on Large Language Models
Yang, Y., Xu, B., Gao, X., & Sun, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3779222 · Cited by 19
Seo, J., Zhang, N., & Rong, C. (2023) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom59040.2023.00062 · Cited by 8
“Notably, the technological forefront has recently witnessed the emergence of specialized models like Code Llama [25], meticulously tailored for coding applications.”
MethodChain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization
Vijayaraghavan, P., Nitsure, A., Mackin, C., Shi, L., Ambrogio, S., Haran, A., Paruthi, V., El-Zein, A., Coops, D., Beymer, D., et al. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740226 · Cited by 8
TaskBench: Benchmarking Large Language Models for Task Automation
Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.18760 · Cited by 24
“Based on our curated datasets, we further introduce T ASK E VAL to systematically evaluate the capability of LLMs in automating tasks, including task decomposition, tool invocation, and parameter prediction.”
MethodSelf-citationA Transformer-Based Approach for Smart Invocation of Automatic Code Completion
de Moor, A., van Deursen, A., & Izadi, M. (2024) · Proceedings of the 1st ACM International Conference on AI-Powered Software · doi:10.1145/3664646.3664760 · Cited by 7
“Recent advancements have expanded the considered context to include not only the subsequent code [3, 15, 19, 26, 35], but also related snippets from other files to enrich the prediction accuracy [27, 45].”
BackgroundPractical Guide to the Use of AI-Enabled Analytics in Research
Kothari, A. N., Kaji, A. H., & Melton, G. B. (2025) · JAMA Surgery · doi:10.1001/jamasurg.2024.6028 · Cited by 7
SQL Autograder: Web-based LLM-powered Autograder for Assessment of SQL Queries
Manikani, K., Chapaneri, R., Shetty, D., & Shah, D. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00460-2 · Cited by 12
“On the other hand, Llama (Touvron et al., 2023), CodeLlama (Roziere et al., 2023), and Vicuna (Chiang et al., 2023) cater to more specific domains.”
InfluentialASKSQL: Enabling cost-effective natural language to SQL conversion for enhanced analytics and search
Bajgoti, A., Gupta, R., & Dwivedi, R. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100641 · Cited by 10
LLaFS++: Few-Shot Image Segmentation With Large Language Models
Zhu, L., Chen, T., Ji, D., Xu, P., Ye, J., & Liu, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3573609 · Cited by 9
“For the LLM, we employ CodeLlama [63] with 7 billion parameters that have been finetuned through instruction tuning.”
Self-citationEnhancing Code Generation for Low-Resource Languages: No Silver Bullet
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00058 · Cited by 10
“Both these families of models achieved state-of-the-art performance across code-related benchmarks [6], [7].”
InfluentialFan, Z., Ghaddar, B., Wang, X., Xing, L., Zhang, Y. A., & Zhou, Z. (2025) · European Journal of Operational Research · doi:10.1016/j.ejor.2025.08.029 · Cited by 40
AskIt: Unified Programming Interface for Programming with Large Language Models
Okuda, K., & Amarasinghe, S. P. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444830 · Cited by 10
“These abilities span a wide range of tasks, including arithmetic operations [2], question answering [3], text summarization [4], language translation [5], paraphrasing [6], text prediction [7], and code generation [8]– [12].”
BackgroundSumLLaMA: Efficient Contrastive Representations and Fine-Tuned Adapters for Bug Report Summarization
Xiang, B., & Shao, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3397326 · Cited by 12
“5 [37].”
MethodQin, Z., Bai, Y., Sohrabizadeh, A., Ding, Z., Hu, Z., Sun, Y., & Cong, J. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740204 · Cited by 8
Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 12
“Inour experiments, we use Open AI GPT models by Open AI and Llama 3 by Meta [Rozière et al., 2023].”
An exploration of prompting LLMs to generate energy-efficient code
Cappendijk, T., de Reus, P., & Oprescu, A. (2025) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1109/greens66463.2025.00010 · Cited by 7
“We use Code Llama-70b, Code Llama-70b-Instruct, Code Llama-70b-Python [15], DeepSeek-Coder-33b-base, and DeepSeek-Coder-33b-instruct [16] and find a decrease in energy consumption for a specific combination of prompt optimisation, LLM, and Python code problems.”
Jiang, X., Dong, Y., Tao, Y., Liu, H., Jin, Z. G., & Li, G. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00133 · Cited by 18
Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement
Khan, Z., Vijay Kumar, B. G., Schulter, S., Fu, Y., & Chandraker, M. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01360 · Cited by 7
“We use the CodeLlama [22] family of models as our initial policy.”
MethodYao, K., Wang, H., Qin, C., Zhu, H., Wu, Y., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688839 · Cited by 7
Harnessing the Power of LLMs: LLM Summarization for Human-Centric DAST Reports
Thool, A., & Brown, C. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00014 · Cited by 7
“Code Llama is a code generation and discussion model built on Llama 2 [47].”
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 9
“The code models like CodeLLaMA [38] and CodeGeex [67] have also built upon their base models to significantly enhance performance in code generation and comprehension.”
MethodHex2Sign: Automatic IDS Signature Generation from Hexadecimal Data using LLMs
Balasubramanian, P., Ali, T., Salmani, M., KhoshKholgh, D., & Kostakos, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825710 · Cited by 8
LAD: Efficient Accelerator for Generative Inference of LLM with Locality Aware Decoding
Wang, H., Li, Y., Xu, H., Wang, Y., Liu, L., Yang, J., & Han, Y. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00111 · Cited by 6
“Large Language Models (LLMs) [2], [4], [46], [57] based on Transformer [47] decoder architecture have been widely applied to various critical content generation tasks, such as chatbots [5], image generation [36] and code assistants [7], [9], [37].”
Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study
Caumartin, G., Qin, Q., Chatragadda, S., Panjrolia, J., Li, H., & Costa, D. E. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00070 · Cited by 9
“We replicate the methodology of Guo et al. and include two small open-source models: Llama 2 (7b) [22] and CodeLlama (7b) [23].”
Unlocking Low Frequency Syscalls in Kernel Fuzzing with Dependency-Based RAG
Zhang, Z., Li, L., Liang, R., & Chen, K. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728913 · Cited by 8
“Recent studies have been exploring the potential of LLMs in code generation [39, 50, 62] and fuzzing.”
A First Look at Conventional Commits Classification
Zeng, Q., Zhang, Y., Qiu, Z., & Liu, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00011 · Cited by 11
Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications
Sabouri, S., Eibl, P., Zhou, X., Ziyadi, M., Medvidović, N., Lindemann, L., & Chattopadhyay, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00199 · Cited by 17
“Others tried attempts to adopt the LLM architecture to increase trustwor-thiness [9], [10], which were also found to be ineffective [11].”
Exploring the Potential of LLMs for Code Deobfuscation
Beste, D., Menguy, G., Hajipour, H., Fritz, M., Ciná, A. E., Bardin, S., Holz, T., Eisenhofer, T., & Schönherr, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97620-9_15 · Cited by 7
ShellBox: Adversarially Enhanced LLM-Interactive Honeypot Framework
Yang, G., Sun, Z., & Wang, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3598779 · Cited by 9
“LLaMA-3 [17] enhances its ability to process complex contextual information through multi-stage instruction tuning and supports coherent analysis across extended text sequences.”
Retrieval-Augmented Chain-of-Thought in Semi-structured Domains
Mavi, V., Saparov, A., & Zhao, C. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.18 · Cited by 6
Evaluating Ambiguous Questions in Semantic Parsing
Papicchio, S., Papotti, P., & Cagliero, L. (2024) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew61823.2024.00050 · Cited by 6
“C ODE [21]).”
Test Code Generation for Telecom Software Systems Using Two-Stage Generative Model
Nabeel, M., Nimara, D. D., & Zanouda, T. (2024) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops59551.2024.10615269 · Cited by 12
“2) CodeLLama [25] (Specialized LLM) trained on code corpus, with a focus on code generation tasks.”
BackgroundSen, Ö., Pohl, C., Hacker, I., Stroot, M., & Ulbig, A. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679448 · Cited by 6
“Code Llama [13] (13 billion parameters), DeepSeek Coder [14] (6.7 billion parameters), and Starcoder 2 [15] (7 billion parameters).”
InfluentialLLM-Aided Compilation for Tensor Accelerators
Hong, C., Bhatia, S., Haan, A., Dong, S. K., Nikiforov, D., Cheung, A., & Shao, Y. S. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691720 · Cited by 7
Large Language Model-Based Optimization for System-Level Test Program Generation
Schwachhofer, D., Domanski, P., Becker, S., Wagner, S., Sauer, M., Pflüger, D., & Polian, I. (2024) · Proceedings · doi:10.1109/dft63277.2024.10753556 · Cited by 6
“Our experiments use Code Llama with 34 billion parameters based on Llama 2 [3].”
IoTCoder: A Copilot for IoT Application Development
Shen, L., & Zheng, Y. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3697447 · Cited by 6
“Code LLMs ( e . g ., CodeLlama [2]) are promising ACM ISBN 979-8-4007-0489-5/24/11. . . $15.00 https://doi.org/10.1145/3636534.3697447 tools designed to generate code according to user requirements described in natural language.”
InfluentialShen, T., Cambria, E., Wang, J., Cai, Y., & Zhang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102860 · Cited by 6
Closed Domain Question-Answering Techniques in an Institutional Chatbot
Saad, M., & Qawaqneh, Z. (2024) · International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) · doi:10.1109/iceccme62383.2024.10796881 · Cited by 7
“Similarly, models like Code LLaMA [27] excel in coding tasks by training on extensive coding datasets.”
Martínez, J., Llinás, B., Botello, J. G., Padilla, J. J., & Frydenlund, E. F. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838967 · Cited by 9
“…languages like Python or JavaScript (Chen et al. 2021; Tian et al. 2023; Denny et al. 2023) where the methods rely on large amounts of open-source, public code (Roziere et al. 2023) overlooking domain-specific programming languages lacking significant open-source code or models (Tarassow 2023).”
(2025) · doi:10.1109/llm4code66737.2025.00014 · Cited by 6
On the performance of large language models on introductory programming assignments
Raihan, N., Goswami, D., Puspo, S. S. C., Siddiq, M. L., Newman, C. D., Ranasinghe, T., Santos, J. C. S., & Zampieri, M. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00968-y · Cited by 8
“For instance, Roziere et al. ( 2023 ) show that CodeLLaMA achieves state-of-the-art results on HumanEval, demonstrating the effectiveness of decoder-only architectures for code generation.”
What undergraduate students need to know and actually know about generative AI
Rismanchian, S., Babar, E. T. R., & Doroudi, S. (2026) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2026.100554 · Cited by 5
StarryStudioAI: Automating UI Design with Code-Based Generative AI and RAG
Patel, B., Gagrani, P., Sheth, S., Thakur, K., Varahan, P., & Shim, S. S. Y. (2025) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc62904.2025.10903712 · Cited by 5
“This enables the generation of structured HTML/CSS code [12].”
Empirical Analysis of Prompt-Based LLMs for Structural Classification of Indian Legal Judgements
Chandana, S. S., Bysani, H., Garg, S., Prasoona, G., Gupta, D., & Sinha, S. (2025) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect63952.2025.10958933 · Cited by 5
“WizardLM-13b-8bit-GGUF: This is a powerful language model developed by WizardLM [27].”
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6
“Similarly, Meta introduced models such as InCoder [3] and CodeLlama [9], while Salesforce developed CodeRL [55] and CodeGen [8].”
InfluentialFuzzy-Assisted Contrastive Decoding Improving Code Generation of Large Language Models
Wang, S., Ding, L., Zhan, Y., Luo, Y., Liu, S., & Ding, W. (2025) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3575060 · Cited by 5
“Model: To better validate the effectiveness and flexibility of the FCD mechanism, commonly used open-source models, e.g., CodeLlama-7b [19], StarCoder [56], WizardCoder-15b [57], and the recently released Llama3-8b, 4 are selected.”
InfluentialAn Empirical Study on Commit Message Generation Using LLMs via In-Context Learning
Wu, Y., Wang, Y., Li, Y., Tao, W., Yu, S., Yang, H.-W., Jiang, W., & Li, J. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00091 · Cited by 22
“…in specific tasks within the software engineering domain, there are a considerable amount of fine-tuned code LLMs (e.g., Codex [36] and Code Llama [37]), and have made tremendous achievements in various tasks, such as commit message generation [21], code review [38], and just-in-time comment…”
ChatSOS: Vector database augmented generative question answering assistant in safety engineering
Tang, H., Chen, D., Chu, Q., & Liu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128644 · Cited by 7
Dumitran, A. M., Badea, A. C., & Muscalu, S.-G. (2024) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista62901.2024.10683837 · Cited by 5
“…freely by Google during beta testing • Codestral (22B) • DeepSeek Coder (6.7B, 33B) [14] • AutoCoder (6.7B, 33B) [15] • CodeLlama (7B, 13B) [16] • Phind-CodeLlama (34B) [18] • StarCoder 2 (15B) [13] • RoLlama2 (7B) [8] • RoMistral (7B) [8] The closed-source models were chosen for their…”
LLM-Aided Compilation for Tensor Accelerators
Hong, C., Bhatia, S., Haan, A., Dong, S. K., Nikiforov, D., Cheung, A., & Shao, Y. S. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691748 · Cited by 7
Tao, Y., Yang, R., Zeng, K., Yin, C., Lu, Y., Lu, W., Shi, Y., Wang, B., & Huang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3549819 · Cited by 7
“For instance, in the domain of common knowledge, GPT series [2], [3], [4] and LLaMa series [5], [6], [7] can efficiently address general text and code-related problems [24], [25], stable-Diffusion series [8] can achieve precise conversion from natural language to images, SORA [9] can accomplish the…”
DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
Yuan, M., Chen, J., Xing, Z., Quigley, A. J., Luo, Y., Luo, T., Mohammadi, G., Lu, Q., & Zhu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00109 · Cited by 8
Generating P4 Dataplanes Using LLMs
Dumitru, M.-V., Bădoiu, V.-A., Gherghescu, A. M., & Raiciu, C. (2024) · IEEE International Conference on High Performance Switching and Routing (HPSR) · doi:10.1109/hpsr62440.2024.10635926 · Cited by 5
“Focusing on machine learning-based code generation, many open LLMs address the task of code generation [3], [21]– [26].”
BackgroundAn Open-source Cross-Industry and Cloud-agnostic Generative AI Platform
Ali, A. R., Kumar, K., Siddiqui, M. A., & Zahid, M. A. H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650688 · Cited by 6
“Code LLaMA has undergone training with a 8,192 token context window, and the three model variants have undergone additional long-context fine-tuning, enabling them to manage a context window of up to 100,000 tokens [31].”
BackgroundOracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5
“Although there are many other open-source large language models [24, 33], our experiment only evaluated LLMCode-Choice using GPT-4, because it is the most representative LLM we can access.”
Symbolic Execution with Test Cases Generated by Large Language Models
Xu, J., Xu, J., Chen, T., & Ma, X. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00031 · Cited by 5
“We select Code Llama [17] which claims excellent performance in code-related tasks, and Gemini [25], a relatively new large model introduced by Google, as potential alternatives to GPT-4.”
MethodLLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites
Sollenberger, Z., Patel, J., Munley, C., Jarmusch, A., & Chandrasekaran, S. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00238 · Cited by 5
“LLMs are being widely considered for tasks such as code generation, code summarization, and code refactoring [11], [12], [13], [14].”
Using Large Language Models to Generate Concise and Understandable Test Case Summaries
Djajadi, N., Deljouyi, A., & Zaidman, A. (2025) · IEEE International Conference on Program Comprehension · doi:10.1109/icpc66645.2025.00040 · Cited by 5
“CodeLLama [24], and a commercial closed-source LLM, ChatGPT [25].”
Prucker, P., Busch, F., Dorfner, F. J., Mertens, C. J., Bayerl, N., Makowski, M. R., Bressem, K. K., & Adams, L. C. (2025) · Clinical Imaging · doi:10.1016/j.clinimag.2025.110557 · Cited by 5
Schwachhofer, D., Domanski, P., Becker, S., Wagner, S., Sauer, M., Pflüger, D., & Polian, I. (2024) · IEEE European Test Symposium (ETS) · doi:10.1109/ets61313.2024.10567741 · Cited by 5
“LLMs are transformer-based neural networks frequently used for code generation, e.g., Code Llama [3].”
Method
Metadata
- Authors
- Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., & Synnaeve, G.
- Affiliations
- École nationale des ponts et chaussées (FR)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.12950
- OpenAlex
- W4386185625
- Cited by
- 531
- References
- 92
- Citations by year
- 2026: 65 · 2025: 232 · 2024: 216 · 2023: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- CodeLlama, code large language models, Llama 2, HumanEval, MBPP, code generation benchmarks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
