Code Llama: Open Foundation Models for Code
Open accessAbstract
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current LLMs consume and learn from task data 'as-is' via finetuning or in-context learning.
“Given a new task, current LLMs consume and learn from the task data 'as-is' via finetuning or in-context learning [9, 10, 11, 12].”
From Self-Adapting Language Models (2025) · cites this paper as [10]
The Introduction describes fine-tuning Code Llama on code and instruction data, and Appendix F reports two-shot prompting, supporting the two ways of using task data. But the paper does not characterize task data as consumed “as-is”: its Self-instruct section describes generating, deduplicating, and filtering examples, while Appendix F describes task-specific prompt instructions. (LLM, conf=0.84)
Experimental — accuracy not yet measuredCode-based LLMs are particularly well suited to address the demand for intelligent tools that can understand, analyze, and generate secure code.
“Code-based LLMs (e.g., CodeLlama(Roziere etal. 2023) and StarCoder(Li etal. 2023d; Lozhkov et al. 2024)) are particularly well suited to address this demand.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
The Introduction identifies Code Llama as an LLM for code generation and infilling, and Related work discusses code-understanding tasks. This supports its use as an example of a code-based LLM. The Red teaming section evaluates responses to malicious-code requests, but does not establish that Code Llama can generate secure code or is particularly well suited to that security-focused demand. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAt the time of writing, only the CodeLlama models are able to handle the very long contexts necessary for SWE-bench.
“At the time of writing, only the CodeLlama models (Rozi` ere et al., 2023) are able to handle the very long contexts necessary.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
The Introduction says Code Llama handles contexts up to 100,000 tokens, supporting its long-context capability. But the section on long sequences in LLMs also identifies Claude as having a 100K-token context window, so the paper does not support the claim that only Code Llama can handle such contexts. It does not establish what context length SWE-bench requires. (LLM, conf=0.88)
Experimental — accuracy not yet measuredCode Llama family are based upon Llama 2 and pre-trained on a 500B token corpus with both code and natural language texts using the infilling objective.
“Code Llama family are based upon Llama 2. It is pre-trained on a 500B token corpus with both code and natural language texts using the infilling objective.”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
The Introduction identifies Code Llama as derived from Llama 2, and §3.4.1 says the models were tuned on 500B additional tokens, mostly code (85%). But the claim conflates that additional training with pre-training from scratch: Llama 2 had already been trained on 2T tokens. The Introduction and Discussion also limit the infilling objective to the 7B, 13B, and 70B models, rather than the whole family. (LLM, conf=0.92)
Experimental — accuracy not yet measuredCodeLlama is a 7B decoder-only model trained on code synthesis.
“which is a 7B decoder-only model trained on code synthesis.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
The Abstract describes Code Llama as a family with 7B, 13B, 34B, and 70B variants, rather than a single 7B model. The Introduction describes training for code generation, and Additional information calls the models autoregressive transformers, consistent with the decoder-only description. The 7B description is accurate for a variant, but not for Code Llama as a whole. (LLM, conf=0.88)
Experimental — accuracy not yet measured
LLMs' proficiency at generating code in mainstream programming languages such as Python and JavaScript does not transfer to logic programming.
“LLMs have achieved remarkable proficiency at generating code in mainstream programming languages such as Python and JavaScript, yet this proficiency does not transfer to logic programming.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Rozière et al. (2023)
Section 3.1.1 reports Python code-generation results, and Section 3.1.2 reports results across several other programming languages. The paper does not establish the claimed failure to transfer that proficiency to logic programming; the cited passages also do not specifically report JavaScript results. (LLM, conf=0.83)
Experimental — accuracy not yet measuredCodeLLaMA-34B is an aligned version of LLaMA-234B, fine-tuned with 500B tokens of code data.
“CodeLLaMA-34B [42] is an aligned version of LLaMA-234B, which is fine-tuned with 500B tokens of code data.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [42]
Section 3.4.1 says Code Llama is based on Llama 2 and was tuned on 500B additional tokens, 85% of which were code—not 500B code tokens. The Introduction and Section 6 distinguish the base Code Llama 34B model from Code Llama–Instruct, the instruction-tuned variant; the claim incorrectly calls the base model aligned. The source passages provide no page numbers. (LLM, conf=0.88)
Experimental — accuracy not yet measuredDifferent LLMs possess unique strengths and specialize in various tasks aspects.
“At the same time, different LLMs possess unique strengths and specialize in various tasks aspects.”
From Mixture-of-Agents Enhances Large Language Model Capabilities (2024) · cites this paper as Roziere et al., 2023
Many previous works have confirmed that LLMs have good code comprehension capabilities.
“There have been many previous works that have confirmed that LLMs do have good code comprehension capabilities (He et al. 2024; Luo et al. 2024; Roziere et al. 2023; Li et al. 2023d).”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Roziere et al. 2023
While numerous language models have been implemented with a primary focus on code generation, most tend to underperform the best generalist language models.
“Moreover, while numerous LMs have been implemented with a primary focus on code generation, most tend to underperform the best generalist LMs (Li et al., 2022; Le et al., 2022; Roziere et al., 2023; Austin et al., 2021; Xu et al., 2022).”
From Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation (2023) · cites this paper as Roziere et al., 2023
SWE-Llama 7b and 13b are fine-tuned models based on CodeLlama.
“Utilizing SWE-bench-train, we release two fine-tuned models, SWE-Llama 7 b and 13 b, based on the CodeLlama (Rozi` ere et al., 2023) model.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Rozière et al., 2023
Prior studies showed that within the same LLM family, larger models are associated with better performance.
“Prior studies showed that within the same LLM family, larger models are associated with better performance [59, 67, 73].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [67]
The main experiments are performed on CodeLlama.
“The main experiments are performed on CODELLAMA (Roziere et al., 2023).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
CodeLlama is more specialized in coding tasks.
“Furthermore CODELLAMA is more specialized in coding tasks thus yielding slightly lower PERPLEX-ITY (TARGET) score compared to MISTRAL despite their comparable sizes.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
Among the LLMs tested, CODELLAMA benefits the most from good test case selection.
“Among the LLMs tested, CODELLAMA (Roziere et al., 2023) benefits the most from good test case selection, hence we focus this work around CODELLAMA.”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Roziere et al., 2023
Both Sun et al. and Mozannar et al. aim to maximise the acceptance rate of the shown completions.
“Moreover, both of these tools aim to maximise the acceptance rate of the shown completions”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
The Single line completion section evaluates Code Llama’s completion accuracy using exact match and BLEU, not whether users accept shown completions. Searches for “acceptance rate,” “Sun et al.,” and “Mozannar” found no passage in our copy addressing the stated aim of the two tools. (LLM, conf=0.76)
Experimental — accuracy not yet measuredOptimising only acceptance rate results in worse completions.
“Specifically, optimising only acceptance rate results in worse completions [32, 49].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
The Single line completion section reports that long-context fine-tuning slightly worsens performance on short code completions, but does not connect that result to acceptance-rate optimization. Searches for “acceptance rate,” “acceptance,” and user acceptance of suggested completions found no passage addressing the claimed trade-off. (LLM, conf=0.74)
Experimental — accuracy not yet measuredThe proxy metrics (harmonic mean of acceptance rate and CodeBERTScore) might not fully capture the usability of the interaction without potentially compromising it in some other way.
“Consistent with earlier research [32, 49], we acknowledge that these proxy metrics might not fully capture the usability of the interaction without potentially compromising it in some other way.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
The Pass@k evaluation section says automatic safety benchmarks do not comprehensively capture model risks, and the General helpfulness vs. coding ability section cautions against relying solely on a reward-model score. Neither addresses whether acceptance rate and CodeBERTScore capture interaction usability or compromise it in another way. Searches for those metrics and for interaction usability found no relevant passage; page numbers were unavailable in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredCopilot contains most of the significant features found in previous work.
“and contains most of the significant features found in previous work [32].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [32]
The “Closed-source vs open-source models” section identifies Codex/copilot as a closed-source code model, but does not describe its features. The “Code understanding and synthesis tasks” section lists tasks addressed by code models generally; it does not say Copilot contains most significant features from previous work. Searches for Copilot’s features and capabilities found no passage establishing that comparison. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2× · see citing works
“Effective handling of long sequences is a major topic of research in transformer-based language modeling (Vaswani et al., 2017).”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 3× · see citing works
“( b) Difference in BLEU scores for single line code completion of long context models compared to their respective base models before fine-tuning.”
MethodInfluentialRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1× · see citing works
“To measure the toxicity of the generated outputs from each of the LLMs, we employ the default ToxiGen classifier, tuned on RoBERTa (Liu et al., 2019).”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 2× · see citing works
“The VADER produces sentiment scores between -1 and 1, where a positive (negative) score indicates a positive (negative) sentiment towards the population mentioned in the prompt.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 11× · see citing works
“ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
BackgroundMethodInfluentialSelf-citationKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 1× · see citing works
“� 500B 20B 20B”
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1× · see citing works
“Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 3× · see citing works
“Collecting supervised data from human annotators or training from human feedback (Ouyang et al., 2022) is expensive for coding tasks as it requires input from professional developers.”
BackgroundMethodModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 1× · see citing works
“Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundRoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952 · cited 9× · see citing works
“For long context fine-tuning (LCFT), we use a learning rate of 2 e − 5 , a sequence length of 16,384, and reset RoPE frequencies with a base value of θ = 10 6 .”
MethodInfluentialLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 2× · see citing works
“Liu et al. (2023b) showed that the inability to recall content placed in the middle of long prompts is a common failure mode in LLMs; our retrieval task is analogous to their setup, albeit tailored to code models which are not fine-tuned to follow instructions.”
BackgroundWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 3× · see citing works
“In parallel, models specifically trained or fine-tuned for code understanding and program synthesis from natural language prompts emerged with LLMs such as Codex (Chen et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b) and CodeGen 2 (Ni…”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 1× · see citing works
“(2022) and use nucleus sampling (Holtzman et al., 2020) ( p = 0 .”
BackgroundFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1× · see citing works
“For end-users, the utility of LLMs is significantly improved by instruction fine-tuning (Ouyang et al., 2022; Wei et al., 2022; OpenAI, 2023; Touvron et al., 2023b), which also helps preventing unsafe, toxic or biased generations.”
BackgroundCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 9× · see citing works
“…& Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a), as well as solving math problems as demonstrated by PaLM (Chowdhery et al., 2022) or Codex (Chen et al., 2021).”
BackgroundMethodInfluentialUnified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,000 · cited 1× · see citing works
“…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 4× · see citing works
“Moreover, the Chinchilla scaling laws (Hoffmann et al., 2022) indicate that larger models would benefit more from training on more tokens.”
BackgroundMethodInfluentialTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899 · cited 6× · see citing works
“The TruthfulQA benchmark comprises 817 questions spread across 38 categories, encompassing topics such as health, finance, law, and politics (Lin et al., 2022).”
BackgroundMethodInfluentialCodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Gotmare, A. D., Hoi, S., Le, H., Savarese, S., & Wang, Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2207.01780 · Cited by 605 · cited 2× · see citing works
“…filter, cluster, or improve the correctness of programs when few candidates must be submitted (Li et al., 2022; Chen et al., 2023a; Le et al., 2022; Zhang et al., 2023), or unit tests them within a reinforcement learning objective to enrich the training signal (Le et al., 2022; Liu et al., 2023a).”
BackgroundTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 2× · see citing works
“To measure math-reasoning capabilities of the proposed method, we report results on the GSM8K benchmark Cobbe et al. (2021), which is comprised of a set of middle-school math word problems.”
MethodResultCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 2× · see citing works
“The Python subsection of the CodeXGLUE code summarization benchmark Lu et al. (2021) can be used as an infilling benchmark (Fried et al., 2023; Li et al., 2023) in which a docstring surrounded by triple quotes has to be inserted between the function header and body in a Python function definition.”
BackgroundMethodUnnatural Instructions: Tuning Language Models with (Almost) No Human LaborPreprint
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09689 · Cited by 516 · cited 1× · see citing works
“FIM Size HumanEval MBPP Test loss pass@1 pass@10 pass@100 pass@1 pass@10 pass@100”
Self-citationGraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480 · cited 3× · see citing works
“The Python subsection of the CodeXGLUE code summarization benchmark Lu et al. (2021) can be used as an infilling benchmark (Fried et al., 2023; Li et al., 2023) in which a docstring surrounded by triple quotes has to be inserted between the function header and body in a Python function definition.”
BackgroundMethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 3× · see citing works
“We start by reporting results for Python code generation using the HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and APPS (Hendrycks et al., 2021) benchmarks.”
MethodCodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 3× · see citing works
“Code Llama 7B outperforms larger models such as CodeGen-Multi or StarCoder, and is on par with Codex (code-cushman-001, Chen et al., 2021).”
MethodGPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · cited 5× · see citing works
“Early observations with LLMs such as GPT-Neo (Black et al., 2021) or GPT-J (Wang & Komatsuzaki, 2021) showed that adding code in the training data makes program synthesis possible even with medium size LLMs. Code from open-source software is now a standard part of the training data for general-purpose LLMs such as PaL…”
MethodInfluentialThe adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · see citing works
Planning with Large Language Models for Code GenerationPreprint
Zhang, S., Chen, Z., Shen, Y., Ding, M., Tenenbaum, J., & Gan, C. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2303.05510 · Cited by 277 · cited 2× · see citing works
“A recent stream of work takes into account program execution or unit tests to filter, cluster, or improve the correctness of programs when few candidates must be submitted (Li et al., 2022; Chen et al., 2023a; Le et al., 2022; Zhang et al., 2023), or unit tests them within a reinforcement learning objective to enrich…”
BackgroundToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2203.09509 · Cited by 276 · cited 6×
“Table 20 presents the percentages of toxic generations for different demographic groups among ToxiGen prompts.”
MethodInfluentialStarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
CodeGen2: Lessons for Training LLMs on Programming and Natural LanguagesPreprint
Nijkamp, E., Hayashi, H., Xiong, C.-M., Savarese, S., & Zhou, Y.-B. (2023) · arXiv.org · doi:10.48550/arxiv.2305.02309 · Cited by 253 · cited 1× · see citing works
“…et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b) and CodeGen 2 (Nijkamp et al., 2023a), GPT-NeoX (Black et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023) and phi-1 (Gunasekar et al.,…”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244 · cited 1× · see citing works
“We employ the Bias in Open-Ended Language Generation Dataset ( BOLD ) (Dhamala et al., 2021) to investigate how the sentiment in the model’s outputs may differ based on demographic attributes.”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · cited 1× · see citing works
“…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Transformer Language Models without Positional Encodings Still Learn Positional Information
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.16634 · Cited by 210 · cited 2×
“Recent work suggests that causal models do not require an explicit encoding of position information (Haviv et al., 2022; Kazemnejad et al., 2023), a hypothesis we did not test in this work as we demonstrated that starting from pretrained Llama 2 models is significantly more efficient than training…”
BackgroundMEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., & Lewis, M. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.07185 · Cited by 209 · cited 1×
“Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundA Length-Extrapolatable TransformerPreprint
Sun, Y., Dong, L., Patra, B., Ma, S.-M., Huang, S.-H., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10554 · Cited by 192 · cited 1× · see citing works
“…is a two-digit number that is randomly sampled based on the overall number of tokens in the prompt: (a) Attention expectations over relative distances between key and value embeddings for different frequency regimes, using the bound derived in (Sun et al., 2023) for embedding dimensionality 1024.”
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 8× · see citing works
“Our code-training for 7B, 13B and 70B Code Llama models features a multitask objective (Fried et al., 2023) consisting of both autoregressive and causal infilling prediction, enabling applications such as real-time completion in source code editors or docstring generation.”
BackgroundMethodInfluentialPenedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164 · cited 1× · see citing works
“It is well-known that data quality is critical in the training and responsible development of LLMs (e.g., Hoffmann et al., 2022; Penedo et al., 2023), and this is also true for code as discussed by Allal et al.”
BackgroundTrain Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161 · cited 2× · see citing works
“3 Press et al. (2022) propose a linear bias for attacking extrapolation; in contrast, our approach seeks to reduce existing bias towards shot-range attention.”
MethodResultUnit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 158 · cited 1× · see citing works
“…fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a), as well as solving math problems as demonstrated by PaLM (Chowdhery et al., 2022) or Codex…”
MethodMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 10× · see citing works
“With the same training process, our larger models outperform their smaller counterparts on almost every metric from HumanEval, MBPP and APPS (Table 2, 3).”
BackgroundMethodInfluentialGunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11644 · Cited by 138 · cited 3× · see citing works
“…2023b) and CodeGen 2 (Nijkamp et al., 2023a), GPT-NeoX (Black et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023) and phi-1 (Gunasekar et al., 2023), consistently demonstrating better performance on code benchmarks than general-purpose LLMs of comparable or even larger size.”
BackgroundMethodContrastive Code Representation Learning
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., & Stoica, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.482 · Cited by 129 · cited 1× · see citing works
“…of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to learn tree-aware positional encodings (Shiv & Quirk, 2019; Peng et al., 2021).”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 109 · cited 6× · see citing works
“For that, we use the MultiPL-E benchmark (Cassano et al., 2023).”
BackgroundMethodInfluentialCan OpenAI's codex fix bugs?: an evaluation on QuixBugs
Prenner, J. A., Babii, H., & Robbes, R. (2022) · Proceedings of the Third International Workshop on Automated Program Repair · doi:10.1145/3524459.3527351 · Cited by 99 · cited 2× · see citing works
“…refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a),…”
BackgroundDOBF: A Deobfuscation Pre-Training Objective for Programming LanguagesPreprint
Rozière, B., Lachaux, M.-A., Szafraniec, M., & Lample, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07492 · Cited by 96 · cited 1× · see citing works
“…of works proposed to incorporate within the training objective structural knowledge of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to…”
BackgroundSelf-citationCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 7× · see citing works
“However, instead of downscaling frequencies linearly as Chen et al. (2023b), we change the base period from which they are derived.”
BackgroundMethodInfluentialLLM Censorship: A Machine Learning Challenge or a Computer Security Problem?Preprint
Glukhov, D., Shumailov, I., Gal, Y., Papernot, N., & Papyan, V. (2023) · arXiv.org · doi:10.48550/arxiv.2307.10719 · Cited by 89 · cited 1× · see citing works
“Dual intent prompts are requests for help with writing code that could be used maliciously but the prompt does not directly address the topic (example “Mosaic Prompts” Glukhov et al. (2023)).”
BackgroundUnsupervised translation of programming languages
Lachaux, M.-A., Roziere, B., Chanussot, L., & Lample, G. (2020) · arXiv · doi:10.48550/arxiv.2006.03511 · Cited by 83 · cited 1× · see citing works
“These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
MethodSelf-citationRepairing Bugs in Python Assignments Using Large Language ModelsPreprint
Zhang, J.-L., Cambronero, J., Gulwani, S., Le, V., Piskac, R., Soares, G., & Verbruggen, G. (2022) · arXiv.org · doi:10.48550/arxiv.2209.14876 · Cited by 68 · cited 2× · see citing works
“…tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022;…”
BackgroundMethodGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · cited 2× · see citing works
“In parallel, models specifically trained or fine-tuned for code understanding and program synthesis from natural language prompts emerged with LLMs such as Codex (Chen et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b) and CodeGen 2 (Ni…”
MethodSantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · cited 2× · see citing works
“In addition to program synthesis from natural language prompts or infilling (Fried et al., 2023; Bavarian et al., 2022; Li et al., 2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al.”
BackgroundLearning to Fix Build Errors with Graph2Diff Neural Networks
Tarlow, D., Moitra, S., Rice, A. C., Chen, Z., Manzagol, P.-A., Sutton, C. A., & Aftandilian, E. E. (2020) · International Conference on Software Engineering · doi:10.1145/3387940.3392181 · Cited by 55 · cited 1× · see citing works
“…et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a), as well as solving math problems as…”
BackgroundThe Impact of Positional Encoding on Length Generalization in Transformers
Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., & Reddy, S. (2023) · arXiv · doi:10.48550/arxiv.2305.19466 · Cited by 54 · cited 1×
“Recent work suggests that causal models do not require an explicit encoding of position information (Haviv et al., 2022; Kazemnejad et al., 2023), a hypothesis we did not test in this work as we demonstrated that starting from pretrained Llama 2 models is significantly more efficient than training…”
BackgroundExtending Context Window of Large Language Models via Positional Interpolation
Chen, S., Wong, S., Chen, L., & Tian, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.15595 · Cited by 50 · cited 7× · see citing works
“However, instead of downscaling frequencies linearly as Chen et al. (2023b), we change the base period from which they are derived.”
BackgroundMethodInfluentialEfficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47 · cited 7× · see citing works
“We observe a further increase in performance when prompting the models in SPM format, like witnessed in Bavarian et al. (2022).”
BackgroundMethodInfluentialBreak-It-Fix-It: Unsupervised Learning for Program RepairPreprint
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 43 · cited 1× · see citing works
“These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
MethodCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 1× · see citing works
“We train in lling models following the concept of causal masking (Aghajanyan et al., 2022; Fried et al., 2023), where parts of a training sequence are moved to the end, and the reordered sequence is predicted autoregressively.”
MethodJung, J., West, P., Jiang, L., Brahman, F., Lu, X., Fisher, J. R., Sorensen, T., & Choi, Y. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.naacl-long.250 · Cited by 42 · cited 1×
“5, following Jung et al. (2023), in order to obtain good performance on simple benchmarks such as HumanEval and MBPP.”
Leveraging automated unit tests for unsupervised code translation
Roziere, B., Zhang, J. M., Charton, F., Harman, M., Synnaeve, G., & Lample, G. (2021) · arXiv · doi:10.48550/arxiv.2110.06773 · Cited by 39 · cited 1× · see citing works
“These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
MethodSelf-citationLongNet: Scaling Transformers to 1, 000, 000, 000 Tokens
Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.02486 · Cited by 34 · cited 2× · see citing works
“Scaling Transformers and LLMs to long input sequences has attracted much recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023).”
BackgroundMeet in the Middle: A New Pre-training Paradigm
Nguyen, A., Karampatziakis, N., & Chen, W.-Z. (2023) · Neural Information Processing Systems · doi:10.52202/075280-0225 · Cited by 29 · cited 1×
“In addition to program synthesis from natural language prompts or infilling (Fried et al., 2023; Bavarian et al., 2022; Li et al., 2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et…”
RLTF: Reinforcement Learning from Unit Test FeedbackPreprint
Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., & Ye, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04349 · Cited by 28 · cited 2× · see citing works
“Liu et al. (2023b) showed that the inability to recall content placed in the middle of long prompts is a common failure mode in LLMs; our retrieval task is analogous to their setup, albeit tailored to code models which are not fine-tuned to follow instructions.”
BackgroundSelf-citationCode translation with compiler representations
Szafraniec, M., Roziere, B., Leather, H., Charton, F., Labatut, P., & Synnaeve, G. (2022) · arXiv · doi:10.48550/arxiv.2207.03578 · Cited by 26 · cited 1× · see citing works
“These tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al.,…”
MethodSelf-citationA Survey on Pretrained Language Models for Neural Code Intelligence
Xu, Y., & Zhu, Y. (2022) · arXiv.org · doi:10.48550/arxiv.2212.10079 · Cited by 19 · cited 2×
“A prominent use-case is the formal interaction with computer systems, such as program synthesis from natural language specifications, code completion, debugging, and generating documentation (for a survey, see Xu & Zhu, 2022, also see Section 5).”
MethodMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17 · see citing works
LongCoder: A Long-Range Pre-trained Language Model for Code Completion
Guo, D., Xu, C., Duan, N., Yin, J., & McAuley, J. (2023) · arXiv · doi:10.48550/arxiv.2306.14893 · Cited by 16 · cited 3× · see citing works
“We compare the original test set from (Guo et al., 2023) to our resampled “LCC-balanced” test set.”
MethodResultInfluentialOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 6×
“ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
BackgroundMethodInfluentialOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068 · cited 2×
“…tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022;…”
BackgroundMethodBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Self-citation
Flagged first, then by citations
15 of the 92 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 Technical Report. arXiv:abs/2305.10403, 2023.
- Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: From low-quality model to high-quality dataset & model for summarization and paraphrasing. arXiv:abs/2305.16635, 2023.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
- Han Peng, Ge Li, Wenhan Wang, Yunfei Zhao, and Zhi Jin. Integrating tree path in transformer for code representation. In NeurIPS, pp. 9343-9354, 2021.
- Vighnesh Leonardo Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In NeurIPS, pp. 12058-12068, 2019.
- Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. arXiv:abs/2009.05617, 2020.
Cited by
Sorted by citations
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“On the 24 th. August 2023, Meta released open-sourced Code Llama [29], a state-of-the-art for publicly available LLMs on coding tasks.”
BackgroundA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3641204.3641221 · Cited by 342 · see citing works
Jailbreaking Black Box Large Language Models in Twenty Queries
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00010 · Cited by 523 · see citing works
“Although still at its infancy, the field of study surrounding large language models (LLMs) has shown significant promise in advancing numerous fields, including code generation [28], business analytics [38], and medicine [32].”
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
“…select a dataset that can appropriately represent the characteristics of the target domain; second, determine the strategy for continuous pre-training; and finally, perform pre-training and adjust the model architecture or optimization algorithm as needed to adapt to the new training objectives.”
ChatEDA: A Large Language Model Powered Autonomous Agent for EDA
Wu, H., He, Z., Zhang, X., Yao, X., Zheng, S., Zheng, H., & Yu, B. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3383347 · Cited by 168 · see citing works
“It is worth noting that CodeLlama [33] models achieve strong performance in coding ability, which are incrementally pretrained on code resources based on Llama2 models.”
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Zhang, K., Li, J., Li, G., Shi, X., & Jin, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.737 · Cited by 138 · see citing works
Ullah, S., Han, M., Pujar, S., Pearce, H. A., Coskun, A. K., & Stringhini, G. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00210 · Cited by 132 · see citing works
“Large Language Models (LLMs), such as OpenAI’s Codex [1], Google’s PaLM2 [2], Meta’s Codellama [3], and StarCoder [4], etc., have demonstrated great potential in performing programming-language related tasks such as code generation, code documentation , and debugging.”
MethodIntent-Based Management of Next-Generation Networks: an LLM-Centric Approach
Mekrache, A., Ksentini, A., & Verikoukis, C. V. (2024) · IEEE Network · doi:10.1109/mnet.2024.3420120 · Cited by 131 · see citing works
“• Real-World Deployment and Testing: Our proposed framework is implemented and tested in real-world scenarios using a single NVIDIA A100 GPU with 40GB of vRAM, leveraging the Code Llama LLM [7].”
NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing
Heo, G., Lee, S., Cho, J., Choi, H., Lee, S., Ham, H., Kim, G., Mahajan, D., & Park, J. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651380 · Cited by 129 · see citing works
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120 · see citing works
“Following [1], [45], [46] for all models we settle on a temperature of 0.8, as it maximizes the number of examples for which at least one correct code is produced within 100 suggestions for k > 1 in pass @ k .”
MethodInfluentialFine-tuning and prompt engineering for large language models-based code review automation
Pornprasit, C., & Tantithamthavorn, C. K. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107523 · Cited by 128 · see citing works
“For example, the open-source community-developed large language models such as Code-LLaMa [27], StarCoder [28], and Magicoder [21]; and the commercial large language models such as GPT-3.”
MethodMaharjan, J., Garikipati, A., Singh, N. P., Cyrus, L., Sharma, M., Ciobanu, M., Barnes, G., Thapa, R., Mao, Q., & Das, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-64827-6 · Cited by 116 · see citing works
LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
Zhang, Z., Wang, C., Wang, Y., Shi, E., Ma, Y., Zhong, W., Chen, J., Mao, M., & Zheng, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728894 · Cited by 131 · see citing works
“We use the CodeLlama-7b-Python-hf model.”
InfluentialNi, B., & Buehler, M. J. (2024) · Extreme Mechanics Letters · doi:10.1016/j.eml.2024.102131 · Cited by 113 · see citing works
LLMs in e-commerce: A comparative analysis of GPT and LLaMA models in product review evaluation
Roumeliotis, K. I., Tselikas, N. D., & Nasiopoulos, D. Κ. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100056 · Cited by 105 · see citing works
LLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M. M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 113 · see citing works
“Code LLama-Instruct [177] Decoder-Only 7, 13 & 34 IT Code LLama No Code Open Code generation and understanding”
BackgroundWhen Software Security Meets Large Language Models: A Survey
Zhu, X., Zhou, W., Han, Q., Ma, W., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124971 · Cited by 105 · see citing works
Reducing hallucination in structured outputs via Retrieval-Augmented Generation
Ayala, O., & Bechard, P. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track) · doi:10.18653/v1/2024.naacl-industry.19 · Cited by 97 · see citing works
“Thus we also fine-tune other pretrained LLMs of this size: CodeLlama-7B (Roziere et al., 2023) and Mistral-7B-v0.”
BackgroundMethodAn AI Agent for Fully Automated Multi‐Omic Analyses
Zhou, J., Zhang, B., Li, G., Chen, X., Li, H., Xu, X., Chen, S., He, W., Xu, C., Liu, L., et al. (2024) · Advanced Science · doi:10.1002/advs.202407094 · Cited by 100 · see citing works
A Performance Study of LLM-Generated Code on Leetcode
Coignion, T., Quinton, C., & Rouvoy, R. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661221 · Cited by 88 · see citing works
“The ranking of the functional correctness of the LLMs is consistent with the previous evaluations of the LLMs on HumanEval [1, 4, 10, 21, 24, 27, 32].”
The Dawn of Natural Language to SQL: Are We Fully Ready?
Li, B., Luo, Y., Chai, C., Li, G., & Tang, N. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3682003 · Cited by 104 · see citing works
“+”
Using LLMs in Software Requirements Specifications: An Empirical Evaluation
Krishna, M., Gaur, B., Verma, A., & Jalote, P. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00056 · Cited by 92 · see citing works
“In this paper, we evaluate the proficiency of LLMs like GPT-4 [11] (ChatGPT) [1] and CodeLlama [2] in formulating the SRS for a software project.”
MethodFakih, M., Dharmaji, R., Moghaddas, Y., Quirós, G., Ogundare, O., & Faruque, M. A. A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3639743 · Cited by 85 · see citing works
Assessing LLMs in malicious code deobfuscation of real-world malware campaigns
Patsakis, C., Casino, F., & Lykousas, N. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124912 · Cited by 83 · see citing works
Multitask-Based Evaluation of Open-Source LLM on Software Vulnerability
Yin, X., Ni, C., & Wang, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3470333 · Cited by 84 · see citing works
“For the code-related LLMs, we select the top four models released recently (in 2023), namely DeepSeek-Coder [9], CodeLlama [11], StarCoder [10], and WizardCoder [68].”
BackgroundMethodInfluentialCognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“Unexpectedly, training these models on internet-scale text also made them useful for many tasks beyond generating text, such as writing code (Li et al., 2022; Rozière et al., 2023; Li et al., 2023), modeling proteins (Meier et al.”
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 132 · see citing works
“LLMs have revolutionized various aspects of software engineering [31], [32], including but not limited to automated code generation [2], [33], [34], [35], software testing [36], [37], [38], [39], debugging [40], program repair [41], [42], and document generation [43].”
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74 · see citing works
On the use of large language models in model-driven engineering
Rocco, J. D., Ruscio, D. D., Sipio, C. D., Nguyen, P. T., & Rubei, R. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01263-8 · Cited by 73 · see citing works
“Recently, different language models have been released for targeting specific purposes, e.g., Code Llama [72] for coding, Llama and GPT models [79,67] for chatbot, to name a few.”
LLM4Decompile: Decompiling Binary Code with Large Language Models
Tan, H., Luo, Q., Li, J., & Zhang, Y. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.203 · Cited by 67 · see citing works
“…training corpus (Hosseini and Dolan-Gavitt, 2022; Armengol-Estapé et al., 2023; Jiang et al., 2023), In contrast, utilizing larger models trained on broader datasets has proven to substantially improve the performance (Hoffmann et al., 2024; Kaplan et al., 2020; Rozière et al., 2023; OpenAI, 2023).”
InfluentialSource Code Summarization in the Era of Large Language Models
Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., & Chen, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00034 · Cited by 95 · see citing works
A review of LLMs and their applications in the architecture, engineering and construction industry
Kampelopoulos, D., Tsanousa, A., Vrochidis, S., & Kompatsiaris, I. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11241-7 · Cited by 69 · see citing works
LLaFS: When Large Language Models Meet Few-Shot Segmentation
Zhu, L., Chen, T., Ji, D., Ye, J., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00296 · Cited by 59 · see citing works
“For the LLM, we employ CodeL-lama [42] with 7 billion parameters that have been fine-tuned through instruction tuning.”
MethodSelf-citationEvolving code with a large language model
Hemberg, E., Moskal, S., & O’Reilly, U.-M. (2024) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-024-09494-2 · Cited by 57 · see citing works
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
“Following precedent in code generation for evaluation in Pass@1 (Chen et al., 2021; Rozière et al., 2023), we simply use greedy decoding for all models.”
MethodInfluentialFormal requirements engineering and large language models: A two-way roadmap
Ferrari, A., & Spoletini, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107697 · Cited by 59 · see citing works
Large language models for drug discovery and development
Zheng, Y., Koh, H. Y., Ju, J., Yang, M., May, L. T., Webb, G. I., Li, L., Pan, S., & Church, G. M. (2025) · Patterns · doi:10.1016/j.patter.2025.101346 · Cited by 89 · see citing works
“General LLMs such as GPT-4 13 and CodeLlama, 80 trained on extensive code datasets, are well-suited for generating these plans.”
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 56 · see citing works
“Additional models include CodeGen [19], [101], StarCoder [40], WizardCoder [67], OctoCoder [102] and CodeLlama [27], which have demonstrated their potential to solve complex programming problems and assist developers in various settings.”
BackgroundMethodHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
TelecomGPT: A Framework to Build Telecom-Specific Large Language Models
Zou, H., Zhao, Q., Tian, Y., Bariah, L., Bader, F., Lestable, T., & Debbah, M. A. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3593184 · Cited by 72 · see citing works
“Other examples of domain-specific LLMs include MedPalM2 in the medical domain, ChatLaw [23] and SaulLM [24] in the legal domain, and Code-Llama [25] in the code generation domain, are among the several examples that motivates the need for domain-specific LLMs in order to ensure reliable integration…”
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · arXiv · doi:10.48550/arxiv.2402.15938 · Cited by 52
“For code generation tasks, we use CodeLlama-7B (Rozière et al., 2023) and CodeGen-6.”
HPC-Coder: Modeling Parallel Programs using Large Language Models
Nichols, D., Marathe, A., Menon, H., Gamblin, T., & Bhatele, A. S. (2024) · ISC High Performance (ISC) · doi:10.23919/isc.2024.10528929 · Cited by 59 · see citing works
“In addition to being used for natural language, such models have recently been applied to many programming language related tasks [2]– [4].”
BackgroundVisEval: A Benchmark for Data Visualization in the Era of Large Language Models
Chen, N., Zhang, Y., Xu, J., Ren, K., & Yang, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456320 · Cited by 53 · see citing works
“5 [48], Gemini-Pro [61] and CodeLlama-7B [53].”
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59 · see citing works
Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., & Li, G. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.716 · Cited by 52 · see citing works
LLM4VV: Developing LLM-driven testsuite for compiler validation
Munley, C., Jarmusch, A., & Chandrasekaran, S. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.034 · Cited by 49 · see citing works
RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
Silva, A., Fang, S., & Monperrus, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3581062 · Cited by 49 · see citing works
“For instance, state-of-the-art models such as LLaMA [19] and CodeLLaMA [24] range from 7B to 70B parameters.”
InfluentialDreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming
Giunchi, D., Numan, N., Gatti, E., & Steed, A. J. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00078 · Cited by 46 · see citing works
“Using LLMs [14,66], our approach leverages the generation of procedural code, aligning with the principles of low-code programming [35].”
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65 · see citing works
“Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
MethodLarge scale foundation models for intelligent manufacturing applications: a survey
Zhang, H., Semujju, S. D., Wang, Z., Lv, X., Xu, K., Wu, L., Jia, Y., Wu, J., Liang, W., Zhuang, R., et al. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02536-7 · Cited by 52 · see citing works
Feng, Y., Zhou, L., Ma, C., Zheng, Y., He, R., & Li, Y. (2024) · GigaScience · doi:10.1093/gigascience/giae082 · Cited by 46 · see citing works
ReposVul: A Repository-Level High-Quality Vulnerability Dataset
Wang, X., Hu, R., Gao, C., Wen, X.-C., Chen, Y., & Liao, Q. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3647634 · Cited by 43 · see citing works
GeoCode-GPT: A large language model for geospatial code generation
Hou, S., Shen, Z., Zhao, A., Liang, J., Gui, Z., Guan, X., Li, R., & Wu, H. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104456 · Cited by 45 · see citing works
Enhanced automated code vulnerability repair using large language models
de-Fitero-Dominguez, D., Garcia-Lopez, E., Garcia-Cabot, A., & Martinez-Herraiz, J.-J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109291 · Cited by 43 · see citing works
“Code Llama (Rozière et al., 2023) and Mistral (A. Q. Jiang et al., 2023) have a foundational understanding of various programming languages, such as C. Fine-tuning them on a dataset focused on C code vulnerabilities sharpens and emphasizes their inherent knowledge, improving their ability to…”
BackgroundMethodInfluentialShi, H. H., Fang, L., Chen, X., Gu, C., Ma, K., Zhang, X., Zhang, Z., Gu, J., & Lim, E. G. (2024) · IET Smart Grid · doi:10.1049/stg2.12191 · Cited by 46 · see citing works
“Applying LLMs to generate and prompt code showcases [175] their adaptability and expertise within professional domains.”
Hajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 51 · see citing works
“We provide the details of these models in Appendix A.”
BackgroundMethodInfluentialLLM-enabled Intent-driven Service Configuration for Next Generation Networks
Mekrache, A., & Ksentini, A. (2024) · IEEE International Conference on Network Softwarization (NetSoft) · doi:10.1109/netsoft60951.2024.10588881 · Cited by 44 · see citing works
“Pre-trained open-source LLMs, like Code Llama [7], lack training on network-specific data, making them challenging to employ for network management purposes without further training.”
MethodInfluentialTeaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
Wang, C., Zhang, J., Feng, Y., Li, T., Sun, W., Liu, Y., & Peng, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714462 · Cited by 40 · see citing works
“Recent developments have introduced a range of LLMs for code (code LLMs), including Codex [15], CodeT5 [54], CodeT5 + [53], InCoder [21], AlphaCode [29], CodeGen [33], and CodeLlama [37], built upon the Transformer model architecture [45].”
InfluentialExploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
“1 For example, Meta released the Code Llama models [67], which are initialized from the Llama 2 models [73] and trained on 500B tokens from a code-heavy dataset.”
InfluentialSelf-citationDeploying and Evaluating LLMs to Program Service Mobile Robots
Hu, Z., Lucchetti, F., Schlesinger, C., Saxena, Y., Freeman, A., Modak, S., Guha, A., & Biswas, J. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3360020 · Cited by 40 · see citing works
“5 [33] ( text-davinci-003 ), and 3) PaLM2 [34] ( text-bison-001 ) as state-of-the-art API-only proprietary models; and 4) CodeL-lama [35] ( Python-34b-hf ) and 5) StarCoder [36] as open-access models.”
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 38 · see citing works
“Protected attributes are highlighted in bold.”
CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680328 · Cited by 41 · see citing works
LLM-PBE: Assessing Data Privacy in Large Language Models
Li, Q., Hong, J., Xie, C., Tan, J., Xin, R., Hou, J., Yin, X., Wang, Z., Hendrycks, D., Wang, Z., et al. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681994 · Cited by 42 · see citing works
“The results of using DEAs on Github are presented in Table 11, where we also evaluate CodeLlama [95], which is an LLM for code based on Llama-2.”
MethodFighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37 · see citing works
Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Ma, W., Liu, S., Zhao, M., Xie, X., Wang, W., Hu, Q., Zhang, J., & Liu, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664606 · Cited by 43 · see citing works
“Next, we also study 3 LLMs, CodeLlama-7b (CL) [ 58 ], StarCoder (SC) [ 34 ], and CodeT5+ [ 70 ] (CodeT5p-770m) (CT5+) using the Java dataset.”
InfluentialAre LLMs ready for Visualization?
Vázquez, P. (2024) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis60374.2024.00049 · Cited by 37 · see citing works
“Before starting the analysis, we performed some tests with different LLMs, including ChatGPT3 [6], ChatGPT4 [14], CodeLlama [17], Mixtral, etc.”
MethodOpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
Xing, S., Qian, C., Wang, Y., Hua, H., Tian, K., Zhou, Y., & Tu, Z. (2025) · 2025 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00113 · Cited by 37 · see citing works
“MLLM for AD Multimodal Large Language Models (MLLMs) [4, 23, 25, 26, 31, 34, 57] extend the capabilities of Large Language Models (LLMs) [6, 15, 39, 40, 42, 49, 50, 54, 55, 62] into the visual realm.”
AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMs
Mingzhe, G., Zhao, J., Lin, Z., Ding, W., Hou, X., Feng, Y., Li, C., & Guo, M. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00033 · Cited by 38 · see citing works
“We use some open-source pre-trained LLMs as our base models, including Codellama-7B [25], DeepSeek-Coder-6.”
MethodBTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs
Izzo, R. A., Bardaro, G., & Matteucci, M. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802304 · Cited by 39 · see citing works
“We selected Llama-2-7b [19], the foundation model, Llama-2-7b-chat [19], designed for conversational tasks, and CodeLlama-7b-Instruct [15], with a focus on code-generation.”
Framework for evaluating code generation ability of large language models
Yeo, S., Ma, Y., Kim, S. C., Jun, H., & Kim, T. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0357 · Cited by 34 · see citing works
“Previous studies [4, 9, 17, 20] have often assessed LLMs from a limited perspective, mainly focusing on particular programming languages or difficulty levels.”
BackgroundInfluentialDeep Learning for Code Intelligence: Survey, Benchmark and Toolkit
Wan, Y., Bi, Z., He, Y., Zhang, J., Zhang, H., Sui, Y., Xu, G., Jin, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3664597 · Cited by 34 · see citing works
“In terms of code embedding, notable advancements have been achieved by employing deep learning and NLP techniques to encode source code.”
InfluentialInteractive Planning Using Large Language Models for Partially Observable Robotic Tasks
Sun, L., Jha, D. K., Hori, C., Jain, S., Corcodel, R., Zhu, X., Tomizuka, M., & Romeres, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610981 · Cited by 35 · see citing works
“Researchers have proposed strategies to prompt LLMs to perform high-quality synthetic data by in-context learning[36], task decomposition[24], [37], as well as methods to improve fine-tuning performance [38], [39], and pre-train model for specific tasks like coding [40].”
MethodGong, A., Qiao, Z., Li, X., Lyu, J., & Li, X. (2024) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2024.105474 · Cited by 33 · see citing works
Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets
Chakraborty, P., Arumugam, K. K., Alfadel, M., Nagappan, M., & McIntosh, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3423712 · Cited by 32 · see citing works
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 32 · see citing works
“The process leads to four LLMs as our subjects, i.e., InCoder [18], CodeGeex [39], StarCoder [40], and CodeLlama [41].”
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 33 · see citing works
Arafat, M. E., Sourav, S., & Tamás, O. (2024) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy62279.2024.10737601 · Cited by 30 · see citing works
“…via real-time replies and natural language Nowadays, LLMs have also gained significant attention, particularly since OpenAI [5] released its now-popular GPT production have emerged, including Anthropic’s Claude, Facebook’s CodeLlama [6], and OpenAI’s Codex, which powers Github’s Copilot [7].”
BackgroundProject-specific code summarization with in-context learning
Yun, S. M., LIN, S.-J., Gu, X., & Shen, B. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112149 · Cited by 29 · see citing works
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 33 · see citing works
“While some work has evaluated the performance of open-source LLMs [36], since then, multiple open-source LLMs have been introduced thanks to the increased interest in LLMs [12], [30], [37], [38], which to the best of our knowledge have not yet been evaluated in the software engineering literature.”
BackgroundOn the Effectiveness of LLM-as-a-Judge for Code Generation and Summarization
Crupi, G., Tufano, R., Velasco, A., Mastropaolo, A., Poshyvanyk, D., & Bavota, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586082 · Cited by 33 · see citing works
“Furthermore, we asked five LLMs ( i.e., CodeL-lama 7B, 13B, and 34B [10], GPT-3.”
InfluentialLLMs are one-shot URL classifiers and explainers
Rashid, F., Ranaweera, N., Doyle, B., & Seneviratne, S. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.111004 · Cited by 30 · see citing works
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30 · see citing works
“After the initial experimental deployment, we finally choose 3 generalized PEFT techniques (which can be successfully run on the experimental models in this paper), namely LoRA [51], AdaLoRA [52], and IA3 [53].”
Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests
Deljouyi, A., Koohestani, R., Izadi, M., & Zaidman, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00032 · Cited by 28 · see citing works
“The prompt component of UTGen uses the code-llama:7b-instruct model from Meta [44] as provided by Ollama 1 .”
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 43 · see citing works
“To comprehend, generate, and interpret human language, leading to profound impacts across diverse domains from tech-focused areas like finance 14,15 and programming 16,17 to human-centric sectors like education 18,19 and healthcare 20,21 .”
SecureQwen: Leveraging LLMs for vulnerability detection in python codebases
Mechri, A., Ferrag, M. A., & Debbah, M. A. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104151 · Cited by 27 · see citing works
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40 · see citing works
“Specifically, we mainly consider these dimensions of them: (1) general or code specified (e.g., GPT-4-Turbo [44] v.s. CodeLlama [6]); (2) scale of parameters; (3) open-source or closed-source (e.g., GPT-3.”
RLCoder: Reinforcement Learning for Repository-Level Code Completion
Wang, Y., Wang, Y., Guo, D., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00014 · Cited by 43 · see citing works
“To evaluate the effectiveness of RLCoder, we compare it with the RawRAG framework [35] and Re-poCoder [13] with five backbone LLMs, i.e., CodeLlama-7B [3], StartCoder-7B [4], StarCoder2-7B, DeepSeekCoder-1B [5], and DeepSeekCoder-7B.”
BackgroundMethodLAAG-RV: LLM Assisted Assertion Generation for RTL Design Verification
Maddala, K., Mali, B., & Karfa, C. (2024) · IEEE International Test Conference India (ITC India) · doi:10.1109/itcindia62949.2024.10651860 · Cited by 26 · see citing works
“Examples of fine-tuned LLMs include Star Coder [16] and CodeLlama [17].”
AgentBoard: An analytical evaluation board of multi-turn LLM agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2365 · Cited by 26 · see citing works
Mixture-of-Agents Enhances Large Language Model Capabilities
Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.04692 · Cited by 25 · see citing works
“For instance, some models excel at complex instruction following (Xu et al., 2023a) while others may be better suited for code generation (Roziere et al., 2023; Guo et al., 2024).”
BackgroundInfluentialNashaat, M., & Miller, J. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428324 · Cited by 26 · see citing works
“In the second phase, the fi ne-tuning dataset is used to fi ne-tune Code Llama [56].”
MethodDelving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study
Liu, S., Keung, J., Yang, Z., Liu, F., Zhou, Q., & Liao, Y. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00055 · Cited by 36 · see citing works
“Decoder-only models of code, like GPT-C [51], CodeGen [40], and Code Llama [47], are on the basis of GPT series [45], which predicts text when given the preceding context.”
MethodStagedVulBERT: Multigranular Vulnerability Detection With a Novel Pretrained Code Model
Jiang, Y., Zhang, Y., Su, X., Treude, C., & Wang, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3493245 · Cited by 25 · see citing works
“In comparison, the recently proposed models such as DeepSeek-Coder [23], CodeLlama [24], StarCoder [25], WizardCoder [26], Mis-tral [27], Phi-2 [28], and ChatGPT [29] serve as baselines.”
MethodSearch-Based LLMs for Code Optimization
Gao, S., Gao, C., Gu, W., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00021 · Cited by 25 · see citing works
“We compare SBLLM against four representative prompting methods on four popular open-source and closed-source LLMs including CodeLlama [20], Gem-ini [21], ChatGPT [22], and GPT-4 [23].”
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 26 · see citing works
“They can provide general-purpose information on a variety of topics, including computer programming (Rozière et al. 2023), writing assistance (Touvron et al. 2023; OpenAI 2023c), cooking instructions (OpenAI 2023c) and many others.”
BackgroundQiskit Code Assistant: Training LLMs for generating Quantum Computing Code
Dupuis, N., Buratti, L., Vishwakarma, S., Forrat, A. V., Kremer, D., Faro, I., Puri, R., & Cruz-Benito, J. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691762 · Cited by 24 · see citing works
“For example, to cite just a few, StarCoder [6], Code Llama [7], and DeepSeek Coder [8] all show impressive performances on various code benchmarks.”
BackgroundLeveraging Local LLMs for Secure In-System Task Automation With Prompt-Based Agent Classification
Sriram, S., Karthikeya, C. S., Kumar, K. K. R. S., Vijayaraj, N., & Thangavel, M. (2024) · IEEE Access · doi:10.1109/access.2024.3505298 · Cited by 24 · see citing works
“Three models were examined – qwen2 [27], Mistral 7B [28], codeLlama 7B [29].”
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25 · see citing works
- Authors
- Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., & Synnaeve, G.
- Affiliations
- École nationale des ponts et chaussées (FR)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2308.12950
- OpenAlex
- W4386185625
- Cited by
- 531
- References
- 92
- Citations by year
- 2026: 65 · 2025: 232 · 2024: 216 · 2023: 18
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- CodeLlama, code large language models, Llama 2, HumanEval, MBPP, code generation benchmarks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
