Qwen2.5-Coder Technical Report
Open accessAbstract
In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B). As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general and math skills. These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will advance research in code intelligence and, with its permissive licensing, support wider adoption by developers in real-world applications.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
6 of 14 cited sources checked · 5 supported · 1 flagged · accuracy not yet measured
- Findingcitation
The YARN mechanism enables the model to handle sequences up to 131,072 (128K) tokens.
“To further leverage the model's extrapolation potential, we applied the YARN mechanism (Peng et al., 2023), enabling the model to handle sequences up to 131,072 (128K) tokens.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Peng et al., 2023 source
- Findingnone given
Public repositories from GitHub created before February 2024, spanning 92 programming languages, were collected.
“We collected public repositories from GitHub created before February 2024, spanning 92 programming languages.”
Quote in the paperSubstantial claim - Findingnone given
The dataset Qwen2.5-Coder-Data comprises five key data types: Source Code Data, Text-Code Grounding Data, Synthetic Data, Math Data and Text Data.
“This dataset comprises five key data types: Source Code Data, Text-Code Grounding Data, Synthetic Data, Math Data and Text Data.”
Quote in the paperNot a substantial claim - Findingown data
The pretraining dataset for Qwen2.5-Coder comprises over 5.5 trillion tokens.
“Significant efforts have been dedicated to constructing a large-scale, coding-specific pretraining dataset comprising over 5.5 trillion tokens.”
Quote in the paperSubstantial claim - Findingown data
The 7:2:1 ratio of Code:Text:Math outperformed the 100:0:0 and 85:10:5 ratios for Qwen2.5-Coder-7B.
“Interestingly, we found that the 7:2:1 ratio outperformed the others, even surpassing the performance of groups with a higher proportion of code.”
Quote in the paperSubstantial claim
- Findingown data
Through 4-stage filtering, the average scores on HumanEval and MBPP increased from 41.6% to 46.8% compared to the baseline.
“Through 4-stage filtering, the average scores on HumanEval and MBPP increased from 41.6% to 46.8% compared to the baseline, demonstrating the value of high-quality Text-Code Grounding Data for code generation.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder outperforms larger code models on a variety of tasks.
“This model outperforms larger code models on a variety of tasks.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B has comparable performance with other models exceeding 20 billion parameters on CrossCodeEval.
“Meanwhile, Qwen2.5-Coder-7B has a comparable performance with other models exceeding 20 billion parameters.”
Quote in the paperSubstantial claim - Findingown data
The pretraining dataset is sourced from public code repositories such as GitHub and large-scale web-crawled data containing code-related texts.
“This dataset is sourced from a broad range of public code repositories, such as those on GitHub, as well as large-scale web-crawled data containing code-related texts.”
Quote in the paperSubstantial claim - Findingown data
The final training mixture is 70% Code, 20% Text, and 10% Math.
“Ultimately, we selected a final mixture of 70% Code, 20% Text, and 10% Math.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct has an advantage in preference alignment on CodeArena.
“The results in Figure 9 demonstrate the advantage of Qwen2.5-Coder-32B-Instruct in preference alignment.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct significantly outperformed other open-source code models (including DS-Coder-V2-Instruct) on CRUXEval.
“significantly outperforming other open-source code models (including DS-Coder-V2-Instruct)”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct achieved an accuracy of 31.4% on LiveCodeBench.
“our Qwen-2.5-Coder-32B-Instructmodel achieved an accuracy of 31.4%”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct is close to the performance of several closed-source APIs on MultiPL-E.
“Instruct model with only 32 billion parameters, bringing it very close to the performance of several closed-source APIs.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder is capable of successfully completing the Needle in the Code task within a 128k length range.
“The figure below shows that Qwen2.5-Coder is capable of successfully completing this task within a 128k length range.”
Quote in the paperSubstantial claim- Rests on an assumption: Long context capability is crucial for code LLMs, serving as the core skill for understanding repository-level code and becoming a code agent.
- Rests on an assumption: Most current code models still have very limited support for length, which hinders their potential for practical application.
- Findingown data
Qwen2.5-Coder instruct models deliver superior code reasoning.
“The Qwen2.5-Coder instruct models stand out for delivering superior code reasoning”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct excels compared to other open-source models on the McEval benchmark, particularly across a wide range of programming languages.
“the Qwen2.5-Coder-32B-Instructmodel excels when compared to other open-source models on the McEval benchmark, particularly across a wide range of programming languages.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B-Instruct outperformed larger models such as CodeStral-22B-v0.1 and DS-Coder-33B-Instruct on LiveCodeBench.
“it also outperformed larger models, such as CodeStral-22B-v0.1 and DS-Coder-33B-Instruct.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder outperforms other code models of the same size on the Text-to-SQL task.
“As shown in Figure 12, Qwen2.5-Coderoutperforms other code models of the same size on the Text-to-SQL task.”
Quote in the paperSubstantial claim- Rests on an assumption: SQL is one of the essential tools in daily software development and production, but its steep learning curve often hinders free interaction between non-programming experts and databases.
- Findingown data
Qwen2.5-Coder-7B-Instruct achieved a Pass@1 accuracy of 37.6% on LiveCodeBench (2407-2409).
“the Qwen-2.5-Coder-7B-Instructmodel achieved an impressive Pass@1 accuracy of 37.6%”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B-Instruct outperformed larger models including CodeStral-22B and DS-Coder-33B-Instruct on CRUXEval.
“the Qwen2.5-Coder-7B-Instructmodel outperformed larger models, including CodeStral-22B and DS-Coder-33B-Instruct”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct achieves comparable or better performance on MdEval compared to LLMs with larger model sizes.
“the Qwen2.5-Coder-32B-Instructachieves a comparable or better performance even compared to LLMs with larger model sizes.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder supports input lengths of up to 128K tokens.
“Through careful data proportioning and organization, we have enabled it to support input lengths of up to 128K tokens.”
Quote in the paperSubstantial claim- Rests on an assumption: Long context capability is crucial for code LLMs, serving as the core skill for understanding repository-level code and becoming a code agent.
- Rests on an assumption: Most current code models still have very limited support for length, which hinders their potential for practical application.
- Findingown data
Qwen2.5-Coder has achieved state-of-the-art performance across all sizes.
“Qwen2.5-Coder has achieved state-of-the-art performance across all sizes”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B-Instruct achieved Input-CoT and Output-CoT accuracies of 65.8% and 65.9% on CRUXEval.
“the Qwen2.5-Coder-7B-Instructmodel achieved Input-CoT and Output-CoT accuracies of 65.8% and 65.9%, respectively”
Quote in the paperSubstantial claim - Findingown data
There is a positive correlation between model size and model performance.
“There is a positive correlation between model size and model performance”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B-Instruct demonstrated a substantial improvement over DS-Coder-V2-Lite-Instruct on CRUXEval, with gains of 12.8% in Input-CoT accuracy and 13.0% in Output-CoT accuracy.
“demonstrating a substantial improvement over the DS-Coder-V2-Lite-Instruct model, with gains of 12.8% in Input-CoT accuracy and 13.0% in Output-CoT accuracy.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct achieved accuracies of 75.2% and 83.4% on Input-CoT and Output-CoT on CRUXEval.
“our Qwen2.5-Coder-32B-Instructmodel achieved accuracies of 75.2% and 83.4% on Input-CoT and Output-CoT, respectively”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B outperforms the previous best dense model, DS-Coder-33B, across all five metrics.
“Qwen2.5-Coder-7Boutperforms the previous best dense model, DS-Coder-33B, across all five metrics.”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-32B-Instruct achieves the best performance of 45.1 on TableBench.
“Figure 13 demonstrates that Qwen2.5-Coder-32B-Instructgets the best performance 45.1 on TableBench.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wu et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Qwen2.5-Coder-7B-Instruct surpasses larger models such as CodeStral-22B and DS-Coder-33B-Instruct on the Aider code editing benchmark.
“Remarkably, it also surpasses larger models such as CodeStral-22B and DS-Coder-33B-Instruct”
Quote in the paperSubstantial claim - Findingown data
Qwen2.5-Coder-7B-Instruct significantly outperformed other models with similar parameter counts on LiveCodeBench.
“significantly outperforming other models with similar parameter counts.”
Quote in the paperSubstantial claim - Study design and conductreasoning
Decontamination was performed on all data, including both pre-training and post-training datasets, using a 10-gram overlap method.
“The filtering was done using a 10-gram overlap method, where any training data with a 10-gram word-level overlap with the test data was removed.”
Quote in the paperSubstantial claim - Premisenone given
In comparison with Claude-3.5-Sonnet and GPT-4o, code LLMs are still falling behind.
“However, in comparison with the recently state-of-the-art proprietary LLMs, Claude-3.5-Sonnet(Anthropic, 2024) and GPT-4o (OpenAI, 2024), the code LLMs are still falling behind, either open-source or proprietary models.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Anthropic, 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites OpenAI, 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Study design and conductnone given
The latest 4 months of LiveCodeBench (2024.07 to 2024.11) questions were selected as the evaluation to strictly avoid test data contamination.
“For the instruction model, we select the latest 4 months of LiveCodeBench (2024.07∼2024.11)questions as the evaluation to strictly avoid test data contamination, truly reflecting the OOD capabilities of the LLM.”
Quote in the paperSubstantial claim - Study design and conductown data
The authors adopted an internal annotated evaluation benchmark called CodeArena, including nearly 400 human-curated samples, to evaluate alignment performance of Qwen2.5-Coder-32B-Instruct with human preferences.
“we adopted an internal annotated evaluation benchmark called CodeArena, including nearly 400 human-curated samples.”
Quote in the paperNot a substantial claim - Study design and conductcitation
A unified prompt template was used as input to ensure a fair comparison between Qwen2.5-Coder and other open-source language models on the Text-to-SQL task.
“To ensure a fair comparison between Qwen2.5-Coder and other open-source language models on this task, we used a unified prompt template as input, following the work of Chang & Fosler-Lussier (2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Chang & Fosler-Lussier (2023) source
- Interpretationreasoning
Qwen2.5-Coder's high CRUXEval scores are due to the focus on executable quality during the code cleaning process.
“thanks to our focus on executable quality during the code cleaning process”
Quote in the paperSubstantial claim - Interpretationown data
The use of finely crafted synthetic data during both pre-training and fine-tuning significantly enhanced Qwen2.5-Coder's capability in Text-to-SQL tasks.
“Thanks to the use of finely crafted synthetic data during both pre-training and fine-tuning, we significantly enhanced Qwen2.5-Coder'scapability in Text-to-SQL tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: SQL is one of the essential tools in daily software development and production, but its steep learning curve often hinders free interaction between non-programming experts and databases.
- Interpretationreasoning
Qwen2.5-Coder's mathematical strengths likely stem from its strong foundation built on Qwen2.5 and the careful mixing of code and mathematical data during training.
“which likely stem from two key factors: first, the model's strong foundation built on Qwen2.5, and second, the careful mixing of code and mathematical data during training, which has ensured a well-balanced performance across these domains.”
Quote in the paperNot a substantial claim
- Classified as probably not a substantial claim. 27
- May combine two claims. 21
- The quote only partly states this claim. 4
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMBPP+ offers 35 times more test cases than the original MBPP.
“Similarly, MBPP+ offers 35 times more test cases than the original MBPP.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Liu et al., 2023 source
The Conclusion & Future Work section says the authors plan to apply EvalPlus to MBPP in the future, rather than reporting an MBPP+ benchmark. The paper reports an 80-fold test-case increase for HumanEval+, not a 35-fold increase for MBPP+. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationHumanEval consists of 164 manually written programming tasks.
“HumanEval consists of 164 manually written programming tasks, each providing a Python function signature and a docstring as input to the model.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Chen et al., 2021 source
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationEvalPlus extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions.
“To further ensure accurate evaluation, EvalPlus (Liu et al., 2023) extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions in HumanEval.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Liu et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMBPP comprises 974 programming problems created by crowdsource contributors.
“MBPP, on the other hand, comprises 974 programming problems created by crowdsource contributors.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Austin et al., 2021 source
Classified as probably not a substantial claim.
PremisecitationRepoEval is a benchmark designed to evaluate repository-level code completion capabilities across three granularities: line, API invocation, and function body completion.
“RepoEval is a benchmark designed to evaluate repository-level code completion capabilities across three granularities: line, API invocation, and function body completion.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Zhang et al., 2023
May combine two claims.
Findingown dataQwen2.5-Coder achieves state-of-the-art performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size.
“These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Findingnone givenIn repo-level pretraining, the context length is extended from 8,192 tokens to 32,768 tokens, and RoPE's base frequency is adjusted from 10,000 to 1,000,000.
“In this stage, the context length is extended from 8,192 tokens to 32,768 tokens, and RoPE's base frequency is adjusted from 10,000 to 1,000,000.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenThe architecture of Qwen2.5-Coder is derived directly from Qwen2.5.
“The architecture of Qwen2.5-Coder is derived directly from Qwen2.5.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenThe inclusion of mathematical data did not negatively impact the model's performance on code tasks.
“Importantly, the inclusion of mathematical data did not negatively impact the model's performance on code tasks.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenQwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens.
“As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenIn file-level pretraining, the maximum training sequence length is 8,192 tokens, covering 5.2T of high-quality data.
“In this stage, the maximum training sequence length is set to 8,192 tokens, covering 5.2T of high-quality data.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataThe Qwen2.5-Coder series surpasses all other models of a similar size on CrossCodeLongEval.
“The Qwen2.5-Coder series surpasses all other models of a similar size.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone givenQwen2.5-Coder-32B-Instruct has become the current SOTA open-source code model, matching the coding capabilities of GPT-4o.
“(1) Powerful: Qwen2.5-Coder-32B-Instruct has become the current SOTA open-source code model, matching the coding capabilities of GPT-4o.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenQwen2.5-Coder series includes six models: 0.5B, 1.5B, 3B, 7B, 14B, 32B.
“This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe final training dataset comprises 5.2 trillion tokens.
“The final training dataset comprises 5.2 trillion tokens.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Findingnone givenQwen2.5-Coder-32B-Instruct possesses good general and mathematical skills.
“While demonstrating strong and comprehensive coding abilities, it also possesses good general and mathematical skills.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataQwen2.5-Coder-32B-Instruct achieves Pass@1 and Pass@2 rates of 60.9% and 73.7%, respectively, on the Aider code editing benchmark.
“Our Qwen2.5-Coder-32B-Instructmodel achieves even higher accuracy, with Pass@1 and Pass@2 rates reaching 60.9% and 73.7%, respectively.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe Qwen2.5-Coder series excels not only in complex coding tasks but also in advanced general-purpose tasks, setting it apart from its competitors.
“The results in Table 20 highlight the versatility of the Qwen2.5-Coderseries, which excels not only in complex coding tasks but also in advanced general-purpose tasks, setting it apart from its competitors.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder achieved a score of 56.5 on CRUXEval-I and 56.0 on CRUXEval-O.
“Qwen2.5-Coderdelivered highly promising results, achieving a score of 56.5 on CRUXEval-I and 56.0 on CRUXEval-O”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder-7B-Instruct achieves a PASS@1 accuracy of 51.9% on the Aider code editing benchmark, significantly outperforming comparable models.
“Despite its relatively modest scale of 7 billion parameters, it achieves an impressive PASS@1 accuracy of 51.9%, significantly outperforming comparable models.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder-32B-Instruct achieves a win rate on CodeEditorBench comparable to DS-Coder-V2-Instruct (86.2% win rate), which has a 236 billion parameter scale.
“The results in Figure 11 show that Qwen2.5-Coder-32B-Instructachieves a win rate comparable to DS-Coder-V2-Instruct (86.2% win rate), which features a significantly larger 236 billion parameter scale.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Guo et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: An effective code assistant must excel in generating code based on given specifications, as well as in modifying or debugging existing code to meet evolving requirements or resolve issues.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenQwen2.5-Coder series instruct models inherited the high performance of the base model on Multi-Programming Language.
“The Qwen2.5-Coder series instruct models have inherited the high performance of the base model on the Multi-Programming Language.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataMBPP-3shot is more suitable for evaluating base models and correlates well with the actual performance of the models.
“Our extensive experiments show that MBPP-3shot is more suitable for evaluating base models and correlates well with the actual performance of the models.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder-7B-Instruct outperformed other instruct models with comparable parameter sizes on BigCodeBench-Instruct, achieving 41.0% on the full subset and 18.2% on the hard subset.
“the Qwen2.5-Coder-7B-Instructmodel outperformed other instruct models with comparable parameter sizes, achieving notably high accuracy scores on both the full and hard subsets, reaching 41.0% on the full subset and 18.2% on the hard subset.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataQwen2.5-Coder shows strengths in mathematics on MATH, GSM8K, MMLU-STEM, and TheoremQA.
“Table 12 highlights Qwen2.5-Coder'sstrengths in mathematics”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenQwen2.5-Coder series instruct models demonstrated outstanding performance in code generation tasks.
“our Qwen2.5-Coderseries instruct models similarly demonstrated outstanding performance in code generation tasks.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataQwen2.5-Coder surpasses the results of other open-source large language models by a significant margin across six core areas: code generation, code reasoning, code editing, text-to-sql, mathematical reasoning and general natural language performance.
“For the evaluation of the instruct models, we rigorously assessed six core areas: code generation, code reasoning, code editing, text-to-sql, mathematical reasoning and general natural language performance with the fewest parameters, surpassing the results of other open-source large language models by a significant margin.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder have shown impressive performance in basic code generation, achieving state-of-the-art results among open-source models of the same size and surpassing even larger models.
“Qwen2.5-Coder have shown impressive performance in basic code generation, achieving state-of-the-art results among open-source models of the same size and surpassing even larger models.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataQwen2.5-Coder-32B-Instruct achieved accuracy rates of 49.6% on the complete split and 27.0% on the hard split on BigCodeBench-Instruct.
“the Qwen2.5-Coder-32B-Instructachieved accuracy rates of 49.6% on the complete split and 27.0% on the hard split”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningQwen2.5-Coder's advantage in general natural language capabilities compared to other coders validates the effectiveness of the Qwen2.5-Coder data mixing strategy.
“Coder's advantage in general natural language capabilities compared to other coders, further validating the effectiveness of Qwen2.5-Coderdata mixing strategy.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataQwen2.5-Coder-32B-Instruct is the best-performing open-source code generation model and surpasses several closed-source APIs on BigCodeBench-Instruct.
“establishing it as the best-performing opensource code generation model and surpassing several closed-source APIs.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone givenTo ensure a broad comparison, the authors included some of the most popular and widely-used open-source instruction-tuned models, notably versions from the DeepSeek-Coder series and Codestral models.
“To ensure a broad comparison, we included some of the most popular and widely-used open-source instruction-tuned models, notably versions from the DeepSeek-Coder series and Codestral models.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenThe pre-training data was curated by collecting public code data and extracting high-quality code-related content from web texts, while filtering out low-quality data using advanced classifiers.
“To ensure the quality of the pre-training data, we have curated a dataset by collecting public code data and extracting high-quality code-related content from web texts, while filtering out low-quality data using advanced classifiers.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe whole edit-format was consistently applied across all the paper's experiments.
“The whole edit-format was consistently applied across all our experiments.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenAll evaluation code is publicly accessible for reproducibility.
“All evaluation code is publicly accessible for reproducibility”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe evaluation on CRUXEval was structured to ensure a fair and thorough comparison across models.
“The evaluation was structured to ensure a fair and thorough comparison across models.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataQwen2.5-Coder-32B-Instruct surpassed all open-source code generation models and reached a level comparable to many closed-source APIs on LiveCodeBench.
“surpassing all open-source code generation models and reaching a level comparable to many closed-source APIs.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone givenSAFIM is a syntax-aware fill-in-the-middle benchmark that emphasizes AST-based code completion, specifically targeting algorithmic blocks, control-flow expressions, and API function calls.
“SAFIM is a syntax-aware fill-in-the-middle benchmark that emphasizes AST-based code completion, specifically targeting algorithmic blocks, control-flow expressions, and API function calls.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisenone givenThe benchmark SAFIM consists of 17,720 examples from 8,590 code files created after April 2022, deliberately avoiding overlap with mainstream pretraining corpora.
“The benchmark consists of 17,720 examples from 8,590 code files created after April 2022, deliberately avoiding overlap with mainstream pretraining corpora.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductcitationThe same evaluation approach used in the original CodeEditorBench paper was employed, relying on win rate as the metric for overall performance across diverse problem types.
“We employed the same evaluation approach used in the original paper, relying on win rate as the metric for overall performance across diverse problem types.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Guo et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: An effective code assistant must excel in generating code based on given specifications, as well as in modifying or debugging existing code to meet evolving requirements or resolve issues.
Classified as probably not a substantial claim.
PremisecitationCrossCodeLongEval is a long context benchmark on cross file code completion tasks.
“CrossCodeLongEval is a long context benchmark on cross file code completion tasks.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wu et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisecitationCrossCodeEval is a benchmark that requires a deep understanding of cross-file context to accurately complete the code.
“CrossCodeEval is a benchmark that requires a deep understanding of this cross-file context to accurately complete the code.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ding et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: Code completion often depends on accessing cross-file context and dependencies.
Probably not a substantial claim: may not pass S4 (based).
Study design and conductreasoningThe standardized prompt template minimizes biases that may arise from prompt variations.
“This standardized prompt template minimizes biases that may arise from prompt variations.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
There is limited evidence regarding the scalability of balancing Code, Math, and Text data to large datasets.
“Although the research community has explored this balance before, there is limited evidence regarding its scalability to large datasets.”
The paper relies on this without support.
Human feedback is highly labor-intensive.
“Given that human feedback is highly labor-intensive, we use a multilingual code sandbox to provide code execution feedback, while an LLM is utilized for human judgment feedback.”
The paper relies on this without support.
Code completion often depends on accessing cross-file context and dependencies.
“In real-world scenarios, code completion often depends on accessing cross-file context and dependencies.”
The paper relies on this without support.
Reasoning based on code helps determine whether a model truly understands the reasoning flow behind the code.
“Code is a highly abstract form of logical language, and reasoning based on code helps us determine whether a model truly understands the reasoning flow behind the code.”
The paper relies on this without support.
Mathematics and coding are closely intertwined, so a powerful code model should exhibit strong mathematical capabilities.
“Mathematics and coding have always been closely intertwined.Mathematics forms the foundational discipline for coding, while coding serves as a vital tool in mathematical fields.As such, we expect an open and powerful code model to exhibit strong mathematical capabilities as well.”
The paper relies on this without support.
Long context capability is crucial for code LLMs, serving as the core skill for understanding repository-level code and becoming a code agent.
“Long context capability is crucial for code LLMs, serving as the core skill for understanding repository-level code and becoming a code agent.”
The paper relies on this without support.
Most current code models still have very limited support for length, which hinders their potential for practical application.
“However, most of the current code models still have very limited support for length, which hinders their potential for practical application.”
The paper relies on this without support.
SQL is one of the essential tools in daily software development and production, but its steep learning curve often hinders free interaction between non-programming experts and databases.
“SQL is one of the essential tools in daily software development and production, but its steep learning curve often hinders free interaction between non-programming experts and databases.”
The paper relies on this without support.
An effective code assistant must excel in generating code based on given specifications, as well as in modifying or debugging existing code to meet evolving requirements or resolve issues.
“An effective code assistant must excel in generating code based on given specifications, as well as in modifying or debugging existing code to meet evolving requirements or resolve issues.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs' proficiency at generating code in mainstream programming languages such as Python and JavaScript does not transfer to logic programming.
“LLMs have achieved remarkable proficiency at generating code in mainstream programming languages such as Python and JavaScript, yet this proficiency does not transfer to logic programming.”
From From Reasoning to Code: GRPO Optimization for Underrepresented Languages (2026) · cites this paper as Hui et al. (2024)
The Multi-Programming Language section (page unavailable) reports strong code-generation results across eight mainstream languages, including Python and JavaScript. It does not evaluate logic programming or establish that proficiency fails to transfer to it; the Code Reasoning section instead evaluates reasoning about Python functions. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCompared to DeepSeek-R1, which consumes 35,200 tokens, SafeCodeRL driven by Qwen2.5-Coder achieves higher functional performance with only 10,500 tokens, reducing token consumption by approximately 70%.
“Compared to DeepSeek-R1 [45], which consumes 35,200 tokens for black-box internal trial-and-error, SafeCodeRL driven by Qwen2.5-Coder [50] achieves higher functional performance with only 10,500 tokens, reducing token consumption by approximately 70%.”
From SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software (2026) · cites this paper as [50]
They employ Qwen2.5, Qwen2.5-Math, and Qwen2.5-Coder models to synthesize trillions of text tokens in different formats, covering dozens of domains.
“Besides, we employ Qwen2.5 (Yang et al., 2024b), Qwen2.5-Math(Yang et al., 2024c), and Qwen2.5-Coder(Hui et al., 2024) models to synthesize trillions of text tokens in different formats, including textbooks, question-answering, instructions, and code snippets, covering dozens of domains.”
From Qwen3 Technical Report (2025) · cites this paper as Hui et al., 2024
Models released at different times (DeepSeek-Coder, Qwen2.5-Coder, Seed-Coder) achieve similarly strong performance with only minor differences in average ES scores when used with LongCodeZip.
“Models released at different times, from DeepSeek-Coder in 2023 [10] to Qwen2.5Coder in 2024 [11] and Seed-Coder in 2025 [12], achieve similarly strong performance with only minor differences in average ES scores.”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [11]
With Qwen3-Coder 480B, Observation Masking achieved a solve rate of 54.8%, a slight improvement over LLM-Summary's 53.8%.
“With the Qwen3-Coder 480B [27, 8] model, Observation Masking achieved a solve rate of 54.8%, a slight improvement over the LLM-Summary's 53.8%.”
From The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management (2025) · cites this paper as [8]
The Abstract identifies the paper as a report on six Qwen2.5-Coder models, ranging from 0.5B to 32B—not Qwen3-Coder 480B. A search of our copy for Observation Masking, LLM-Summary, and the stated solve rates found no passage addressing the comparison. (LLM, conf=0.76)
Experimental — accuracy not yet measured
LeetCode and DevEval are among the most widely adopted benchmarks for evaluating LLM code generation capabilities.
“we argue that LeetCode and DevEval are among the most widely adopted benchmarks for evaluating LLM code generation capabilities [30, 43, 66]”
From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as [30]
Section 7.2 (page unavailable) mentions LeetCode as a source of problems for LiveCodeBench, not as a benchmark whose adoption is assessed. Searches for DevEval and for the claimed adoption of either benchmark found no passage addressing the claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
The stated 7B Aider Pass@1 score belongs to another result
MajorThe prose reports 51.9% for Qwen2.5-Coder-7B-Instruct, but Table 19 gives it 55.6%. The table assigns 51.9% to DS-Coder-V2-Instruct.
“it achieves an impressive PASS@1 accuracy of 51.9%”
“| Qwen2.5-Coder-7B-Instruct | 7B | 55.6 | 68.4 |”
The reported 7B LiveCodeBench accuracy disagrees with the table
MajorThe prose gives Qwen2.5-Coder-7B-Instruct a Pass@1 accuracy of 37.6%, whereas the LiveCodeBench Pass@1 column lists 18.2 for that model.
“the Qwen-2.5-Coder-7B-Instructmodel achieved an impressive Pass@1 accuracy of 37.6%”
“| Qwen2.5-Coder-7B-Instruct | 7B 88.4 | 84.1 | 83.5 | 71.7 | 41.0 | 18.2 | 18.2 |”
The middle data-mixture ratio differs between text and table
MinorThe prose specifies a ratio of 85:10:5, but Table 3 lists 85:15:5. The table's components also total 105 rather than 100.
“we compared three different Code for Qwen2.5-Coder-7B:Text ratios -100:0:0, 85:10:5, and 70:20:10.”
“| 85 | 15 | 5 | 43.3 | 36.2 | 26.1 | 52.5 | 56.8 | 57.1 | 70.0 | 48.9 |”
The 0.5B model's head size does not match its hidden size
MinorFor the 0.5B model, a hidden size of 896 divided among 14 query heads gives 64 dimensions per head, not the reported 128.
Reported 128 · recomputed 64
“| Configuration | 0.5B | 1.5B | 3B | 7B | 14B | 32B | | --- | --- | --- | --- | --- | --- | --- | | Hidden Size | 896 | 1,536 | 2048 | 3,584 | 5120 | 5120 | | # Layers | 24 | 28 | 36 | 28 | 48 | 64 | | # Query Heads | 14 | 12 | 16 | 28 | 40 | 40 | | # KV Heads | 2 | 2 | 2 | 4 | 8 | 8 | | Head Size | 128 | 128 | 128 | 128 | 128 | 128 |”
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“With the rapid development of large language models (LLMs) (Brown, 2020; Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Jiang et al., 2023; Bai et al., 2023b; Yang et al., 2024; Anthropic, 2024; OpenAI, 2024), code-specific language models have garnered significant attention in the…”
BackgroundMeigs, J. (1954) · The Yale Journal of Biology and Medicine · doi:10.1515/cclm.1994.32.8.631 · Cited by 5,224
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 1× · see citing works
“After file-level pretraining, we turn to repo-level pretraining, aimed at enhancing the model’s long-context capabilities.”
MethodSelf-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1× · see citing works
“With the rapid development of large language models (LLMs) (Brown, 2020; Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Jiang et al., 2023; Bai et al., 2023b; Yang et al., 2024; Anthropic, 2024; OpenAI, 2024), code-specific language models have garnered significant attention in the…”
BackgroundSelf-citationJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1× · see citing works
“The filtering was done using a 10-gram overlap method, where any training data with a 10-gram string-level overlap with the test data was removed.”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1× · see citing works
“5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
BackgroundYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 2× · see citing works
“…natural language understanding, we selected MMLU (Hendrycks et al., 2021) and its variant MMLU-Redux (Gema et al., 2024), along with four other benchmarks: ARC-Challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2019), and HellaSwag (Zellers et al., 2019).”
MethodTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899 · see citing works
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 2× · see citing works
“…natural language understanding, we selected MMLU (Hendrycks et al., 2021) and its variant MMLU-Redux (Gema et al., 2024), along with four other benchmarks: ARC-Challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2019), and HellaSwag (Zellers et al., 2019).”
MethodDirect Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686 · cited 1× · see citing works
“5-Coder does not produce inflated results due to test set leakage, we performed decontamination on all data, including both pre-training and post-training datasets.”
MethodTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 2× · see citing works
“We removed key datasets such as HumanEval, MBPP, GSM8K, and MATH.”
MethodCode Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1× · see citing works
“Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
MethodGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · cited 1× · see citing works
“With the rapid development of large language models (LLMs) (Brown, 2020; Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Jiang et al., 2023; Bai et al., 2023b; Yang et al., 2024; Anthropic, 2024; OpenAI, 2024), code-specific language models have garnered significant attention in the…”
BackgroundSelf-citationThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 2× · see citing works
“…natural language understanding, we selected MMLU (Hendrycks et al., 2021) and its variant MMLU-Redux (Gema et al., 2024), along with four other benchmarks: ARC-Challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2019), and HellaSwag (Zellers et al., 2019).”
MethodHellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 408 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 1× · see citing works
“5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
BackgroundLiu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0943 · Cited by 384 · cited 2× · see citing works
“5, a top-tier open-source LLM, Qwen2.”
BackgroundSelf-citationJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · cited 1× · see citing works
“5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
BackgroundSelf-citationMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 4× · see citing works
“To evaluate general natural language understanding, we selected MMLU (Hendrycks et al., 2021) and its variant MMLU-Redux (Gema et al., 2024), along with four other benchmarks: ARC-Challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2019), and HellaSwag…”
MethodInfluentialDirect Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 1× · see citing works
“5-Coder’s mathematical performance, we selected five popular benchmarks, including MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), MMLU-STEM (Hendrycks et al., 2020) and TheoremQA (Chen et al., 2023).”
MethodLi, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Cao, R., Geng, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.03111 · Cited by 278
Self-citationTheoremQA: A Theorem-driven Question Answering datasetPreprint
Chen, W., Yin, M., Ku, M. W. F., Wan, Y., Ma, X., Xu, J.-Y., Xia, T., Wang, X., & Lu, P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.12524 · Cited by 278 · cited 1× · see citing works
“5-Coder’s mathematical performance, we selected five popular benchmarks, including MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), MMLU-STEM (Hendrycks et al., 2020) and TheoremQA (Chen et al., 2023).”
MethodStarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
TableBench: A Comprehensive and Complex Benchmark for Table Question AnsweringPreprint
Wu, X.-J., Yang, J., Chai, L.-Z., Zhang, G., Liu, J.-H., Du, X.-R., Liang, D., Shu, D., Cheng, X., Sun, T., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.09174 · Cited by 195 · cited 1× · see citing works
“However, in comparison with the recently state-of-the-art proprietary LLMs, Claude-3.”
ResultSelf-citationYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.08887 · Cited by 193 · see citing works
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.14196 · Cited by 180 · cited 10× · see citing works
“5-Coder-7B-Base stands as the leading model among those over 6 billion parameters, matching the performance of the formidable 33 billion parameter model, DS-Coder-33B-Base.”
MethodInfluentialChatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
ReACC: A Retrieval-Augmented Code Completion Framework
Lu, S., Duan, N., Han, H., Guo, D., Hwang, S.-W., & Svyatkovskiy, A. (2022) · arXiv · doi:10.48550/arxiv.2203.07722 · Cited by 96 · see citing works
AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions
Li, Z.-M., Zang, Q., Ma, D., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.20424 · Cited by 87
Self-citationWinoGrande: An Adversarial Winograd Schema Challenge at ScalePreprint
Sakaguchi, K., Bras, R. J. L., Bhagavatula, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.10641 · Cited by 84 · see citing works
StarCoder 2 and The Stack v2: The Next Generation
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.19173 · Cited by 74 · cited 2× · see citing works
“Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
MethodSelf-citationYang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10671 · Cited by 63 · see citing works
Self-citationSantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · cited 2× · see citing works
“Built upon pretrained LLMs, code LLMs such as the StarCoder series (Li et al., 2023; Lozhkov et al., 2024), CodeLlama series (Roziere et al., 2023), DeepSeek-Coder series (Guo et al., 2024), CodeQwen1.”
MethodTheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 52 · see citing works
Efficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47 · cited 1× · see citing works
“5-Coder utilizes the Fill-In-the-Middle (FIM) training strategy, as introduced in Bavarian et al. (2022), enabling the model to generate code that is contextually coherent.”
MethodLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2403.07974 · Cited by 42 · cited 1× · see citing works
“5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
BackgroundEvaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle TasksPreprint
Gong, L., Wang, S., Elhoushi, M., et al. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2403.04814 · Cited by 41 · see citing works
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Ding, Y., Wang, Z., Ahmad, W. U., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11248 · Cited by 37
WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning
Yu, Z., Zhang, X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., & Yin, Q. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.280 · Cited by 31 · cited 1× · see citing works
“If a given sample contains very little code data or even no code snippets, the sample will possibly be classified into “No Programming Language” tag.”
BackgroundBigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.15877 · Cited by 29 · see citing works
Self-citationMagicoder: Empowering Code Generation with OSS-Instruct
Wei, Y., Wang, Z., Liu, J., Ding, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2312.02120 · Cited by 27 · cited 1×
“We keep the instruction data of the mainstream programming languages and randomly discard a portion of the instruction data of the long-tail languages.”
MethodSelf-citationM2rc-Eval: Massively Multilingual Repository-level Code Completion EvaluationPreprint
Liu, J.-H., Deng, K., Liu, C., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.21157 · Cited by 26 · cited 1× · see citing works
“However, in comparison with the recently state-of-the-art proprietary LLMs, Claude-3.”
ResultSelf-citationYaRN: Efficient Context Window Extension of Large Language Models
Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023) · arXiv · doi:10.48550/arxiv.2309.00071 · Cited by 21 · cited 1× · see citing works
“To further leverage the model’s extrapolation potential, we applied the YARN mechanism (Peng et al., 2023), enabling the model to handle sequences up to 131,072 (132K) tokens.”
MethodMdEval: Massively Multilingual Code Debugging
Liu, S.-K., Chai, L.-Z., Yang, J., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2411.02310 · Cited by 20
Self-citationMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17 · cited 3× · see citing works
“5-Coder’s proficiency in handling multiple programming languages, we selected the MultiPL-E (Cas-sano et al., 2022) and choose to evaluate eight mainstream languages from these benchmark, including Python, C++, Java, PHP, TypeScript, C#, Bash and JavaScript.”
BackgroundMethodHow to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings
Chang, S., & Fosler-Lussier, E. (2023) · arXiv · doi:10.48550/arxiv.2305.11853 · Cited by 14 · see citing works
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., & Chen, W. (2023) · arXiv · Cited by 10 · see citing works
Self-citationUniCoder: Scaling Code Large Language Model via Universal Code
Sun, T., Chai, L., Yang, J., Yin, Y., Guo, H., Liu, J., Wang, B., Yang, L., & Li, Z. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.100 · Cited by 10 · see citing works
Self-citationCRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., & Wang, S. I. (2024) · arXiv · doi:10.48550/arxiv.2401.03065 · Cited by 7 · cited 4× · see citing works
“For both CRUXEval-I and CRUXEval-O, we used a chain-of-thought (CoT) approach, requiring the LLM to output steps sequentially during simulated execution.”
MethodInfluentialRepoformer: Selective Retrieval for Repository-Level Code Completion
Wu, D., Ahmad, W. U., Zhang, D., Ramanathan, M. K., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.10059 · Cited by 7 · see citing works
Gema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04127 · Cited by 1 · cited 2× · see citing works
“To evaluate general natural language understanding, we selected MMLU (Hendrycks et al., 2021) and its variant MMLU-Redux (Gema et al., 2024), along with four other benchmarks: ARC-Challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2019), and HellaSwag…”
MethodMcEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 1 · cited 5× · see citing works
“5-Coder-7B-Instruct model excels when compared to other open-source models on the McEval benchmark, particularly across a wide range of programming languages.”
BackgroundMethodInfluentialSelf-citationCodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Guo, J., Li, Z., Liu, X., Ma, K., Zheng, T., Yu, Z., Pan, D., LI, Y., Liu, R., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.03543 · Cited by 1 · cited 1× · see citing works
“However, in comparison with the recently state-of-the-art proprietary LLMs, Claude-3.”
ResultUniCoder: Scaling Code Large Language Model via Universal Code
Sun, T., Chai, L., Yang, J., Yin, Y., Guo, H., Liu, J., Wang, B., Yang, L., & Li, Z. (2024) · arXiv · doi:10.48550/arxiv.2406.16441 · cited 1×
“Specifically, we use the LLM to generate the instruction from the code snippets within 1024 tokens and then we use the code LLM to generate the response Sun et al. (2024).”
MethodSelf-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“With the rapid development of large language models (LLMs) (Brown, 2020; Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Jiang et al., 2023; Bai et al., 2023b; Yang et al., 2024; Anthropic, 2024; OpenAI, 2024), code-specific language models have garnered significant attention in the…”
BackgroundCrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Ding, Y., Wang, Z., Ahmad, W., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2023
Flagged first, then by citations
Cited by
Sorted by citations
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185 · see citing works
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Self-citationXu, H., Zhang, N., Yin, Z., & Njock, P. G. A. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106257 · Cited by 43 · see citing works
OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale
Li, H., Wu, S., Zhang, X., Huang, X., Zhang, J., Jiang, F., Wang, S., Zhang, T., Chen, J., Shi, R., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749723 · Cited by 56 · see citing works
Self-citationWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17 · see citing works
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 15 · see citing works
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13 · see citing works
Mathur, S., van der Vleuten, N., Yager, K. G., & Tsai, E. H. R. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/add9e4 · Cited by 11 · see citing works
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 7 · see citing works
Lv, J., Si, J., Gao, D., & Bao, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.10.005 · Cited by 6 · see citing works
STEP: A structured prompt optimization method for SCADA system tag generation using LLMs
Ma, F.-Y., Li, D., Liu, Y., Lan, D., & Pang, Z.-B. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100832 · Cited by 9 · see citing works
Fang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
Action Engine: Automatic Workflow Generation in FaaS
Esashi, A., Lertpongrujikorn, P., Kato, S., & Salehi, M. A. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107947 · Cited by 5 · see citing works
“Two key factors that contribute to this evaluation challenge are as follows: 1.”
McIntosh, F., Murina, S., Chen, L., et al. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100020 · Cited by 5 · see citing works
Support, Not Automation: Towards AI-supported Code Review For Code Quality and Beyond
Heander, L. G., Söderberg, E., & Rydenfält, C. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728505 · Cited by 4 · see citing works
Toward AI-driven fire imagery: Attributes, challenges, comparisons, and the promise of VLMs and LLMs
Boroujeni, S. P. H., Mehrabi, N., Afghah, F., McGrath, C. P., Bhatkar, D., Biradar, M. A., & Razi, A. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100763 · Cited by 5 · see citing works
Large Language Model Guided Self-Debugging Code GenerationPreprint
Adnan, M., Kuhn, C. C. N., & Xu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5396508 · Cited by 6 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
When Retrieval Augmentation Meets API Documentation: Can LLMs Code with Less-Common Libraries?
Chen, J., Chen, S., Cao, J., Shen, J., & Cheung, S.-C. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821424 · Cited by 3 · see citing works
Idrissi-Yaghir, A., Arzideh, K., Schäfer, H., Eryılmaz, B., Bahn, M., Wen, Y., Borys, K., Hartmann, E. M., Schmidt, C. S., Pelka, O., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73540 · Cited by 5 · see citing works
“Qwen 2.5 Coder [17] is a series of code-specific language models developed by Alibaba Cloud’s Qwen team.”
InfluentialOn the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3 · see citing works
VERINA: Benchmarking Verifiable Code Generation
Ye, Z., Yan, Z., He, J., Kasriel, T., Yang, K., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2505.23135 · Cited by 3 · see citing works
“The costs marked with asterisks include the additional expenses incurred during iterative proof refinement experiments, which required up to 64 refinement attempts per datapoint.”
Župan, M. (2025) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.70011 · Cited by 2 · see citing works
Xu, P., Wu, H., Wang, J., Lin, R., & Tan, L. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4298-4_2 · Cited by 2 · see citing works
COSMosFL: Ensemble of Small Language Models for Fault Localisation
Cho, H., Kang, S., An, G., & Yoo, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00007 · Cited by 2 · see citing works
“5-Coder 7B [13].”
FullStack Bench: Evaluating LLMs as Full Stack Coders
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.00535 · Cited by 2 · see citing works
Self-citationNguyen, T. A. H., Nguyen, Q.-D., & Nguyen, L. T. T. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-4606-7_26 · Cited by 1 · see citing works
An end-to-end approach for fixing concurrency bugs via SHB-based context extractor
Li, Z., Yi, Q., Xiao, K., Ji, Z., & Liang, H. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112888 · Cited by 1 · see citing works
“5-Coder [97] typically employ interpolation techniques like YaRN to extend their context windows from 4K/8K tokens up to 128K tokens or more [101].”
Structural Semantic Enhancement: Better integrating code semantics for vulnerability detection
Wang, S., Wu, Y., Cui, Z., & Chen, L. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107824 · Cited by 2 · see citing works
Self-citationSundaram, G., Venktesh, B., V, & Sundharakumar, K. B. (2025) · International Conference on Emerging Technologies in Computing and Communication (ETCC) · doi:10.1109/etcc65847.2025.11108633 · Cited by 1 · see citing works
“…Llama [7] test DocString generation on CodeXGLUE [8] using Bilingual Language Understanding (BLUE) score, while the other three models used in the study Code Gemma [9], DeepSeek Coder [10], and Qwen 2.5 Coder [11] does not state information related to DocString benchmarking in their technical work.”
InfluentialLi, D., Li, Y., Chen, X., & Zhang, W. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3607 · Cited by 1 · see citing works
Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localization
Caumartin, G., & Melo, G. (2026) · doi:10.1145/3786161.3788460 · Cited by 1 · see citing works
ECCOS: Efficient Capability and Cost Coordinated Scheduling for Multi-LLM ServingPreprint
Mei, K., Xu, W., Lin, S., & Zhang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5159339 · Cited by 1 · see citing works
Self-citationDoes Few-Shot Learning Help LLM Performance in Code Synthesis?
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.02906 · Cited by 3 · see citing works
“5-coder (Hui et al., 2024), Magicoder (Wei et al., 2023), Artigenz-Coder (Agarwal et al., 2024) etc. have been fine-tuned specifically for programming tasks, showing improved performance in code completion and bug fixing.”
SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents
Wang, Y., Shi, Y., Yang, M., Zhang, R., He, S., Lian, H., Chen, Y., Ye, S., Cai, K., & Gu, X. (2026) · arXiv · doi:10.48550/arxiv.2601.16746 · Cited by 2 · see citing works
“5-Coder-7B-Instruct (Hui et al., 2024).”
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7 · see citing works
“5-Coder [ 11] also undergoes instruction tuning and RL, with a focus on long-context fidelity through multi-stage alignment.”
InfluentialFrom Reasoning to Code: GRPO Optimization for Underrepresented Languages
PENNINO, F., RAIMONDI, B., RONDELLI, M., GURIOLI, A., & GABBRIELLI, M. (2026) · Theory and Practice of Logic Programming · doi:10.1017/s1471068426100489 · Cited by 1 · see citing works
Context as a Tool: Context Management for Long-Horizon SWE-Agents
Liu, S., Yang, J., Jiang, B., Li, Y., Guo, J., Liu, X., & Dai, B. (2025) · arXiv · doi:10.48550/arxiv.2512.22087 · Cited by 1 · see citing works
Self-citationEnhancing Python Code Embeddings: Fusion of Code2vec with Large Language Models
Ngo, L. H., & Rivalan, J. (2025) · Journal of Engineering Research and Sciences · doi:10.55708/js0401001
Li, Y., Xu, F., Zhang, Z., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5340544
EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
Guo, T., Dong, H., Leng, Y., Liu, F., Lin, C., Xiao, N., & Zhang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_24
Machines flunking an exam: Evaluating large language models on course-related open questionsPreprint
Huang, J., Wei, Y., Zhang, L., Lai, R., Lai, F., & Zheng, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6679662/v1
Self-citationTALE: Token-Adaptive Low-Rank KVCache Approximation with Reconstruction Elimination
Lee, J., Hwang, S.-W., Qiao, A., Campos, D., Yao, Z., & He, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.39
Li, Y., Peng, J., Yang, L., Li, Y., Shi, F., Xue, P., & Xu, C. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4088-4_7
Becsei, Á., Stéger, J., Visontai, D., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-28341-7
CodeRankEval: Benchmarking and Analyzing LLM Performance for Code Ranking
Chen, L.-G., Xiao, Z., Xu, Y.-J., An, R.-C., Wang, X., Li, Y.-N., Li, Y.-H., Wang, Y.-D., Zeng, Z.-R., Gao, Q., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-5514-9
Chang, I., & Park, G.-M. (2026) · Journal of Broadcast Engineering · doi:10.5909/jbe.2026.31.1.77
OPENXRD: a comprehensive benchmark framework for LLM/MLLM XRD question answering
Vosoughi, A., Shahnazari, A., Xi, Y., Zhang, Z., Hess, G., Xu, C., & Abdolrahim, N. (2026) · Digital Discovery · doi:10.1039/d5dd00519a
“5-coder-32b-instruct 78,79 .”
Enhancing Attention-Based Context Attribution via Token Selection and Think-Twice Mechanism
Hsu, T.-H., Huang, S.-Y., Lin, C.-Y., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_40
The Creation of Reasonable Robot Control Behavior in the Form of Executable Code
Скороходов, М. С., Latalin, V. E., Борисович, Р. Р., & Sboev, A. G. (2025) · Moscow University Physics Bulletin · doi:10.3103/s002713492570314x
From Code to Rubrics: A Multimodal Evaluation of LLM Systems for Automated Feedback Generation
Haseeb, M., Amjad, A. I., Amjad, M., & Sheng, V. S. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795458
Struggling at the Start: Structural Causes of Decoding Difficulty in Code Generation
Zhu, Y., Li, J., Jiang, H., Li, S., Zhang, J., & Li, A. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795840
Self-citationAutomatic Addition of SEO Metadata to News Articles Using Qwen-Coder
Salem, H., & Toshchev, A. S. (2026) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105526700202
“First, this model supports extended context, that is, it can process a significantly larger volume of input text in a single request (up to 256 thousand tokens) [7].”
InfluentialBeyond Language Boundaries: Uncovering Programming Language Families for Code Language Models
Yun, S. M., Gu, X., Huang, J. Y., & Shen, B. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797138
“Such integration would simultaneously advance syntactic and intrinsic language analysis while facilitating the exploration of interrelationships and shared characteristics across these linguistic domains.”
Influence-Aware Bayesian-Inspired Token Reweighting for Improved Code Generation
Zhu, Y., Li, G., Mei, H., Jin, Z. G., Li, J., Zheng, Q., & Zhang, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808200
Self-citationSivapiran, S., & Uddin, G. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808123
“In terms of model selection, We recommend prioritizing code-specialized models over general-purpose alternatives and considering larger variants within model families (>=7B parameters) (O3, 5, 20).”
CertiCoder: Towards MISRA-Compliant C Code Generation with LLMs
Gou, M., Yao, Z., Ma, H., Zhang, E.-D., Zhou, J., & He, F. (2026) · Proc. ACM Softw. Eng. · doi:10.1145/3797120
Self-citationPeng, C., Jiang, M., Zhou, Y., & Wu, L. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3808141
“• 𝑓 𝑣 : All private and public state variables in the contract of the focal function.”
InDe-LLM: Defending against Jailbreak Attacks in LLM-Powered Systems via Intention Disentangling
Wang, Y., Zhang, Q., Zhou, C., Go, G., Shi, D., & Jiang, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797136
A multiregional image–text dataset and benchmark for vision-language modeling of plant diseasesPreprint
Nguyen, T. V., Quoc, K. N., Harwath, D. F., Quach, L.-D., & Dao, P. D. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.01.735881
Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., & Zharov, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.21433
“With Qwen3-Coder 480B [27, 8], the best-performing model in our experiments, Observation Masking is not only 52% cheaper than the Raw Agent baseline but also improves on the solve rate by 2.6 %.”
InfluentialA Unified Approach to Routing and Cascading for LLMs
Dekoninck, J., Baader, M., & Vechev, M. (2024) · arXiv · doi:10.48550/arxiv.2410.10347
“Second, to simulate a use-case where ex-ante quality estimation is accurate, we use the Math and Coder models from the Q WEN -2.5 model family (Yang et al., 2024; Hui et al., 2024) and evaluate them on a combination of Min-erva Math (Lewkowycz et al., 2022) and LiveCodeBench (Jain et al., 2024).”
InfluentialZhou, C., Zhang, X., Song, D., Chen, X., Gu, W., Ma, H., Tian, Y., Zhang, M., & Hu, L. (2025) · arXiv · doi:10.48550/arxiv.2502.09183
“5-Coder-32B-Instruct (Hui et al., 2024), DeepSeek-V3 (DeepSeek-AI, 2024), and Llama3.”
InfluentialWhat Makes In-Context Examples Effective for Code Generation?
Li, D., Chen, S., Cao, J., & Cheung, S.-C. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3832275
SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training
Song, H., Huang, L., Sun, S., Jiang, J., Le, R., Cheng, D., Chen, G., Hu, Y., Chen, Z., Jia, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.03411
“5-Coder-32B model) [4].”
InfluentialOmniRouter: Budget and Performance Controllable Multi-LLM Routing
Mei, K., Xu, W., Guo, M., Lin, S., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.20576 · Cited by 2 · see citing works
Self-citationOn the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection
Hannan, M. A., Ni, R., Zhang, C., Jia, L., Mangal, R., & Pasareanu, C. S. (2026) · Proceedings 2026 Workshop on LLM Assisted Security and Trust Exploration · doi:10.14722/last-x.2026.23025
Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models
Xu, Z., Harvill, J., Fan, Z., Sun, Y., Ding, H., & Wang, H. (2026) · arXiv · doi:10.48550/arxiv.2604.15153
Jiang, J., Shen, J., Kim, S., Yoo, K. M., Kim, J., & Kim, S. (2026) · arXiv · doi:10.48550/arxiv.2603.05863
AlphaResearch: Accelerating New Algorithm Discovery with Language Models
Yu, Z., Feng, K., Zhao, Y., He, S., Zhang, X.-P., & Cohan, A. (2025) · arXiv · doi:10.48550/arxiv.2511.08522
MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
Ke, Z., Xu, A., Ming, Y., Nguyen, X.-P., Chin, R., Xiong, C., & Joty, S. (2025) · arXiv · doi:10.48550/arxiv.2505.14996
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Seo, M., Baek, J., Lee, S., & Hwang, S. J. (2025) · arXiv · doi:10.48550/arxiv.2504.17192
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
Cui, T., Zeng, P., Xu, W., Xiao, H., Xue, S., Gao, Q., Ouyang, W., Heng, P. A., Yin, Z., Wu, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.13912
“5-Coder-3B-Instruct [52] with LoRA [53].”
- Authors
- Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y., Zhang, Y., Yang, A., Men, R., Huang, F., Zheng, B., Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., & Lin, J.
- Affiliations
- Alibaba Group (China) (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2409.12186
- OpenAlex
- W4403796339
- Cited by
- 65
- References
- 53
- Citations by year
- 2026: 38 · 2025: 27
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- Qwen2.5-Coder, code generation, code completion, code reasoning, code repair, synthetic data generation, code intelligence, large language models, Qwen2.5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
