Seed-Coder: Let the Code Model Curate Data for Itself

“To further improve model performance and better align the distribution between the pretraining model and post-training data, we constructed high-quality datasets by combining the quality score with iteratively trained fastText [Joulin et al., 2016, 2017] models.”

“The context includes the pre-commit code snapshot’s README, directory structure, and top 5 relevant files retrieved via BM25 [Robertson and Zaragoza, 2009].”

“HumanEval [Chen et al., 2021] consists of 164 manually written programming tasks in Python, each of which provides a function signature with a docstring for LLMs to complete a solution, whose functional correctness is judged by a handful of test cases.”

“We collected challenging real-world coding problems, including CodeContests [Li et al., 2022] and ICPC problems 2 , and gathered model-generated solutions from DeepSeek-R1 [DeepSeek-AI et al., 2025] on these tasks.”

“We believe that these limitations align with insights from the widely cited essay “The Bitter Lesson” [Sutton, 2019]: AI researchers often favor human-centric methods due to their short-term advantages, yet these approaches inevitably plateau and even inhibit long-term progress; the breakthrough…”

“To build Seed-Coder-8B-Instruct, we first applied supervised fine-tuning (SFT) to adapt our base models to a wide range of real-world tasks, followed by direct preference optimization (DPO; Rafailov et al. [2023]) to further enhance specific capabilities such as code generation and reasoning.”

“MBPP [Austin et al., 2021] is created by crowd-sourcing participants to write 974 Python programming problems, each of which is comprised of a function signature with a docstring, as well as a few test cases.”

“To rigorously benchmark the correctness of LLM-generated code, EvalPlus [Liu et al., 2023] extends the test cases of the original HumanEval by 80 × to build HumanEval + .”

“Specifically, we employed the widely adopted 10-gram filtering method [Guo et al., 2024, Hui et al., 2024], which removed all data that had any 10-gram word overlap with key benchmark datasets, including HumanEval, MBPP, MHPP, BigCodeBench, LiveCodeBench, Aider, FullStack Bench, etc.”

“We first classified the topic of each instance using tagging techniques [Lu et al., 2023].”

“Although synthetic data offers an effective and cost-efficient alternative to real-13 world data [Liu et al., 2024a], it often lacks the stylistic diversity observed in actual user prompts.”

“Single-line MultiPL-HumanEval FIM [Allal et al., 2023] refines MultiPL-E [Cassano et al., 2023] into a set of single-line infilling challenges across Python, Java, and JavaScript, which serves as a classic benchmark for FIM evaluation.”

“During the RL training process, similar to DeepCoder [Luo et al., 2025], we used the open-source verl framework [Sheng et al., 2025] for GRPO training [Shao et al., 2024], and adopted optimization techniques similar to DAPO [Yu et al., 2025, ByteDance Seed, 2025].”

“One possible explanation for this outperformance is the positional bias in attention mechanisms [Yu et al., 2024, Hsieh et al., 2024]: in SPM mode, the beginning of the suffix and the end of the prefix are positioned at opposite ends of the input token sequence, both crucial for middle-content…”

“For each level, we performed exact-deduplication using SHA256 hashes of contents and near-deduplication via the MinHash algorithm [Kiveris et al., 2014, Lee et al., 2022].”

“During the RL training process, similar to DeepCoder [Luo et al., 2025], we used the open-source verl framework [Sheng et al., 2025] for GRPO training [Shao et al., 2024], and adopted optimization techniques similar to DAPO [Yu et al., 2025, ByteDance Seed, 2025].”

“The fully upgraded version StarCoder2 consists of 3B, 7B, and 15B parameters models trained on 3.3 to 4.3 trillion tokens of the Stack v2 [Lozhkov et al., 2024], with 67.5TB code data spanning 619 programming languages.”

“We evaluated Seed-Coder and competitor models on two benchmarks to assess their effectiveness in realistic software engineering scenarios: • SWE-bench Verified [Jimenez et al., 2024] is a benchmark designed to evaluate the ability of LLMs to solve real-world GitHub issues.”

“Single-line MultiPL-HumanEval FIM [Allal et al., 2023] refines MultiPL-E [Cassano et al., 2023] into a set of single-line infilling challenges across Python, Java, and JavaScript, which serves as a classic benchmark for FIM evaluation.”

“However, in contrast to the more commonly adopted PSM mode, we found that SPM performs slightly better during the training process, which is consistent with [Bavarian et al., 2022a].”

“To mitigate these issues, LiveCodeBench [Jain et al., 2025] introduces live updates by continuously collecting new problems from coding contests across prominent competitive programming platforms – LeetCode, AtCoder, and CodeForces – and tagging each problem with a release date.”

“The original MBPP benchmark only contains Python tasks, while MBXP [Athiwaratkun et al., 2023] transforms the problems and unit tests in MBPP to 10+ programming languages.”

“CrossCodeEval [Ding et al., 2023] is composed of single-line infilling tasks across Python, TypeScript, C# and Java.”

“Recently, a challenging benchmark named Mostly Hard Python Problems (MHPP) was introduced [Dai et al., 2024], consisting of 210 human-curated problems accompanied by unit tests.”

“To further improve the model’s ability to respond to real-world queries, we additionally incorporated code-related data sampled from WildChat [Zhao et al., 2024] into our SFT dataset.”

“Performance of instruct models on MBXP. tasks, the recently proposed benchmark BigCodeBench [Zhuo et al., 2025] challenges LLMs to invoke multiple function calls as tools from 139 libraries across 7 domains, encompassing 1,140 Python tasks with rich context.”

“• OpenHands [Wang et al., 2025] is a fully autonomous agent platform that has no requirements of any fixed workflow.”

“For each level, we performed exact-deduplication using SHA256 hashes of contents and near-deduplication via the MinHash algorithm [Kiveris et al., 2014, Lee et al., 2022].”

“As introduced in Sec 5.1.3, CRUXEval [Gu et al., 2024] is designed to measure LLM’s code reasoning capability by requiring the model to predict output from a given input (CRUXEval-O), or predict the input from a given output (CRUXEval-I).”

“During the RL training process, similar to DeepCoder [Luo et al., 2025], we used the open-source verl framework [Sheng et al., 2025] for GRPO training [Shao et al., 2024], and adopted optimization techniques similar to DAPO [Yu et al., 2025, ByteDance Seed, 2025].”

“However, as pointed out by Yue et al. [2025], a large volume of distillation data may exceed the inherent capabilities of the base model and thus fail to accurately reflect its true potential.”

“One possible explanation for this outperformance is the positional bias in attention mechanisms [Yu et al., 2024, Hsieh et al., 2024]: in SPM mode, the beginning of the suffix and the end of the prefix are positioned at opposite ends of the input token sequence, both crucial for middle-content…”

“Our evaluation protocols and primary baselines are adapted from [20].”

“In contrast, autoregressive models must resort to cumbersome workarounds for infilling, such as permuting the target span to the end of the sequence (Fried et al., 2023; Guo et al., 2024a; Hui et al., 2024; Seed et al., 2025).”

“Following Seed-Coder (Seed et al., 2025), we concatenate multiple source files within the same repository to form training samples at the project-level code structure, preventing the model from learning only isolated code snippets and promoting understanding of real-world engineering context.”

“0 (Seed et al., 2025) under SearchSWE, remain competitive, while several high-cost configurations do not lie on the performance frontier.”

“• Seed-Coder-8B-Reasoning (Seed et al., 2025) is an 8B-parameter open-source coding LLM optimized for code generation, leveraging Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning.”

“Building on the observations, we propose Stable-DiffCoder, a diffusion language model that reuses the autoregressive Seed-Coder [49] pipeline and data but achieves stronger performance.”

“…tasks such as mathematical problems (Xie et al., 2025; Shao et al., 2024; Li et al., 2025c; Yu et al., 2025b; Li et al., 2025d) or code execution (Seed et al., 2025; Hui et al., 2024; Wang et al., 2025a), GRMs can effectively evaluate a much broader range of tasks including social intelligence…”

“These RL-driven approaches have proven effective across mathematical problem-solving [23, 41] and code generation [22], establishing RL as a key technique for eliciting advanced capabilities in large-scale models.”

“That includes twelve models spanning six distinct model families: SeedCoder [46], OpenCoder [41], Qwen 2.5 [43], QwenCoder 2.5 [47], DeepSeek Coder V1 [48], Llama 3 [49] and CodeL-lama [9].”

“5-Coder [ 13], Qwen3 [39], and Seed-Coder [32] demonstrating strong performance across a variety of coding benchmarks.”

“Large Language Models (LLMs) have demonstrated remarkable performance in software engineering and have been widely adopted for code generation [6, 12, 23].”

“5-Coder in 2024 [11] and Seed-Coder in 2025 [12], achieve similarly strong performance with only minor differences in average ES scores.”

“General-purpose code models (OpenAI, 2025; Li et al., 2022; Roziere et al., 2023; Seed et al., 2025; Hui et al., 2024) have achieved strong results on benchmarks such as HumanEval (Chen et al., 2021) and SWE-bench (Jimenez et al., 2023), especially for well-represented languages with abundant…”

“…[27]; Qwen3-235B-A22B-Instruct-2507 and Qwen3-235B-A22B-Thinking-2507 [77]; Qwen3-Coder-30B-A3B-Instruct and Qwen3-Coder-480B-A35B-Instruct [55]; Seed-Coder-8B-Instruct [59]; Kimi-Dev-72B [82]; Kimi-K2-Instruct-0905 and Kimi-K2-Thinking [67]; KAT-Dev and KAT-Dev-72B-Exp [86]; and GLM-4.”

“…[31], and AutoTriton [15] employ multi-round RL to train models for generating efficient kernels; the CUDA-L1/L2 [17, 25] series and Seed-Coder [22] utilize large-scale sampling and contrastive learning to enable models to generate matrix multiplication operators that surpass closed-source…”

“Empirical results indicate that even state-of-the-art reasoning models, such as Qwen3 (Yang et al., 2025) and Seed-Coder-Reasoning (Seed et al., 2025), struggle to generate functionally correct CUDA kernels.”

“5-Coder (Hui et al., 2024), OpenCoder (Huang et al., 2025), OlympicCoder (Hug-ging Face, 2025), and Seed-Coder (Seed et al., 2025); diffusion baselines (Ho et al., 2020) include Dream (Ye et al., 2025), LLaDA (Nie et al., 2026), Diffu-Coder (Gong et al., 2025), Dream-Coder (Xie et al., 2025), and d1 (Zhao et al., 2026…”

“Few adjustments were necessary, since the problems already specified programs and tests using standard I/O.”

“…remarkable improvements in reasoning performance, particularly on benchmarks with verifiable answers such as mathematics (Guo et al., 2025; Zeng et al., 2025; Team et al., 2025; He et al., 2025), coding (Seed et al., 2025; Liu et al., 2025), and symbolic problem solving (Xie et al., 2025).”

“As shown in Figure 1, LLMs can interpret natural-language specifications and generate corresponding structural modeling code [15, 18, 36], enabling automated construction of simulation-ready models.”

“Recently, Sun et al. (2025) uses classification as an exploratory probe to detect differentiating output patterns of non-reasoning LLMs, such as Grok-2 and Gemini-1.”

“We also validate with Seed-Coder-8B-Instruct (ByteDance Seed et al., 2025), obtaining similar results (see Section G).”

“For data curation , Computational Thinking [192] structures decomposition around abstraction and modularity, whereas Seed-Coder [127] self-evolves by generating and filtering training data.”

“An isolated variable name or operator may not carry complete semantics on its own; only when combined into an assignment statement, conditional branch, or return expression does it constitute code evidence that truly affects program behavior.”

“1.1) [1] and are powered by the SeedCode LLM via API [25].”

“It has been widely used in synthesizing high-quality training data [3, 4, 41].”

“5-Coder in 2024 [11] and Seed-Coder in 2025 [12], achieve similarly strong performance with only minor differences in average ES scores.”

“Advanced training strategies, including domain knowledge distillation [16] and self-curated data synthesis (e.g., Seed-Coder [36]), along with reasoning-enhanced models like Seed-Thinking [23], continue to improve code understanding and generation.”

“Firstly, code-oriented LLMs (Rozière et al., 2023; Li et al., 2023; Seed et al., 2025; Huang et al., 2024) such as DeepSeek-Coder (Guo et al., 2024) and Qwen3-Coder (Yang et al., 2025a) have demonstrated considerable capabilities in code understanding and generation.”

“7B-Instruct [27], and Seed-Coder-8B-Instruct [46].”

“Data-LevelMitigation DataCleaningandFiltering Execution-Feedback-BasedFiltering Qwen2 [139] ,TeleChat [131] ,KimiK2 [117] ,Seed-Coder [110] Rule-orTool-BasedFilteringandSanitization LessIsMore [146] ,Qwen [9] ,PackageHallucinations [113] ,CIDRe [26] , Infinite-Instruct [137] ,QualityIn,QualityOut…”

“Large language models (LLMs) have recently achieved remarkable success in automatic programming, spanning domains from competitive programming [1, 16, 23] to software engineering [9, 14, 36].”

“Our framework extends these ideas by employing both theorem provers and LLM-based judges to iteratively select high-quality autoformalization candidates, resulting in a monotonic improvement process at test time.”

“Large Language Models (LLMs) have achieved remarkable success across real-world applications—such as virtual assistants [2], content creation [17], and scientific discovery [20]—driven by their powerful capabilities in context comprehension, generation, and reasoning [3, 21].”

“We use Seed-Coder [14] to sample 10 implementations per specification.”

“…such as aiXcoder (Jiang et al. 2025b ; Li et al. 2025), StarCoder (Li et al. 2023), DeepSeek (Guo et al. 2024), Qwen (Hui et al. 2024), and Seed (Seed et al. 2025) series – on Exact Match (EM) and Edit Similarity (ES) for text-based FIM benchmarks, and on Pass@1 for Santa - coder-fim-task, a…”

“…such as aiXcoder (Jiang et al. 2025b ; Li et al. 2025), StarCoder (Li et al. 2023), DeepSeek (Guo et al. 2024), Qwen (Hui et al. 2024), and Seed (Seed et al. 2025) series – on Exact Match (EM) and Edit Similarity (ES) for text-based FIM benchmarks, and on Pass@1 for Santa - coder-fim-task, a…”

“For example, models improve in coding capabilities when they have been trained with a large amount of coding data (Aryabumi et al., 2025; Seed et al., 2025; Wei et al., 2024).”

“The main jailbreak tables concern DIJA and PAD.”

“Our autoregressive baseline is the production edit-prediction model used before the diffusion cutover, trained from Seed-Coder [8].”

“Accordingly, constructing and curating code corpora has become a central research focus (Li et al., 2023; Huang et al., 2025; Seed et al., 2025).”

“5-Coder-7B as the victim model and Seed-Coder-8B as the proxy.”

“• SeedCoder 8B [25] adopts a model-centric data pipeline that leverages LLMs to curate code pretraining data, minimizing reliance on hand-crafted filtering rules.”

“We further evaluate several mainstream and recent strong LLMs under the same Text-to-Code-to-TS prompting setup, including IQuest-Coder-V1-7B-Instruct [Yang et al., 2026], Devstral-Instruct-24B [Rastogi et al., 2025], Seed-Coder-8B-Instruct [Seed et al., 2025], and Qwen3.”

“5-Coder-7B [25], OpenCoder-8B [26], and Seed-Coder-8B [27], which are open-source models optimized for code generation across multiple programming languages; (3) Verilog-specialized models : HaVen [28], VeriPrefer [29], and CodeV-R1 [30], which are fine-tuned specifically on Verilog corpora to…”

“We replicate the knowledge-component study on five additional backbones (R1-Distill-Qwen-7B (Guo et al., 2025), Seed-Coder-5 8B (Seed et al., 2025), OpenCoder-8B (Huang et al., 2025), Qwen2.”

“We first use Seed-Coder [18] to sample 10 candidate implementations for each specification from VeriCoder’s dataset [19].”

“5-9B [74], OLMo-3-7B [71], Ministral-3-14B [43], and Seed-Coder-8B [7].”

“For CanItEdit, we report Pass@1 for both lazy and descriptive NL descriptions [48, 49].”

“5-Coder (7B) [5], and Seed-Coder (8B) [9].”

“…have primarily focused on two complementary directions: (1) enhancing the utility of real-world code data through improvements in data quality [Seed et al., 2025], and (2) adopting synthesis strategies to generate high-quality synthetic code data [Qin et al., 2025] [Wang et al., 2023] [Ge et…”