The DOI matches its DataCite record; 45 of the 45 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
Open accessAbstract
We introduce ShinkaEvolve: a new open-source framework leveraging large language models (LLMs) to advance scientific discovery with state-of-the-art performance and unprecedented efficiency. Recent advances in scaling inference time compute of LLMs have enabled significant progress in generalized scientific discovery. These approaches rely on evolutionary agentic harnesses that leverage LLMs as mutation operators to generate candidate solutions. However, current code evolution methods suffer from critical limitations: they are sample inefficient, requiring thousands of samples to identify effective solutions, and remain closed-source, hindering broad adoption and extension. ShinkaEvolve addresses these limitations, introducing three key innovations: a parent sampling technique balancing exploration and exploitation, code novelty rejection-sampling for efficient search space exploration, and a bandit-based LLM ensemble selection strategy. We evaluate ShinkaEvolve across diverse tasks, demonstrating consistent improvements in sample efficiency and solution quality. ShinkaEvolve discovers a new state-of-the-art circle packing solution using only 150 samples, designs high-performing agentic harnesses for AIME mathematical reasoning tasks, identifies improvements to ALE-Bench competitive programming solutions, and discovers novel mixture-of-expert load balancing loss functions that illuminate the space of optimization strategies. Our results demonstrate that ShinkaEvolve achieves broad applicability with exceptional sample efficiency. By providing open-source accessibility and cost-efficiency, this work democratizes open-ended discovery across diverse computational problems.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
ShinkaEvolve is an open-source LLM-driven evolutionary program search framework with three innovations—weighted parent sampling, code novelty rejection sampling, and bandit-based LLM ensemble selection—designed for sample efficiency. It reports state-of-the-art or improved results across circle packing, AIME agent scaffolds, ALE-Bench, and mixture-of-experts load-balancing loss discovery, including a new circle-packing solution in 150 evaluations.
Key findings
- Circle packing: ShinkaEvolve discovers a new state-of-the-art solution using only 150 samples/evaluations, outperforming AlphaEvolve; the discovered solution has sum of radii 2.635983099011548 under relaxed verification and 2.6359828390115476 after exact adjustment, with an exact-verification replication score of 2.63597770931127.
- AIME 2024: the evolved agent scaffold uses 7 LLM calls (3 generation + 3 review + 1 synthesis) within the 10-call limit, reaches a Pareto frontier between performance and query budget, generalizes to 2025 AIME problems, and transfers to gpt-4.1-mini, gpt-4.1, and o4-mini.
- ALE-Bench LITE: ShinkaEvolve improves ALE-Agent solutions by approximately 2.3% on average across 10 tasks; on ahc039, the combined solution would rank 2nd on the AtCoder leaderboard (3140 performance) versus ALE-Agent’s 5th (2880).
- MoE load balancing: the discovered loss improves over the global-batch LBL across λ ∈ {0.001, 0.01, 0.1} in a 2.7B-parameter MoE; ablations show weighted parent sampling outperforms random search and hill climbing, bandit-based LLM ensembling outperforms single and fixed ensembles, and embedding-based rejection sampling strongly outperforms no rejection sampling.
Methods
- ShinkaEvolve maintains an archive of island subpopulations, samples parents and inspirations, mutates programs via diff edits, full rewrites, and crossover, rejects near-duplicates using embedding cosine similarity above 0.95 with an optional LLM novelty judge, executes programs for scalar fitness plus public metrics and textual feedback, adapts LLM sampling via UCB1, and periodically appends meta-scratchpad insights to mutation prompts.
- It is evaluated on four domains: circle packing (26 circles, 150 generations), AIME 2024 (30 problems, 75 generations, max 10 LLM queries, gpt-4.1-nano, 3 runs per candidate), ALE-Bench LITE (10 AtCoder heuristic tasks, 50 generations, C++), and MoE LBL discovery (556M-parameter/82M-active small MoE with 64 experts and 8 active, 2B fineweb tokens, 30 iterations; 2.7B-parameter/404M-active large MoE, ~30B tokens, seven benchmarks).
- Ablations on circle packing compare parent selection strategies (weighted sampling vs. random search and hill climbing), LLM ensembling (single LLM vs. fixed ensemble vs. bandit-based ensemble), and novelty rejection sampling (none vs. embedding-based vs. embedding plus LLM-as-a-novelty-judge).
Limitations
- Stated in the paper: the implementation uses fixed configurations with limited automatic control over exploration-exploitation balance; task specification requires manual human expertise for objective functions and evaluation; and the framework is constrained to problems with well-defined numerical objectives.
- Evident in the text: ALE-Bench modifications often stayed algorithmically close to the ALE-Agent initialization, indicating potential overfitting to the initialization solution; the MoE architecture used for evolving and testing the LBL was quite similar and the training budget was limited; API costs from large-scale LLM usage could create economic barriers; and the main circle-packing run used relaxed verification with 1×10⁻⁶ slack before exact post-processing.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
31 Own result · 20 Own interpretation · 10 No basis given · 9 Own method · 5 Partly supported · 3 Supported · 2 Sources not found · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ShinkaEvolve discovers a new stateof-the-art circle packing solution using only 150 samples”
- Also stated in: § 4.1. Circle Packing: Reproducing & Improving AlphaEvolve Results · p. 6 · plain
“ShinkaEvolve outperforms AlphaEvolve's solution within less than 150 program evaluations.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1. Circle Packing: Reproducing & Improving AlphaEvolve Results · p. 6 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on one task, ahc039 , the combination of ShinkaEvolve with ALE-Agent resulted in the second place submission on the AtCoder leaderboard if they had participated.”
- Also stated in: § 4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization · p. 8 · plain
“The base solution by ALE-Agent applies simulated annealing with kd-tree data structure (5th, 2880 performance). ShinkaEvolve further improved the solution (2nd, 3140 performance)”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization · p. 8 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we observe that the proposed changes by ShinkaEvolve remained algorithmically close to the original ALE-Agent's initialization solution.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“designs high-performing agentic harnesses for AIME mathematical reasoning tasks”
- Also stated in: § 4.2. AIME: Evolving Agent Scaffolds for Math Reasoning · p. 7 · plain
“ShinkaEvolve discovers scaffold designs that significantly outperform hand-designed baselines, including simple single-query agents and sophisticated majority-voting approaches.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.2. AIME: Evolving Agent Scaffolds for Math Reasoning · p. 7 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“identifies improvements to ALE-Bench competitive programming solutions”
- Also stated in: § 4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization · p. 8 · plain
“ShinkaEvolve is able to improve the solutions discovered by ALE-Agent by approximately 2.3% across the 10 tasks on average (Figure 7).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization · p. 8 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bandit-based LLM ensembling slightly improves the performance over a fixed uniform ensemble distribution.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the fixed ensemble shows moderate improvements over single LLM usage”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the adaptive bandit strategy achieves the highest performance”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consistent with our previous analysis, ShinkaEvolve's LBL appears to improve from the original LBL across both axes.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the consistent generalization results across training budgets and coefficients λ provide an optimistic outlook for future extensions to much longer training regimes, where even small efficiency gains could scale to significant cost savings.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Listing 5 | ShinkaEvolve Discovered ahc025 Solution.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, we do not observe significant evidence for overfitting to the public test cases.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As described in Section 4 and detailed in Appendix C, ShinkaEvolve discovers a new twist on the global-batch LBL from Equation 3, which was used for seeding evolutionary search.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Embeddingbased rejection sampling with LLM as a novelty judge strongly outperforms no rejection sampling.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The additional LLM-as-a-novelty-judge offers marginal improvements”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we replicated the solution using the exact verification code from AlphaEvolve Figure 11 with a score of 2.63597770931127.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Listing 2 | ShinkaEvolve Discovered Circle Packing Solution.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“weighted sampling maintains steady improvement throughout evolution.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Code embedding-based rejection sampling provides substantial performance gains over no rejection sampling”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Random search demonstrates the poorest convergence”
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Listing 3 | ShinkaEvolve Discovered AIME Agent Scaffold Design.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The adjustment from the relaxed to exact formulation reduces the sum of radii for our discovered solution by a negligible amount, from 2.635983099011548 to 2. 6359828390115476, representing a relative change of less than 10 -6 .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Listing 6 | ShinkaEvolve Discovered Mixture of Experts Load Balancing Loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Listing 4 | ShinkaEvolve Discovered ahc039 Solution.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ShinkaEvolve complements this popular LBL with a new term, specifically targeted toward regularizing the MoE layers with individual under-specialized experts.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average solution score improves by a neglible amount from 1923.5 to 1927.0.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ShinkaEvolve discovers an augmentation of this loss with an additional regularization term to target under-specialized experts.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate ShinkaEvolve's ability to innovate beyond human and LLM-generated solutions with comprehensive experimental validation across four distinct problem domains: mathematical optimization (circle packing), agentic design (AIME tasks), competitive programming (ALE-Bench), and LLM training design (mixture-of-expert load balancing loss).”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ShinkaEvolve addresses these limitations, introducing three key innovations: a parent sampling technique balancing exploration and exploitation, code novelty rejection-sampling for efficient search space exploration, and a banditbased LLM ensemble selection strategy.”
- Also stated in: § 1. Introduction · p. 2 · plain
“ShinkaEvolve addresses these challenges through three key algorithmic innovations that work synergistically to enhance sample efficiency.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
- Also stated in: § 1. Introduction · p. 2 · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates the potential of overfitting to the initialization solution.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“smaller improvements on 2023 problems suggest potential saturation due to training data contamination”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“larger gains on 2025 problems indicate successful generalization to recent, unseen challenges.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While this does not resemble the traditional competitive programming setting, it allows us to assess the generalization ability of the discovered solutions.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“by preventing redundant mutations.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This closes a potential blind spot of the global-batch LBL: the dot product 𝑓 · 𝑃 can look 'balanced' even if few experts are barely touched.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that embedding similarity is already an effective proxy for novelty assessment without requiring additional computational overhead.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This work introduces ShinkaEvolve, an evolutionary framework addressing critical limitations in LLM-driven scientific discovery through improved sample efficiency and open-source accessibility.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: LLM-driven scientific discovery has critical limitations in sample efficiency and open-source accessibility.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“highlighting the importance of leveraging fitness-based parent selection.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The framework is constrained to problems with well-defined numerical objectives, limiting its applicability to diverse evaluation domains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This illustrates an important principle: surrogate relaxed tasks can be effectively used during evolution and subsequently post-processed to discover exact state-of-the-art solutions.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The framework's exceptional sample efficiency reduces computational barriers for resource-constrained environments.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“by dynamically prioritizing more effective models based on their contribution to fitness improvements.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 20
- The quote only partly states this claim. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, due to the non-differentiability of the top𝐾 expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL).”
What the sources say
- Partly supportedcites Shazeer et al. (2017) sourceExperimental — accuracy not yet measured
Section 4 (pp. 5–6) introduces auxiliary losses to counter self-reinforcing expert imbalance and unequal loads. Appendix A (p. 13) says discrete expert counts cannot be used directly in backpropagation, so the load loss uses a smooth estimator. But Section 2.1 (p. 4) says selected gate values have nonzero derivatives and that discontinuities have not been a problem in practice; the paper does not say top-K non-differentiability makes an auxiliary load-balancing loss critical. (LLM, conf=0.88)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We follow the evaluation protocol of ALE-Agent (Imajuku et al., 2025) and use the score calculated on the public test cases as the fitness function.”
What the sources say
- Supportedcites Imajuku et al., 2025 source
Appendix B.2 (p. 28) describes ALE-Agent selecting candidates using a priority score calculated from their performance on a fixed suite of 50 test cases. That score combines acceptance ratio and problem-specific score; it is not necessarily the raw contest score alone. The paper separately uses private test cases for final evaluation (Appendix A.2.2, p. 26). (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The ALE-Bench benchmark (Imajuku et al., 2025) is a collection of heuristic programming problems previously used in competitive programming contests (AtCoder).”
What the sources say
- Supportedcites Imajuku et al., 2025 source
The Introduction (p. 2) says ALE-Bench collects past tasks from the AtCoder Heuristic Contest. Figure 1 (p. 1) likewise describes it as collecting past contest tasks, and Section 3.1 (p. 3) identifies AHC as an AtCoder score-based algorithmic competition. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Liu et al., 2024b; Qiu et al., 2025; Team, 2024).”
What the sources say
- Partly supportedcites Dai et al., 2024 sourceExperimental — accuracy not yet measured
Sections 3.1 and 3.2 describe how fine-grained expert segmentation and shared expert isolation can improve specialization and parameter efficiency; Section 4.4 reports improved overall performance in ablations. The paper does not characterize these architectural changes as minor or show a significant change in computational efficiency—the comparisons hold computational cost or activated parameters constant. (LLM, conf=0.82)
- Partly supportedcites Jiang et al., 2024 sourceExperimental — accuracy not yet measured
The Size and Efficiency section reports that Mixtral uses fewer active parameters per token than Llama 2 70B while outperforming it across most categories, supporting an efficiency difference between architectures. But the paper does not test minor design variations, and its Routing analysis (§5; page unavailable) finds no obvious topic-based expert specialization. It therefore does not show that minor variations significantly affect both efficiency and specialization ability. (LLM, conf=0.82)
- Not matchedcites Liu et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“One of the most widely adopted designs is the 'global-batch' LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3 (Yang et al., 2025).”
What the sources say
- Partly supportedcites Shazeer et al. (2017) sourceExperimental — accuracy not yet measured
Section 4 (pp. 5–6) introduces batchwise expert-balancing losses, and Section 3.1 (p. 4) describes combining batches across devices for the MoE layer. This supports identifying Shazeer et al. with a batch-based load-balancing design, but the paper does not establish that it is among the most widely adopted designs. The Qwen 3 example has a separate citation and is not attributed to this source here. (LLM, conf=0.82)
- Partly supportedcites Yang et al. (2025) sourceExperimental — accuracy not yet measured
Section 2 (Architecture; page not provided) says the Qwen3 MoE models adopt a global-batch load balancing loss. It does not establish that the design is widely adopted or used by several other models, and it cites Qiu et al. (2025), not Shazeer et al. (2017), for that loss. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Devising an effective load balancing loss that simultaneously encourages efficiency and expert specialization, without hindering expressivity, remains an open design challenge that has driven much of the recent progress in MoEs (Dai et al., 2024; Du et al., 2022; Fedus et al., 2022; Muennighoff et al., 2024; Qiu et al., 2025; Shazeer et al., 2017; Xue et al., 2024; Zoph et al., 2022).”
What the sources say
- Partly supportedcites Dai et al., 2024 sourceExperimental — accuracy not yet measured
Section 3.3, “Load Balance Consideration” (page not provided), says load imbalance can cause routing collapse and computation bottlenecks, while excessive balance constraints compromise model performance. It therefore supports a tradeoff in designing load-balancing losses. The paper does not say that such a loss itself encourages expert specialization or that this specific design challenge has driven much of the recent progress in MoEs; it pursues specialization primarily through architectural strategies. (LLM, conf=0.85)
- Not matchedcites Du et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fedus et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The LBL prevents collapse toward uneven token distributions and under-specialized experts.”
What the sources say
- Partly supportedcites Shazeer et al. (2017) sourceExperimental — accuracy not yet measured
Section 4 (pp. 5–6) describes a self-reinforcing imbalance in expert selection and says the load-balancing loss encourages experts to receive similar numbers of examples. It also says experts specialize, but does not establish that the load-balancing loss prevents under-specialization; its balancing constraint is soft rather than a guarantee against collapse. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“To ensure the robustness of our results, we additionally validated our solutions using AlphaEvolve 's (Novikov et al., 2025) exact verification code.”
What the sources say
- Supportedcites Novikov et al., 2025 source
Appendix B, p. 38, states that verification code for its reported mathematical constructions is available in the accompanying Google Colab. This supports the citation’s identification of AlphaEvolve’s verification code; the source does not independently document the citing authors’ validation step. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cross-LLM model transfer experiments validate robustness, with successful adaptation to gpt-4.1-mini , gpt-4.1 , and o4-mini demonstrating that discovered architectures capture generalizable strategies rather than model-specific optimizations (Figure 6, right).”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ShinkaEvolve was evaluated over 150 evolutionary generations, demonstrating remarkable sample efficiency compared to existing approaches that typically require thousands of evaluations (Figure 1).”
boostedcorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“discovers novel mixture-of-expert load balancing loss functions that illuminate the space of optimization strategies.”
- Also stated in: § 4.4. LLM Training: Evolving Losses for Balanced and Effective Experts · p. 9 · boosted
“showing that ShinkaEvolve's new loss achieves a consistent edge across our training configurations, growing larger with the value of the 𝜆 coefficient.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 4.4. LLM Training: Evolving Losses for Balanced and Effective Experts · p. 9 · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The search reveals a Pareto frontier between efficiency and performance (Figure 6, left), with 7 LLM queries yielding maximum performance while an alternative scaffold achieves comparable results using the full 10query budget.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evolved algorithm (Section C.1) combines three key innovations: (1) a sophisticated initialization strategy that places circles in a structured goldenangle spiral pattern with strategic corner and edge positioning, (2) a hybrid optimization approach integrating SLSQP gradient-based refinement with simulated annealing for global exploration, and (3) intelligent perturbation mechanisms that”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating on 2023 and 2025 AIME problems shows different transfer patterns (Figure 6, middle): smaller improvements on 2023 problems suggest potential saturation due to training data contamination, while larger gains on 2025 problems indicate successful generalization to recent, unseen challenges.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hill climbing shows strong initial performance but plateaus quickly”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that our discovered solution can be made trivially exact by reducing each circle's radius by 1 × 10 -8 , demonstrating the high precision of our approach.”
boosteddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Weighted sampling consistently outperforms both random search and hill climbing across all tasks.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The discovery of the solution requires more samples to be evaluated.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The bandit-based LLM ensemble significantly outperforms both single LLM and fixed ensemble approaches.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ShinkaEvolve achieves state-of-the-art results across four domains: circle packing with 150 evaluations (orders of magnitude improvement), sophisticated AIME reasoning scaffolds, ALE-Bench algorithmic improvements, and novel mixture-of-expert load balancing.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Weighted parent sampling outperforms random search and hill climbing.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluate ShinkaEvolve across diverse tasks, demonstrating consistent improvements in sample efficiency and solution quality.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results demonstrate that ShinkaEvolve achieves broad applicability with exceptional sample efficiency.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
- Rests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
In the paper
“For the main ShinkaEvolve run presented in the paper, we employed the verification script provided by OpenEvolve (Sharma, 2025), which allows for 1 × 10 -6 numerical slack.”
What the sources say
- Not matchedcites Sharma, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“After evolution, we evaluate the discovered scaffold designs on 2023 and 2025 AIME problems (AIM, 2023, 2025) to assess generalization as well as robustness to different base agent language models.”
What the sources say
- Not matchedcites AIM, 2023, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This grounding provides ShinkaEvolve a two-fold search objective: minimize CE while improving load balance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, ShinkaEvolve's new term can be seen as a safety net that adaptively activates and vanishes once an expert crosses the floor, providing dead experts and avoiding over-regularizing well-balanced layers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To avoid local noise affecting the cross-entropy calculations, we average it over the last 10M tokens.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally and to combat stochasticity in LLM queries, we evaluated each candidate evaluated across three independent runs on the complete question set.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Task specification requires manual human expertise for objective functions and evaluation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the expense of pretraining, we run ShinkaEvolve for only 30 iterations, focusing on gpt-4.1 , gemini-2.5-pro , and claude-sonnet-4 .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our implementation uses fixed configurations with limited automatic control over exploration-exploitation balance, which may vary across domains.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ShinkaEvolve's open-source release further democratizes advanced evolutionary optimization, making it accessible to researchers and practitioners previously lacking access to proprietary systems.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This formulation encourages token usage across experts to align with the router's average soft assignment probabilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, API costs from large-scale LLM usage could create economic barriers, potentially constraining democratization goals.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also note that the architecture used for evolving and testing the employed LBL was quite similar, and the training budget was still limited.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we also note that the architecture used for evolving and testing the employed LBL was quite similar, and the training budget was still limited.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Intuitively , this term softly affects the MoE router of any layer, with experts getting allocated a fraction of tokens less than 𝜏 . The multiplier 𝑠 ( 𝑃ℓ ) makes this push stronger when the layer's routing entropy 𝐻 ( 𝑃ℓ ) is low and the router is concentrating on fewer dominating experts.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.
“This inefficiency stems from naive exploration strategies that fail to effectively leverage accumulated knowledge from previous generations.”
The paper relies on this without support.
Most leading code evolution systems remain closed-source, creating barriers to reproducibility and limiting community-driven improvements.
“Additionally, most leading systems remain closed-source, creating barriers to reproducibility and limiting community-driven improvements.”
The paper relies on this without support.
The performance of different LLMs to propose mutations varies across problem domains and based on the current state of the sampled archive parents and inspiration programs.
“The performance of different LLMs to propose mutations can vary across problem domains and based on the current state of the sampled archive parents and inspiration programs.”
The paper relies on this without support.
The program archive is non-stationary.
“to account for the non-stationarity of the program archive”
The paper relies on this without support.
Similar code variants can waste computational resources without advancing the search frontier.
“Similar code variants can waste computational resources without advancing the search frontier.”
The paper relies on this without support.
LLM-driven scientific discovery has critical limitations in sample efficiency and open-source accessibility.
“addressing critical limitations in LLM-driven scientific discovery through improved sample efficiency and open-source accessibility.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ B.4. Mixture-of-Experts Load Balancing Loss · p. 26
“𝜏 = 0 . 064 / 𝑁𝐸 as a minimum usage threshold”
§ C.4. Mixture-of-Experts Load Balancing Loss · p. 52
“min_threshold = 0.01 * (64.0 / num_experts)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4. LLM Training: Evolving Losses for Balanced and Effective Experts · p. 9
“in terms of final perplexity (Figure 8, left) and downstream task performance (Figure 8, center)”
§ 4.4. LLM Training: Evolving Losses for Balanced and Effective Experts · p. 9
“Figure 8 | ShinkaEvolve for discovering Mixture-of-Experts Load Balancing Loss Functions. Left: Downstream task performance across seven benchmarks. Middle: Final perplexity across different missroute fractions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ C.4. Mixture-of-Experts Load Balancing Loss · p. 52
“attention_mask: Optional[torch.Tensor] = None”
§ C.4. Mixture-of-Experts Load Balancing Loss · p. 52
“bsz, seqlen = attention_mask.shape n_tokens = bsz * seqlen”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization · p. 8
“ultimately replacing simulated annealing with a more focused optimization combining greedy moves and targeted local search.”
§ C.3.2. ALE-Bench LITE task: ahc025 · p. 49
“// Targeted Simulated Annealing if (D > 1) {”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning , pages 5547-5569. PMLR, 2022.
- Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE , pages 329-336, 2008.
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.
- Chris Lu, Samuel Holt, Claudio Fanconi, Alex James Chan, Jakob Nicolaus Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024a. URL https://openreview.net/forum?id=erjQDJ0z9L .
Cited by
Sorted by citations
ThetaEvolve: Test-time Learning on Open Problems
Wang, Y., Su, S.-R., Zeng, Z., Xu, E., Ren, L., Yang, X., Huang, Z., He, X., Ma, L., Peng, B., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.23473
“We consider two open tasks, circle packing (CP) and the first autocorrelation inequality (FACI), and report the best values mentioned in AlphaEvolve-v2 (Georgiev et al., 2025) and its variant ShinkaEvolve (Lange et al., 2025).”
InfluentialCORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended DiscoveryPreprint
Qu, A., Zheng, H., Zhou, Z., Yan, Y., Tang, Y., Ong, S. Y., Hong, F., Zhou, K., Jiang, C., Kong, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.01658 · Cited by 68
“Subsequent systems refine the search orchestration through adaptive sampling, island-based architectures, and Pareto-based selection (Sharma, 2025; Lange et al., 2025; Khrulkov et al., 2025; Assumpc¸˜ao et al., 2026; Yan et al., 2026; Agrawal et al., 2025), while AdaEvolve (Cemri et al., 2026) and…”
K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World ModelPreprint
Cao, S., Mao, Z., Gonzalez, J. E., & Stoica, I. G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.19128 · Cited by 35
“ShinkaEvolve (Lange et al., 2025a) proposes a population-based evolutionary framework that combines performance-driven selection with novelty-aware rejection to improve sample efficiency.”
DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven EvolutionPreprint
Jiang, J., Ding, T., & Zhu, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.02919 · Cited by 25
“Specifically, ShinkaEvolve (Lange et al., 2025), GigaEvo (Khrulkov et al., 2025), and CodeEvolve (As-sumpção et al., 2025) introduce advanced population management techniques, including novelty-based rejection sampling, MAP-Elites quality-diversity algorithms, and island-based crossover operators.”
SkyDiscover: A Flexible, Adaptive Framework for AI-Driven Scientific and Algorithmic Discovery
Liu, S., Cemri, M., Agarwal, S., et al. (2026) · CAIS · doi:10.1145/3786335.3813221 · Cited by 27
“RecentsystemssuchasAlphaEvolve[9],OpenEvolve[10], GEPA [1], and ShinkaEvolve [6] have demonstrated the promise of this paradigm across domains ranging from mathematical optimization to systems design.”
Metadata
- Authors
- Lange, R. T., Imajuku, Y., & Cetin, E.
- ORCID
- Imajuku, Yuki
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.19349
- OpenAlex
- W7083291402
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- software evolution, large language models, sample efficiency, circle packing, loss function design, scientific discovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-24, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-24 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
