ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to the non-differentiability of the topK expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL).

“However, due to the non-differentiability of the top𝐾 expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL).”

• cites Shazeer et al. (2017) sourcePartly supported

  Section 4 (pp. 5–6) introduces auxiliary losses to counter self-reinforcing expert imbalance and unequal loads. Appendix A (p. 13) says discrete expert counts cannot be used directly in backpropagation, so the load loss uses a smooth estimator. But Section 2.1 (p. 4) says selected gate values have nonzero derivatives and that discontinuities have not been a problem in practice; the paper does not say top-K non-differentiability makes an auxiliary load-balancing loss critical. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

We follow the evaluation protocol of ALE-Agent and use the score calculated on the public test cases as the fitness function.

“We follow the evaluation protocol of ALE-Agent (Imajuku et al., 2025) and use the score calculated on the public test cases as the fitness function.”

• cites Imajuku et al., 2025 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The ALE-Bench benchmark is a collection of heuristic programming problems previously used in competitive programming contests (AtCoder).

“The ALE-Bench benchmark (Imajuku et al., 2025) is a collection of heuristic programming problems previously used in competitive programming contests (AtCoder).”

• cites Imajuku et al., 2025 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Minor design variations have been shown to significantly affect both efficiency and specialization ability.

“Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Liu et al., 2024b; Qiu et al., 2025; Team, 2024).”

• cites Dai et al., 2024 sourcePartly supported

  Sections 3.1 and 3.2 describe how fine-grained expert segmentation and shared expert isolation can improve specialization and parameter efficiency; Section 4.4 reports improved overall performance in ablations. The paper does not characterize these architectural changes as minor or show a significant change in computational efficiency—the comparisons hold computational cost or activated parameters constant. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Jiang et al., 2024 sourcePartly supported

  The Size and Efficiency section reports that Mixtral uses fewer active parameters per token than Llama 2 70B while outperforming it across most categories, supporting an efficiency difference between architectures. But the paper does not test minor design variations, and its Routing analysis (§5; page unavailable) finds no obvious topic-based expert specialization. It therefore does not show that minor variations significantly affect both efficiency and specialization ability. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Liu et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

One of the most widely adopted designs is the 'global-batch' LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3.

“One of the most widely adopted designs is the 'global-batch' LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3 (Yang et al., 2025).”

• cites Shazeer et al. (2017) sourcePartly supported

  Section 4 (pp. 5–6) introduces batchwise expert-balancing losses, and Section 3.1 (p. 4) describes combining batches across devices for the MoE layer. This supports identifying Shazeer et al. with a batch-based load-balancing design, but the paper does not establish that it is among the most widely adopted designs. The Qwen 3 example has a separate citation and is not attributed to this source here. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Yang et al. (2025) sourcePartly supported

  Section 2 (Architecture; page not provided) says the Qwen3 MoE models adopt a global-batch load balancing loss. It does not establish that the design is widely adopted or used by several other models, and it cites Qiu et al. (2025), not Shazeer et al. (2017), for that loss. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Devising an effective load balancing loss that simultaneously encourages efficiency and expert specialization, without hindering expressivity, remains an open design challenge that has driven much of the recent progress in MoEs.

“Devising an effective load balancing loss that simultaneously encourages efficiency and expert specialization, without hindering expressivity, remains an open design challenge that has driven much of the recent progress in MoEs (Dai et al., 2024; Du et al., 2022; Fedus et al., 2022; Muennighoff et al., 2024; Qiu et al., 2025; Shazeer et al., 2017; Xue et al., 2024; Zoph et al., 2022).”

• cites Dai et al., 2024 sourcePartly supported

  Section 3.3, “Load Balance Consideration” (page not provided), says load imbalance can cause routing collapse and computation bottlenecks, while excessive balance constraints compromise model performance. It therefore supports a tradeoff in designing load-balancing losses. The paper does not say that such a loss itself encourages expert specialization or that this specific design challenge has driven much of the recent progress in MoEs; it pursues specialization primarily through architectural strategies. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Du et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fedus et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The LBL prevents collapse toward uneven token distributions and under-specialized experts.

“The LBL prevents collapse toward uneven token distributions and under-specialized experts.”

• cites Shazeer et al. (2017) sourcePartly supported

  Section 4 (pp. 5–6) describes a self-reinforcing imbalance in expert selection and says the load-balancing loss encourages experts to receive similar numbers of examples. It also says experts specialize, but does not establish that the load-balancing loss prevents under-specialization; its balancing constraint is soft rather than a guarantee against collapse. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The solutions were additionally validated using AlphaEvolve's exact verification code.

“To ensure the robustness of our results, we additionally validated our solutions using AlphaEvolve 's (Novikov et al., 2025) exact verification code.”

• cites Novikov et al., 2025 sourceSupported

May combine two claims.

Cross-LLM model transfer experiments validate robustness, with successful adaptation to gpt-4.1-mini, gpt-4.1, and o4-mini demonstrating that discovered architectures capture generalizable strategies rather than model-specific optimizations.

“Cross-LLM model transfer experiments validate robustness, with successful adaptation to gpt-4.1-mini , gpt-4.1 , and o4-mini demonstrating that discovered architectures capture generalizable strategies rather than model-specific optimizations (Figure 6, right).”

May combine two claims.

ShinkaEvolve was evaluated over 150 evolutionary generations on circle packing, demonstrating remarkable sample efficiency compared to existing approaches that typically require thousands of evaluations.

“ShinkaEvolve was evaluated over 150 evolutionary generations, demonstrating remarkable sample efficiency compared to existing approaches that typically require thousands of evaluations (Figure 1).”

• AssumptionRests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.

The quote only partly states this claim.

ShinkaEvolve discovers a novel MoE load balancing loss that achieves a consistent edge over global-batch LBL across training configurations.

“discovers novel mixture-of-expert load balancing loss functions that illuminate the space of optimization strategies.”

• Also stated in: § 4.4. LLM Training: Evolving Losses for Balanced and Effective Experts · p. 9 · boosted

  “showing that ShinkaEvolve's new loss achieves a consistent edge across our training configurations, growing larger with the value of the 𝜆 coefficient.”

May combine two claims.

The search for AIME scaffolds reveals a Pareto frontier between efficiency and performance, with 7 LLM queries yielding maximum performance while an alternative scaffold achieves comparable results using the full 10-query budget.

“The search reveals a Pareto frontier between efficiency and performance (Figure 6, left), with 7 LLM queries yielding maximum performance while an alternative scaffold achieves comparable results using the full 10query budget.”

May combine two claims.

The evolved circle packing algorithm combines structured golden-angle spiral initialization, hybrid SLSQP gradient-based refinement with simulated annealing, and intelligent perturbation mechanisms.

“The evolved algorithm (Section C.1) combines three key innovations: (1) a sophisticated initialization strategy that places circles in a structured goldenangle spiral pattern with strategic corner and edge positioning, (2) a hybrid optimization approach integrating SLSQP gradient-based refinement with simulated annealing for global exploration, and (3) intelligent perturbation mechanisms that”

May combine two claims.

The discovered AIME scaffold shows smaller improvements on 2023 AIME problems and larger gains on 2025 AIME problems.

“Evaluating on 2023 and 2025 AIME problems shows different transfer patterns (Figure 6, middle): smaller improvements on 2023 problems suggest potential saturation due to training data contamination, while larger gains on 2025 problems indicate successful generalization to recent, unseen challenges.”

May combine two claims.

Hill climbing shows strong initial performance but plateaus quickly.

“Hill climbing shows strong initial performance but plateaus quickly”

May combine two claims.

The discovered circle packing solution can be made trivially exact by reducing each circle's radius by 1e-8, demonstrating high precision.

“We found that our discovered solution can be made trivially exact by reducing each circle's radius by 1 × 10 -8 , demonstrating the high precision of our approach.”

May combine two claims.

Weighted sampling consistently outperforms both random search and hill climbing across all tasks.

“Weighted sampling consistently outperforms both random search and hill climbing across all tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The discovery of the exact solution requires more samples to be evaluated.

“The discovery of the solution requires more samples to be evaluated.”

May combine two claims.

The bandit-based LLM ensemble significantly outperforms both single LLM and fixed ensemble approaches.

“The bandit-based LLM ensemble significantly outperforms both single LLM and fixed ensemble approaches.”

May combine two claims.

ShinkaEvolve achieves state-of-the-art results across four domains: circle packing with 150 evaluations (orders of magnitude improvement), sophisticated AIME reasoning scaffolds, ALE-Bench algorithmic improvements, and novel mixture-of-expert load balancing.

“ShinkaEvolve achieves state-of-the-art results across four domains: circle packing with 150 evaluations (orders of magnitude improvement), sophisticated AIME reasoning scaffolds, ALE-Bench algorithmic improvements, and novel mixture-of-expert load balancing.”

May combine two claims.

Weighted parent sampling outperforms random search and hill climbing on Circle Packing.

“Weighted parent sampling outperforms random search and hill climbing.”

May combine two claims.

ShinkaEvolve achieves consistent improvements in sample efficiency and solution quality across diverse tasks.

“We evaluate ShinkaEvolve across diverse tasks, demonstrating consistent improvements in sample efficiency and solution quality.”

• AssumptionRests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.

May combine two claims.

ShinkaEvolve achieves broad applicability with exceptional sample efficiency.

“Our results demonstrate that ShinkaEvolve achieves broad applicability with exceptional sample efficiency.”

• AssumptionRests on an assumption: Existing code evolution methods are sample inefficient because they use naive exploration strategies that fail to leverage accumulated knowledge from previous generations.

Probably not a substantial claim: may not pass S3 (load-bearing).

The main ShinkaEvolve circle packing run used OpenEvolve's verification script allowing 1e-6 numerical slack.

“For the main ShinkaEvolve run presented in the paper, we employed the verification script provided by OpenEvolve (Sharma, 2025), which allows for 1 × 10 -6 numerical slack.”

• cites Sharma, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

After evolution, discovered scaffold designs were evaluated on 2023 and 2025 AIME problems to assess generalization and robustness to different base agent language models.

“After evolution, we evaluate the discovered scaffold designs on 2023 and 2025 AIME problems (AIM, 2023, 2025) to assess generalization as well as robustness to different base agent language models.”

• cites AIM, 2023, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

This grounding provides ShinkaEvolve a two-fold search objective: minimize CE while improving load balance.

“This grounding provides ShinkaEvolve a two-fold search objective: minimize CE while improving load balance.”

May combine two claims.

The new term can be seen as a safety net that adaptively activates and vanishes once an expert crosses the floor, providing dead experts and avoiding over-regularizing well-balanced layers.

“Thus, ShinkaEvolve's new term can be seen as a safety net that adaptively activates and vanishes once an expert crosses the floor, providing dead experts and avoiding over-regularizing well-balanced layers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To avoid local noise affecting the cross-entropy calculations, we average it over the last 10M tokens.

“To avoid local noise affecting the cross-entropy calculations, we average it over the last 10M tokens.”

Classified as probably not a substantial claim.

For AIME, each candidate was evaluated across three independent runs on the complete question set to combat stochasticity.

“Additionally and to combat stochasticity in LLM queries, we evaluated each candidate evaluated across three independent runs on the complete question set.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Task specification requires manual human expertise for objective functions and evaluation.

“Task specification requires manual human expertise for objective functions and evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Given the expense of pretraining, we run ShinkaEvolve for only 30 iterations.

“Given the expense of pretraining, we run ShinkaEvolve for only 30 iterations, focusing on gpt-4.1 , gemini-2.5-pro , and claude-sonnet-4 .”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ShinkaEvolve's implementation uses fixed configurations with limited automatic control over exploration-exploitation balance, which may vary across domains.

“Our implementation uses fixed configurations with limited automatic control over exploration-exploitation balance, which may vary across domains.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

ShinkaEvolve's open-source release further democratizes advanced evolutionary optimization, making it accessible to researchers and practitioners previously lacking access to proprietary systems.

“ShinkaEvolve's open-source release further democratizes advanced evolutionary optimization, making it accessible to researchers and practitioners previously lacking access to proprietary systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This formulation encourages token usage across experts to align with the router's average soft assignment probabilities.

“This formulation encourages token usage across experts to align with the router's average soft assignment probabilities.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

API costs from large-scale LLM usage could create economic barriers, potentially constraining democratization goals.

“However, API costs from large-scale LLM usage could create economic barriers, potentially constraining democratization goals.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The training budget was still limited.

“However, we also note that the architecture used for evolving and testing the employed LBL was quite similar, and the training budget was still limited.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The architecture used for evolving and testing the employed LBL was quite similar.

“However, we also note that the architecture used for evolving and testing the employed LBL was quite similar, and the training budget was still limited.”

May combine two claims.

The new regularization term softly affects the MoE router, with experts allocated a fraction of tokens less than τ, and the multiplier s(Pl) makes the push stronger when routing entropy is low.

“Intuitively , this term softly affects the MoE router of any layer, with experts getting allocated a fraction of tokens less than 𝜏 . The multiplier 𝑠 ( 𝑃ℓ ) makes this push stronger when the layer's routing entropy 𝐻 ( 𝑃ℓ ) is low and the router is concentrating on fewer dominating experts.”