CoChat
Sign Up
PreprintCited by 6Evolutionary Algorithms and ApplicationsAI-based Problem Solving and PlanningArtificial Intelligence in Games

Automated Discovery Has No Universally Superior Harness

Open access
Gupta, Akshat, Lei, Jermaine, Lu, Alexander, Anumanchipalli, Gopala, Choshen, Leshem·2026·arXiv
DOI 10.48550/arxiv.2607.18235arXiv 2607.18235
Open this paper in CoChat — free accountOpen PDF

Abstract

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper decomposes OpenEvolve-style and TTT-Discover-style discovery harnesses into components and evaluates 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts; no fixed harness is reliably superior across settings. Early partial-run performance predicts final performance, and an adaptive online allocation policy improves average final score over single-harness commitment and an unpruned harness ensemble under the same budget.

Key findings

  • No tested fixed harness is significantly better than a simple baseline after correction. The strongest observed configuration was an epsilon-greedy harness with K=1 and ε=20%, with cross-pair majority-win probability P_maj=0.914 and raw p_cross=0.023, but after Holm correction p=0.678; the full OpenEvolve-style configurations had the lowest observed majority-win score, P_maj=0.033 and p_cross=0.978.
  • Adding harness components does not produce monotonic gains. Larger Top-K archives alone produced no pair-level p<0.05; full OpenEvolve-style configurations produced nominally significant gains only for GPT-OSS-20B on Heilbronn Triangle while often reducing performance on Circle Packing and the Second Autocorrelation Inequality; TTT-Discover-style gains were largest after UCT/PUCT parent selection, with later depth and multi-parent changes not consistently improving performance.
  • Early discovery progress predicts final performance. At the 10% checkpoint, Spearman correlations ranged from 0.000 to 0.474; at the 50% checkpoint, correlation exceeded 0.70 for 11 of 12 model-problem pairs and was 0.651 for GPT-OSS-120B on the Second Autocorrelation Inequality.
  • Adaptive online allocation outperformed non-adaptive baselines. The strongest policy, a three-stage 12→5→2→1 schedule pruning at 25%, 50%, and 75%, achieved an average score of 85.75%, compared with 84.54% for the unpruned harness portfolio, 82.49% for single-harness commitment, and 84.35% for five Sequential BoN runs; it outperformed the unpruned portfolio on 11 of 12 model-problem pairs.

Methods

  • The authors decompose OpenEvolve-style evolutionary search and TTT-Discover-style PUCT search into components starting from a greedy Sequential Best-of-N baseline, then evaluate 30 budget-matched harnesses across 12 model-problem pairs: four models (Qwen2.5-3B-Instruct, Qwen3-4B-Instruct-2507, GPT-OSS-20B, GPT-OSS-120B) and three mathematical discovery tasks (circle packing, Heilbronn triangle, second autocorrelation inequality), using more than 3.1 million LLM rollouts.
  • Rollout budgets are matched within each model-problem pair: Qwen models use 1600 rollouts per run (100 iterations × 16 children), GPT-OSS-20B uses 320 (40 × 8), and GPT-OSS-120B uses 160 (40 × 4). Each harness is evaluated with five independent runs; Sequential BoN reference pools use 100 runs for Qwen pairs and 30 runs for GPT-OSS pairs.
  • Statistical analysis uses bootstrap best-of-five tests against Sequential BoN, cross-pair majority-win probabilities with permutation tests and Holm correction, and 100,000 Monte Carlo trials. The adaptive allocation experiment simulates policies over empirical run pools under a five-full-run-equivalent budget, comparing single-harness commitment, an unpruned harness ensemble, and adaptive pruning schedules over 100,000 resampling trials.

Limitations

  • The paper states that pair-level p-values are unadjusted and interpreted as nominal, exploratory evidence; cross-pair significance uses Holm correction. It also notes that the five-run choice is larger than the common three-run protocol used in prior work but still limited. GPT-OSS-120B uses a smaller rollout budget (160) to keep inference cost tractable, and budgets are matched within each model-problem pair, not across different models. GPT-OSS-20B has incomplete coverage in the full epsilon-greedy grid because of compute constraints. The TTT-Discover ablation covers only the search harness, not the test-time-training component.
  • Evident from the text: the evaluation covers only three mathematical tasks and four models (12 model-problem pairs), so transfer beyond these settings is untested; the harness pool is limited to OpenEvolve- and TTT-Discover-style variants; the adaptive allocation policy is evaluated by resampling existing empirical run pools rather than by running new live discovery experiments; and the paper proposes no new bandit or early-stopping algorithm, using allocation instead as a causal test of its findings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.18235

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 6 cited sources checked · 2 supported · 0 flagged

  • Findingown data

    Harness choices can produce significant gains for individual model-problem pairs while hurting others.

    “harness choices can produce significant gains for individual modelproblem pairs while hurting others.”
    Quote in the paperSubstantial claim
  • Findingown data

    Lightweight exploration methods and simpler tree-search algorithms have stronger observed cross-pair performance than complete OpenEvolve-style recipes.

    “Lightweight exploration methods and simpler tree-search algorithms have stronger observed cross-pair performance than the complete OpenEvolve-style recipes”
    Quote in the paperSubstantial claim
  • Findingown data

    Full OpenEvolve-style configurations often reduce performance on Circle Packing and the Second Autocorrelation Inequality.

    “while often reducing performance on Circle Packing and the Second Autocorrelation Inequality.”
    Quote in the paperSubstantial claim
  • Findingown data

    Additional OpenEvolve components do not produce monotonic gains.

    “Additional components do not produce monotonic gains”
    Quote in the paperSubstantial claim
  • Findingown data

    Full-history exploration produces nominally significant gains for select model-problem pairs, including Qwen2.5-3B on Circle Packing and both GPT-OSS models on Heilbronn Triangle.

    “Full-history exploration produces nominally significant gains for select model-problem pairs, including Qwen2.5-3B on Circle Packing and both GPT-OSS models on Heilbronn Triangle”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 24
  • May combine two claims. 36
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The adaptive allocation approach is related to algorithm portfolios, Successive Halving, Hyperband, and asynchronous early-stopping methods such as ASHA.

    “This approach is related to algorithm portfolios (Gomes & Selman, 2001), Successive Halving (Jamieson & Talwalkar, 2016), Hyperband (Li et al., 2018), and asynchronous early-stopping methods such as ASHA (Li et al., 2020)”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Gomes & Selman, 2001 source
    • cites Jamieson & Talwalkar, 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Li et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Our five-run choice is larger than the common three-run protocol used in prior work.

    “Our five-run choice is larger than the common three-run protocol used in prior work(Cemri et al., 2026;Wang et al., 2025; Liu et al., 2026b;Assumpção et al., 2025)”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Cemri et al., 2026 source
    • cites Wang et al., 2025 source
    • cites Liu et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Every adaptive schedule exceeds Sequential BoN on average, with the strongest policy improving the average score from 84.35% to 85.75%.

    “Nevertheless, every adaptive schedule exceeds it on average, with the strongest policy improving the average score from 84.35% to 85.75%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Every evaluated adaptive pruning schedule exceeds both baselines, with average scores ranging from 84.87% to 85.75%.

    “Every evaluated adaptive pruning schedule exceeds both baselines, with average scores ranging from 84.87% to 85.75%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After Holm's correction, the p-value for the epsilon-greedy-style harness rises to 0.678 and is no longer significant.

    “However after Holm's correction the p-value rises to pholm cross = 0.678 and is no longer significant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Full-history exploration scores lower for other pairs and is not universally significant.

    “but scores lower for other pairs and is not universally significant”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Lightweight exploration can help in selected settings, but its benefits do not transfer uniformly across model-problem pairs.

    “lightweight exploration can help in selected settings, but its benefits do not transfer uniformly across model-problem pairs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Subsequent changes to search depth and multi-parent expansion do not improve performance consistently and sometimes reverse those gains.

    “Subsequent changes to search depth and multi-parent expansion do not improve performance consistently and sometimes reverse those gains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The three-stage 12→5→2→1 schedule achieves the highest average score, 85.75%, compared with 84.54% for the unpruned harness portfolio, 82.49% for single-harness commitment, and 84.35% for five Sequential BoN runs.

    “It achieves the highest average score, 85.75%, compared with 84.54% for the unpruned harness portfolio, 82.49% for single-harness commitment, and 84.35% for five Sequential BoN runs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At the 50% checkpoint, the correlation with final performance exceeds 0.70 for 11 of the 12 model-problem pairs, and remains positive at 0.651 for GPT-OSS-120B on the Second Autocorrelation Inequality.

    “At the 50% checkpoint, the correlation with final performance exceeds 0.70 for 11 of the 12 model-problem pairs, and remains positive at 0.651 for GPT-OSS-120B on the Second Autocorrelation Inequality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    No harness improves performance for all problems given a model or all models on a specific problem.

    “Overall, no harness improves performance for all problems given a model or all models on a specific problem.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The PUCT and UCT ablations reinforce the main pattern from the simpler archive-based experiments.

    “Overall, the PUCT and UCT ablations reinforce the main pattern from the simpler archive-based experiments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingreasoning

    GPT-OSS-20B benefits from or at least remains robust under more structured search, while weaker models show greater sensitivity and less reliable improvement.

    “GPT-OSS-20B benefits from or at least remains robust under more structured search, while weaker models show greater sensitivity and less reliable improvement.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.
    • Rests on an assumption: Performance on the three tasks reflects the models' general proposal quality for discovery.

  • May combine two claims.

    Findingown data

    Tree search improves selected pairs, especially Heilbronn Triangle, but later TTT-Discover-style additions do not consistently help.

    “Tree search improves selected pairs, especially Heilbronn Triangle, but later TTT-Discover-style additions do not consistently help.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Qwen3-4B reaches high-scoring regions but has a broader distribution.

    “Qwen3-4B reaches high-scoring regions but has a broader distribution”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Intermediate breadth-depth allocations can be competitive, while the OpenEvolve endpoint does not uniformly dominate across models and tasks.

    “Intermediate breadth-depth allocations can be competitive, while the OpenEvolve endpoint does not uniformly dominate across models and tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    On the second autocorrelation inequality, the three models are much closer in score.

    “On the second autocorrelation inequality, the three models are much closer in score”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The second autocorrelation inequality provides the clearest negative example.

    “The second autocorrelation inequality provides the clearest negative example.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    On Heilbronn Triangle, the separation between models is especially large: GPT-OSS-20B remains high, whereas Qwen3-4B and Qwen2.5-3B show much lower distributions.

    “On Heilbronn Triangle, the separation between models is especially large: GPT-OSS-20B remains high, whereas Qwen3-4B and Qwen2.5-3Bshow much lower distributions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    On Heilbronn triangle, GPT-OSS-20B remains the most reliable, but the OpenEvolve-style settings do not consistently improve the Qwen models.

    “On Heilbronn triangle, GPT-OSS-20B again remains the most reliable, but the OpenEvolve-style settings do not consistently improve the Qwen models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    The second autocorrelation inequality lacks a consistent breadth-depth preference across models, with most settings remaining close to their corresponding baselines.

    “The second autocorrelation inequality also lacks a consistent breadth-depth preference across models, with most settings remaining close to their corresponding baselines.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Deeper OpenEvolve-style refinement is competitive but does not uniformly dominate simpler search variants.

    “The main trend is that deeper OpenEvolve-style refinement is competitive but does not uniformly dominate simpler search variants.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    On Circle Packing, GPT-OSS-20B remains near ceiling for both P = 2 and P = 4, while Qwen3-4B remains competitive but variable.

    “On Circle Packing, GPT-OSS-20B remains near ceiling for both P = 2 and P = 4, while Qwen3-4B remains competitive but variable.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Fixed harnesses-including simple baselines and the ingredients of complex recipes-can yield statistically significant gains on some model-problem pairs while degrading others, so no fixed choice can be safely committed to in advance.

    “our controlled decompositions show that fixed harnesses-including simple baselines and the ingredients of complex recipes-can yield statistically significant gains on some model-problem pairs while degrading others, so no fixed choice can be safely committed to in advance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Larger archives alone yield no significant gains, while exploration helps some model-problem pairs but hurts others.

    “Larger archives alone yield no significant gains, while exploration helps some model-problem pairs but hurts others.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    On circle packing, GPT-OSS-20B remains strong across the progression, while Qwen3-4B achieves competitive scores in several intermediate settings.

    “On circle packing, GPT-OSS-20B remains strong across the progression, while Qwen3-4B achieves competitive scores in several intermediate settings.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    GPT-OSS-20B remains strong across the tested values on circle packing, while the Qwen models are more sensitive.

    “GPT-OSS-20B remains strong across the tested values on circle packing, while the Qwen models are more sensitive.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Qwen2.5-3B shows weaker and less stable results on Circle Packing.

    “Qwen2.5-3Bagain shows weaker and less stable results.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    On circle packing, GPT-OSS-20B is tightly concentrated near the top of the score range, indicating both strong performance and low variance.

    “On circle packing, GPT-OSS-20B is tightly concentrated near the top of the score range, indicating both strong performance and low variance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Moving from Sequential BoN to lightweight archive exploration or UCT/PUCT-style search can improve aggregate performance across our suite, but adding the full OpenEvolve-style or TTT-Discover style machinery does not consistently help and can reverse gains obtained by simpler intermediate variants.

    “Moving from Sequential BoN to lightweight archive exploration or UCT/PUCT-style search can improve aggregate performance across our suite, but adding the full OpenEvolve-style or TTT-Discover style machinery does not consistently help and can reverse gains obtained by simpler intermediate variants.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Added complexity does not produce monotonic gains, and full OpenEvolve-style variants often underperform simpler intermediates.

    “Added complexity does not produce monotonic gains, and full OpenEvolve-style variants often underperform simpler intermediates.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    GPT-OSS-20B is the first model that consistently treats circle packing as an optimization problem rather than only a construction problem.

    “GPT-OSS-20B is the first model that consistently treats circle packing as an optimization problem rather than only a construction problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-OSS-20B remains strong across the sweep, but the Qwen models do not improve reliably as N changes.

    “GPT-OSS-20B remains strong across the sweep, but the Qwen models do not improve reliably as N changes.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Fixed harnesses do not reliably generalize, but early progress is informative.

    “Fixed harnesses do not reliably generalize, but early progress is informative.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Findingown data

    Intermediate breadth-depth settings remain near the baseline, while the one-island and four-island endpoints reduce performance for both Qwen models.

    “Intermediate breadthdepth settings remain near the baseline, while the one-island and four-island endpoints reduce performance for both Qwen models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The null distribution of the five-run maximum is approximated using bootstrap resampling from the baseline runs.

    “we approximate the null distribution of the five-run maximum using bootstrap resampling from the n 0 baseline runs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The Sequential BoN reference pool uses 100 runs for each Qwen model-problem pair and 30 runs for each GPT-OSS model-problem pair.

    “To construct the Sequential BoN reference pool, we use 100 runs for each Qwen model-problem pairs and 30 runs for each GPT-OSS model-problem pairs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study uses a prespecified significance threshold of 0.05.

    “We use a prespecified significance threshold of 0.05”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each harness is evaluated with five independent runs.

    “Each harness is evaluated with five independent runs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study uses R=100,000 Monte Carlo trials for both bootstrap and permutation procedures.

    “We use R = 100, 000 Monte Carlo trials for both the bootstrap and permutation procedures.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The policy-level evaluation repeats the process for 100,000 resampling trials.

    “We repeat this process for 100,000 resampling trials”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Pair-level p-values are unadjusted and interpreted as nominal, exploratory evidence.

    “These pair-level p-values are unadjusted and are interpreted as nominal, exploratory evidence.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All comparisons are budget matched by rollout counts.

    “All comparisons are budget matched by rollout counts”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The consistent improvement of adaptive schedules over the unpruned ensemble supports the conclusion that partial-run scores are useful for deciding where to spend remaining compute, rather than gains arising only from trying multiple harnesses.

    “Because the unpruned ensemble draws from the same harness pool but does not use intermediate feedback, this consistent improvement supports the conclusion that partial-run scores are useful for deciding where to spend the remaining compute, rather than the gains arising only from trying multiple harnesses.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationown data

    Early evolution progress is strongly correlated with final performance.

    “This shows that early evolution progress is strongly correlated with final performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Discovery systems should adapt not only their candidate solutions, but also the harness used to generate them.

    “This points toward discovery systems that adapt not only their candidate solutions, but also the harness used to generate them.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Holm's step-down correction is applied to raw cross-pair p-values across all evaluated harness configurations.

    “We apply Holm's step-down correction to the raw cross-pair p-values across all evaluated harness configurations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Those runs are aligned by P rather than by an identical batch size.

    “Those runs are therefore aligned by P rather than by an identical batch size.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The Sequential BoN reference row serves as a baseline-vs-baseline check: its nearly symmetric win rates and non-significant p-value indicate no evidence that either sampled side systematically wins.

    “The Sequential BoN reference row therefore serves as a baseline-vs-baseline check: its nearly symmetric win rates and non-significant p-value indicate no evidence that either sampled side systematically wins.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Sequential BoN is a fair and meaningful baseline for comparing search harnesses.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    When fixed harnesses fail to transfer and early evaluator feedback is informative, harness selection is better treated as an online resource-allocation problem than as a one-time design decision.

    “when fixed harnesses fail to transfer and early evaluator feedback is informative, harness selection is better treated as an online resource-allocation problem than as a one-time design decision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The absence of a star indicates insufficient evidence under this test, not evidence of no effect.

    “The absence of a star indicates insufficient evidence under this test, not evidence of no effect.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Every null distribution uses 100,000 bootstrap samples of five runs.

    “Every null distribution uses 100,000 bootstrap samples of five runs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The bootstrap procedure with 100,000 samples of five runs provides a valid null distribution for the one-sided tests.

  • May combine two claims.

    Interpretationown data

    Search structure matters, but its effect is conditional on the model's proposal quality.

    “Search structure matters, but its effect is conditional on the model's proposal quality.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.
    • Rests on an assumption: Performance on the three tasks reflects the models' general proposal quality for discovery.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each non-baseline configuration contains only five runs.

    “Because each non-baseline configuration contains only five runs”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Longer refinement chains are valuable when local improvements compound, but they can also overcommit the search to a narrow trajectory.

    “This suggests that longer refinement chains are valuable when local improvements compound, but they can also overcommit the search to a narrow trajectory.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • May combine two claims.

    Interpretationown data

    Search-budget allocation is conditional on proposal quality: deeper search can amplify a promising lineage, but it can also overcommit to a weak trajectory.

    “Search-budget allocation is therefore conditional on proposal quality: deeper search can amplify a promising lineage, but it can also overcommit to a weak trajectory.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The exact child batch size differs by model in the P sweep because GPT-OSS-20B uses a smaller total budget.

    “The exact child batch size differs by model in the P sweep because GPT-OSS-20B uses a smaller total budget.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We emphasize distributional trends and consistency across model-problem pairs rather than treating an individual setting as a definitive optimum.

    “we emphasize distributional trends and consistency across model-problem pairs rather than treating an individual setting as a definitive optimum.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The figure should be read as a comparison of parallel-parent search structure rather than a strict comparison of identical batch sizes.

    “The figure should therefore be read as a comparison of parallel-parent search structure rather than a strict comparison of identical batch sizes.”
    Quote in the paperNot a substantial claim
59 of 59

  • The paper relies on this without support.

    The final-score distribution is often skewed, heavy-tailed, non-continuous and task-dependent.

    “Since the final-score distribution is often skewed, heavy-tailed, non-continuous and task-dependent, we avoid normality or any parametric assumptions.”

  • The paper relies on this without support.

    Discovery runs are expensive and inherently stochastic.

    “Because discovery runs are expensive and inherently stochastic”

  • The paper relies on this without support.

    Existing composite discovery systems are typically evaluated with only a handful of independent trials, far too few to separate genuine methodological ingredients from run-to-run variance.

    “Due to compute and cost constraints, these composite systems are typically evaluated with only a handful of independent trials-far too few to separate genuine methodological ingredients from run-to-run variance.”

  • The paper relies on this without support.

    Existing discovery harnesses typically combine several design choices into a single composite recipe.

    “Existing discovery harnesses typically combine several design choices-including archive construction, parent selection, exploration, search structure, and budget allocation-into a single composite”

  • The paper relies on this without support.

    The three tasks (circle packing, Heilbronn triangle, second autocorrelation inequality) and the models used are representative of discovery tasks and LLMs for evaluating search harnesses.

    “The tasks include two geometric optimization problems and one non-geometric autocorrelation objective.”

  • The paper relies on this without support.

    Sequential BoN is a fair and meaningful baseline for comparing search harnesses.

    “using Sequential BoN as the baseline”

  • The paper relies on this without support.

    The bootstrap procedure with 100,000 samples of five runs provides a valid null distribution for the one-sided tests.

    “Every null distribution uses 100,000 bootstrap samples of five runs.”

  • The paper relies on this without support.

    Performance on the three tasks reflects the models' general proposal quality for discovery.

    “conditional on the model's proposal quality”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Claimed lowest OpenEvolve majority-win score is not the lowest listed

    Minor

    The text identifies 0.033 as the lowest observed majority-win score, but the full leaderboard lists an OpenEvolve configuration at 0.000.

  • Figure 12 caption gives the wrong baseline pool size for GPT-OSS

    Minor

    The caption calls the baseline distributions 100-run distributions, whereas the paper states that GPT-OSS baseline pools contain 30 runs.

  • GPT-OSS-20B settings described as missing appear in the results

    Minor

    Appendix D.6 says GPT-OSS-20B lacks the 10% exploration and K = 30 settings, but the reported results include a GPT-OSS-20B configuration with both.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 6

  1. Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, volume 2025, pp. 21344-21377, 2025.
  2. Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp. 367-377. PMLR, 2020.
  3. Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Ben- jamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of machine learning and systems, 2:230-246, 2020.
  4. Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of machine learning research, 18 (185):1-52, 2018.
  5. John R Rice. The algorithm selection problem. In Advances in computers, volume 15, pp. 65-118. Elsevier, 1976.
  6. Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristics with reflective evolution. Advances in neural information processing systems, 37:43571-43608, 2024.

Cited by

Sorted by citations

Authors
Gupta, A., Lei, J., Lu, A., Anumanchipalli, G., & Choshen, L.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.18235
arXiv
2607.18235
OpenAlex
W7169874844
Cited by
0
References
0
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
automated discovery, evolutionary search, budget allocation, adaptive allocation, parent selection, exploration strategy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review