CoChat
Sign Up
PreprintCited by 14Evolutionary Algorithms and ApplicationsAI-based Problem Solving and PlanningArtificial Intelligence Applications

AlphaResearch: Accelerating New Algorithm Discovery with Language Models

Open access
Yu, Zhaojian, Feng, Kaiyue, Zhao, Yilun, He, Shilin, Zhang, Xiao-Ping, Cohan, Arman·2025·arXiv
DOI 10.48550/arxiv.2511.08522arXiv 2511.08522
Open this paper in CoChat — free accountOpen PDF

Abstract

LLMs have made significant progress in complex but easy-to-verify problems, yet they still struggle with discovering the unknown. In this paper, we present \textbf{AlphaResearch}, an autonomous research agent designed to discover new algorithms on open-ended problems by iteratively running the following steps: (1) propose new ideas (2) program to verify (3) optimize the research proposals. To synergize the feasibility and innovation of the discovery process, we construct a novel dual environment by combining the execution-based verifiable reward and reward from simulated real-world peer review environment in AlphaResearch. We construct \textbf{\dataset}, a set of questions that includes an eight open-ended algorithmic problems competition to benchmark AlphaResearch. Experimental results show that AlphaResearch achieves stronger discovery performance than other agentic discovery systems on six open-ended problems. Notably, the algorithm discovered by AlphaResearch on the \emph{``packing circles''} problem achieves the best-of-known performance, surpassing the results of human researchers and strong baselines from recent work (e.g., AlphaEvolve). Additionally, we conduct a comprehensive analysis of the benefits and remaining challenges of autonomous research agent, providing valuable insights for future research.

AI snapshot

Written from the paper’s full text.

TL;DR

AlphaResearch is an autonomous research agent that combines LLM idea generation with a dual verification environment: a peer-review reward model trained on ICLR records and program-execution-based evaluation. On the new AlphaResearchComp benchmark of 8 open-ended algorithm-discovery problems, it achieves a 2/8 win rate against human researchers, including best-known results on packing circles.

Key findings

  • AlphaResearch achieves a 2/8 win rate against human researchers on AlphaResearchComp, with 6/8 problems still underperforming human bests.
  • On packing circles, it reaches 2.636 for n=26 and 2.939 for n=32, surpassing human researchers (2.634 and 2.936) and AlphaEvolve (2.635 and 2.937).
  • AlphaResearch-RM-7B reaches 72% classification accuracy on unseen ICLR 2025 ideas, outperforming human annotators (65.0%), GPT-5 (53.0%), and Qwen2.5-7B-Instruct (37.0%).
  • In ablations, AlphaResearch-RM-7B filtered 151 ideas below threshold, including 108 correct rejections of execution failures and 43 erroneous rejections of viable ideas, for 71.5% accuracy (108/151).

Methods

  • AlphaResearch iteratively proposes new ideas, verifies them in a dual environment, and optimizes proposals; ideas are filtered by AlphaResearch-RM-7B, then implemented as programs and evaluated by a code executor.
  • The reward model is fine-tuned from Qwen2.5-7B-Instruct on ICLR peer-review records from 2017–2024 (24,445 training records) and tested on 100 ICLR 2025 records.
  • Experiments use o4-mini as the research agent on AlphaResearchComp, an 8-problem benchmark spanning geometry, number theory, harmonic analysis, and combinatorial optimization, with the proposed excel@best metric.

Limitations

  • Stated: the authors hope to expand coverage to more realistic applications such as accelerating tensor computations; experiments use the simplest and most straightforward approaches; the reward model is based on small models and 24,445 ICLR peer-review records, with scaling left for future work.
  • Evident from the text: AlphaResearch still fails on 6/8 benchmark problems and does not improve over human bests on Littlewood polynomials or MSTD(n=30), indicating current LLMs struggle to consistently find better algorithms than human researchers.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2511.08522

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 16 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    AlphaResearch constructs a dual research environment combining execution-based verification and simulated real-world peer review.

    “we construct a novel dual research environment by combining the execution-based verify and simulated real-world peer review environment.”
    • Also stated in: § 1 INTRODUCTION · plain
      “we construct a novel dual research-based environment (Tian et al., 2024), where novel insights are forged by the simulated real-world peer-reviewed environment and execution-based verification.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Tian et al., 2024 source

      Section 2.3.2 (p. 5) supports the execution-based verification component: SciCode checks generated code with numerical and domain-specific test cases. The copy does not describe AlphaResearch, a dual research environment, or simulated peer review; the scientist reviews in Section 2.3.3 (p. 5) are validation of SciCode problems, not a simulated peer-review environment. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingown data

    Approximately 30%∼40% of newly proposed ideas fall below the RM threshold and are thus discarded.

    “approximately 30%∼40% of newly proposed ideas fall below the RM threshold and are thus discarded.”
    Quote in the paperNot a substantial claim
  • Findingown data

    AlphaResearch gets a 2/8 win rate in head-to-head comparison with human researchers.

    “AlphaResearch gets a 2/8 win rate in head-to-head comparison with human researchers, demonstrate the possibility of accelerating algorithm discovery with LLMs.”
    • Also stated in: § 4.1 SETUP · plain
      “AlphaResearch achieved a 2/8 win rate (excel@best > 0) against human researchers”
    • Also stated in: § 4.1 SETUP · plain
      “AlphaResearch's performance still underperforms human researchers in 6 out of 8 problems.”
    Quote in the paperSubstantial claim
  • Findingown data

    AlphaResearch achieves better performance than OpenEvolve and ShinkaEvolve on the Packing Circles (n=26) problem.

    “AlphaResearch achieves better performance than others.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.
  • Findingown data

    The execution-based reward initially grows rapidly, then slowly plateaus for optimal performance seeking.

    “the execution-based reward initially grows rapidly, then slowly plateaus for optimal performance seeking.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 10
  • May combine two claims. 23

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Frontier LLMs like GPT-5 and Gemini 2.5 could achieve expert-level performance in complex tasks such as mathematics and programming.

    “Recent progress has shown that frontier LLMs like GPT-5 (OpenAI, 2025) and Gemini 2.5 (Comanici et al., 2025) could achieve expert-level performance in complex tasks such as mathematics (Trinh et al., 2024;Lin et al., 2025) and programming (Jimenez et al., 2024;Jain et al., 2025).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites OpenAI, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Comanici et al., 2025 source
    • cites Trinh et al., 2024 source

      The Main section reports that AlphaGeometry approaches the performance of an average IMO gold medallist on 30 geometry problems. That supports expert-level performance in a specific area of mathematics, but AlphaGeometry is a neuro-symbolic system, not a frontier LLM such as GPT-5 or Gemini 2.5. The Proving results section reports that GPT-4 alone solved none of the 30 problems; this source does not establish the cited models’ mathematical performance. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisereasoning

    Execution-based verification systems like AlphaEvolve can rigorously validate whether code runs and meets constraints, but this verification alone might not be completely sufficient for discovery.

    “Execution-based verification systems like AlphaEvolve Novikov et al. (2025) can rigorously validate whether code runs and meets constraints, but this verification alone might not be completely sufficient for discovery.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Novikov et al., 2025 source

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 1.546 on third autocorrelation inequality, improving over the human best of 1.458 by -6.03% (lower is better).

    “| third autocorrelation inequality ↓ | 1.458 35.746 | 1.546 | -6.03% | |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 32 on littlewood polynomials (n=512), matching the human best of 32.

    “| littlewood polynomials (n=512) ↓ | 32 | 32 | 32 | 0 |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 12.92 on minimizing max-min distance ratio (d=2, n=16), improving over the human best of 12.89 by -0.23% (lower is better).

    “| minimizing max-min distance ratio ↓ | 12.89 | 15.55 | 12.92 | -0.23% |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    Both GPT-5 and Qwen2.5-7B-Instruct have lower than 50% accuracy when identifying good ideas from ICLR 2025 records.

    “Both GPT-5 and Qwen2.5-7B-Instructhave lower than 50% accuracy when identifying the good ideas from ICLR 2025 records.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 0.756 on autoconvolution peak minimization, improving over the human best of 0.755 by -0.13% (lower is better).

    “| autoconvolution peak minimization ↓ | 0.755 | 1.512 | 0.756 | -0.13% |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch-RM-7B filters 151 bad ideas, of which 108 fail to execute and 43 succeed.

    “AlphaResearch-RM-7B filters 151 bad ideas where 108 ideas fail to executed and 43 successes.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A reward model score below the threshold indicates a bad idea.

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 2.939 on packing circles (n=32), improving over the human best of 2.936 by 0.10%.

    “| packing circles (n=32) ↑ | 2.936 | 0 | 2.939 | 0.10% |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 0.6735 on spherical code (d=3, n=30), which is -0.01% compared to the human best of 0.6736.

    “| spherical code (d=3, n=30) ↑ | 0.6736 0.5130 0.6735 | -0.01% | | |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch's new construction for packing 32 circles in a unit square achieves a sum of radii ≥ 2.9395, improving on AlphaEvolve's ≥ 2.9379.

    “New construction of AlphaResearch (right) improving the best known AlphaEvolve (right) bounds on packing circles to maximize their sum of radii.Left: 32 circles in a unit square with sum of radii ≥ 2.9379.Right: 32 circles in a unit square with sum of radii ≥ 2.9395”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch achieves better performance than OpenEvolve and slightly surpasses ShinkaEvolve on packing circles (n=26) at the first 500 steps.

    “AlphaResearch achieves better performance than OpenEvolve and slightly surpasses ShinkaEvolve, which demonstrates its advantages on accelerating algorithm discovery.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 1.04 on MSTD (n=30), matching the human best of 1.04.

    “| MSTD (n=30) ↑ | 1.04 | 1.04 | 1.04 | 0 |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Findingown data

    AlphaResearch-RM-7B demonstrates 72% classification accuracy on unseen ICLR 2025 ideas, significantly outperforming baseline models and human experts.

    “AlphaResearch-RM-7B demonstrates 72% classification accuracy on unseen ICLR 2025 ideas, significantly outperforming baseline models and human experts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AlphaResearch-RM-7B successfully filtered 151 ideas below threshold, yielding 108 correct rejections of execution failures and 43 erroneous rejections of viable ideas.

    “AlphaResearch-RM-7B successfully filtered 151 ideas below the threshold.This process yielded 108 correct rejections of execution failures while making 43 erroneous rejections of viable ideas.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AlphaResearch achieves a best score of 2.636 on packing circles (n=26), improving over the human best of 2.634 by 0.32%.

    “| packing circles (n=26) ↑ | 2.634 | 0 | 2.636 | 0.32% |”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

  • May combine two claims.

    Premisereasoning

    AlphaResearch discovers new algorithms by iteratively proposing new ideas, verifying them in the dual research environment, and optimizing proposals.

    “AlphaResearch discovers new algorithm by iteratively running the following steps: (1) propose new ideas (2) verify the ideas in the dual research environment (3) optimize the research proposals for better performance.”
    • Also stated in: § 1 INTRODUCTION · plain
      “AlphaResearch discovers new algorithms by iteratively running the following steps: (i) proposing new research ideas, (ii) verify the ideas in the dual researchbased environment, and (iii) optimizing the proposals for higher reward from the environment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The initialization strategy simulates a variety of real-world scenarios for the research agent, facilitating a thorough evaluation process.

    “This initialization strategy simulates a variety of real-world scenarios for the research agent, thereby facilitating a thorough evaluation process.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous studies demonstrate that LLMs can generate novel ideas at a human expert level.

    “Previous studies demonstrate that LLMs can generate novel ideas at a human expert level (Si et al., 2024;Wang et al., 2024a).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Si et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Wang et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    The reward model AlphaResearch-RM-7B is trained on ICLR peer review records from 2017 to 2024, with test set from ICLR 2025, disjoint dates to prevent knowledge contamination.

    “We sample a subset of ICLR 2025 records as a test set, where the dates of train and test are disjoint, which prevents knowledge contamination between the train and test split.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The human best for third autocorrelation inequality is 1.4581, attributed to Carlos Vinuesa (2009).

    “| third autocorrelation inequality | 1.4581 | Carlos Vinuesa (2009) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Carlos Vinuesa (2009)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The verifiable reward by code executor significantly simplifies the action spaces of AlphaResearch, thereby enhancing the efficiency of the discovery process.

    “The verifiable reward by code executor significantly simplify the action spaces of AlphaResearch, thereby enhancing the efficiency of the discovery process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Once the generated optimal program surpasses current human-best achievements, these validated novel ideas could form feasible algorithms, thereby pushing the boundaries of human research forward.

    “Once the generated optimal program surpasses current human-best achievements, these validated novel ideas could form feasible algorithms, thereby pushing the boundaries of human research forward.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    AlphaResearch-RM-7B's performance on the AlphaResearch-RM test set demonstrates the model's generalization capabilities and the efficacy of incorporating feedback from a simulated real-world peer-review environment.

    “This outcome effectively demonstrates the model's generalization capabilities and the efficacy of incorporating feedback from a simulated real-world peer-review environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    AlphaResearchComp enables demonstration of reproducibility of established mathematical results and potential for discovery beyond current human-best achievements.

    “This design enables AlphaResearch to demonstrate both the reproducibility of established mathematical results and the potential for discovery beyond current human-best achievements.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Three authors independently assessed 100 randomly sampled examples to benchmark human performance.

    “To benchmark human performance, three of the authors independently assessed 100 randomly sampled examples, and we report their average scores.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The base model Qwen2.5-7B-Instruct's release date is earlier than the ICLR 2025 author-reviewer rebuttal period, preventing knowledge contamination.

    “We select Qwen2.5-7B-Instruct as our base model, whose release date (Sept, 2024) is earlier than the ICLR 2025 author-reviewer rebuttal period (Oct, 2024).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The evaluation program includes a verification module that validates whether programs conform to problem constraints and a measurement module that outputs performance scores.

    “The evaluation program E(•) includes two modules: (i) Verification module that validates whether p k conforms to the problem constraints.(ii) Measurement module that output the score r k of program performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    Each problem's baseline is supported by verifiable resources in the corresponding field.

    “Each problem's baseline is supported by verifiable resources in the corresponding field.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The human best for littlewood polynomials (n=512) is 32, attributed to Rudin-Shapiro (1959/1952).

    “| littlewood polynomials (n=512) | 32 | Rudin-Shapiro (1959/1952) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rudin-Shapiro (1959/1952)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

30 of 30

  • The paper relies on this without support.

    ICLR peer-review records are a suitable training signal for a reward model that evaluates algorithmic research proposals.

    “Train reward models with realworld peer-reviewed records.”

  • The paper relies on this without support.

    A reward model score below the threshold indicates a bad idea.

    “idea scrapping due to lower RM score than threshold”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MSTD baseline is described as having 30 elements but lists eight

    Minor

    The appendix describes the approximately 1.04 benchmark value as applying to |A|=30, then gives an eight-element set for that value. Elsewhere, N=30 denotes the indicator-array length rather than the set's cardinality.

  • Six problems are called underperformances, but two are ties

    Minor

    Table 4 shows AlphaResearch equaling the human result on Littlewood polynomials and MSTD. It therefore underperforms on four problems, not six, although it fails to surpass the human result on six.

  • GPT-5 is described as below 50% despite scoring 53%

    Minor

    The text says both baseline models scored below 50%, but Table 2 reports 53.0% for GPT-5.

  • Appendix says the benchmark has ten problems instead of eight

    Minor

    The appendix introduces ten benchmark problems, whereas the paper repeatedly specifies eight and Table 3 lists eight.

4 of 4

Cites

The publisher hasn’t deposited this paper’s reference list.

Cited by

Sorted by citations

Authors
Yu, Z., Feng, K., Zhao, Y., He, S., Zhang, X.-P., & Cohan, A.
Affiliations
  • Tsinghua University (CN)
  • New York University (US)
  • Yale University (US)
  • ByteDance (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2511.08522
arXiv
2511.08522
OpenAlex
W4416191637
Semantic Scholar
50cf5b62cea0c8e73a01263f0b9c18f5463a06c1
Cited by
0
References
0
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
algorithm discovery, large language models, open-ended problems, program synthesis, verifiable rewards, circle packing, benchmark dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review