Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 3 of the 16 sources its claims cite were checked for support.

Towards end-to-end automation of AI research

Open access

Abstract

The automation of science is a long-standing ambition in artificial intelligence (AI) research 1,2 . Although the community has made substantial progress in automating individual components of the scientific process, a system that autonomously navigates the entire research life cycle—from conception to publication—has remained out of reach. Here we present a pipeline for automating the entire scientific process end to end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyses data, writes the entire scientific manuscript, and performs its own peer review. Its ideas, execution and presentation are of sufficient quality that the manuscript generated by this AI system passed the first round of peer review for a workshop of a top-tier machine learning conference. The workshop had an acceptance rate of 70%. Our system leverages modern foundation models 3–5 within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold for conducting research on a specific topic and a template-free, open-ended mode that leverages agentic search for wider scientific exploration 6,7 . Both settings produce diverse ideas and automatically test, report on and evaluate them. This achievement demonstrates the growing capacity of AI for making scientific contributions and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be important risks, including taxing overwhelmed review systems and adding noise to the scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery.

Overview

CoChat Summary

TL;DR

The paper presents The AI Scientist, an agentic pipeline that autonomously generates machine-learning research ideas, code, experiments, plots, manuscripts, and peer reviews. One fully AI-generated manuscript received an average score of 6.33 (6, 7, 6) at an ICLR 2025 workshop, above the workshop’s average acceptance threshold, though it was withdrawn per protocol.

Key findings

  • The AI Scientist completed an end-to-end research workflow; one of three submissions to the ICLR 2025 ICBINB workshop received an average score of 6.33 (individual scores 6, 7, 6), above the average acceptance threshold. The organizers said it would likely have been accepted if not withdrawn; the workshop acceptance rate was 70% versus 32% for the ICLR 2025 main conference.
  • The Automated Reviewer performed comparably to human reviewers: balanced accuracy 69% versus 66%, and F1 score 0.62 versus 0.49 for inter-human agreement. On contamination checks, balanced accuracy decreased from 69% before the knowledge cutoff to 66% in the clean 2025 dataset, remaining comparable to human reviewers.
  • Paper quality increased with better base models and more test-time compute; recent model generations produced papers approaching borderline acceptability for machine-learning conference workshops.
  • The template-free system can generate its own starting code, uses agentic tree search, and typically takes from several hours to over 15 h depending on problem complexity.

Methods

  • Two AI Scientist modes: template-based (human-provided code template, Aider, sequential plan with up to five experiments, 7,200 s timeout per experiment, up to four debug reattempts) and template-free (open-ended, parallelized agentic tree search, four experiment stages, VLM critique, HuggingFace dataset access, direct LaTeX generation). It uses foundation models including o3, Claude Sonnet 4, GPT-4o, o4-mini, and o1.
  • The Automated Reviewer uses o4-mini, follows NeurIPS reviewer guidelines, produces structured scores and accept/reject decisions, and ensembles five reviews plus a meta-review acting as area chair. It was validated on ICLR OpenReview data, including 1,000 papers from 2017–2024 and a clean 2025 dataset.
  • Human evaluation submitted three generated manuscripts to the ICLR 2025 ICBINB workshop with IRB approval and organizer cooperation; reviewers were blind to which submissions were AI-generated, and all AI-generated submissions were withdrawn after review regardless of outcome.

Limitations

  • Stated: only one of three submissions was accepted, and workshops have much higher acceptance rates than main conferences (70% versus 32%). The AI Scientist cannot yet meet top-tier publication standards nor consistently meet workshop standards; failure modes include naive or underdeveloped ideas, incorrect implementations, lack of deep methodological rigour, experimental errors, duplicated figures, and hallucinations such as inaccurate citations.
  • Stated: it currently conducts computational experiments only. Ethical and societal concerns include overwhelming peer review, artificially inflating research credentials, repurposing others’ ideas without credit, eliminating scientist jobs, and conducting unethical or dangerous experiments; more research is needed for safe, aligned open-ended exploratory AI.
  • Evident from the text: human filtering selected the most promising outputs at each stage before submission, so the submitted set was not fully unfiltered autonomous output; the paper notes that without filtering, final papers would still be produced but along with other papers at greater total cost. The paper also states that the comparison between Automated Reviewer and human reviewers is not exact because ICLR and NeurIPS paper pools differ, creating a distribution shift.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-026-10265-5

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The AI Scientist workflow. a, The AI Scientist consists of distinct phases covering automated idea generation, tree-based experimentation, manuscript writing and reviewing. The experimentation phase uses an agentic tree search to generate and refine code implementations. This is structured into four stages: (1) initial investigation, (2) hyperparameter tuning, (3) research agenda execution and (4) ablation studies. From one experimental stage to the next, the best-performing checkpoint is selected to seed the next stage of the tree search. b, Scores for The AI Scientist papers across model releases. Paper quality consistently improves with the underlying model release date (as judged by The Automated Reviewer), indicating consistent future improvements with improving foundation models. The observed correlation is statistically significant (P < 0.00001). Shaded regions represent the standard error. Points represent mean scores with error bars and shaded regions indicating the standard error (n = 6 for template-free points, n = 3 for template-based points). Full experimental details, including model versions and replication counts, are provided in Supplementary Information section A.2.9. c, Automated review versus conference decisions. The Automated Reviewer achieves performance comparable with that of human reviewers, as validated by openly available decisions from past conferences (Table 1). Bars represent mean balanced accuracy; error bars show 95% bootstrapped confidence intervals (5,000 replicates). For replicability, each automated review is a 5-run ensemble. Two-sample z-tests on subsampled accuracy (automated n = 698/876, human n = 412) showed no significant difference before the training cutoff (P = 0.319) or post-cutoff (P = 0.921). Non-parametric bootstrap tests on F1 scores showed automated outperformance (P < 0.001).
Fig. 1 The AI Scientist workflow. a, The AI Scientist consists of distinct phases covering automated idea generation, tree-based experimentation, manuscript writing and reviewing. The experimentation phase uses an agentic tree search to generate and refine code implementations. This is structured into four stages: (1) initial investigation, (2) hyperparameter tuning, (3) research agenda execution and (4) ablation studies. From one experimental stage to the next, the best-performing checkpoint is selected to seed the next stage of the tree search. b, Scores for The AI Scientist papers across model releases. Paper quality consistently improves with the underlying model release date (as judged by The Automated Reviewer), indicating consistent future improvements with improving foundation models. The observed correlation is statistically significant (P < 0.00001). Shaded regions represent the standard error. Points represent mean scores with error bars and shaded regions indicating the standard error (n = 6 for template-free points, n = 3 for template-based points). Full experimental details, including model versions and replication counts, are provided in Supplementary Information section A.2.9. c, Automated review versus conference decisions. The Automated Reviewer achieves performance comparable with that of human reviewers, as validated by openly available decisions from past conferences (Table 1). Bars represent mean balanced accuracy; error bars show 95% bootstrapped confidence intervals (5,000 replicates). For replicability, each automated review is a 5-run ensemble. Two-sample z-tests on subsampled accuracy (automated n = 698/876, human n = 412) showed no significant difference before the training cutoff (P = 0.319) or post-cutoff (P = 0.921). Non-parametric bootstrap tests on F1 scores showed automated outperformance (P < 0.001).CC BY · PubMed Central
Fig. 2 Selected sections from a paper generated by The AI Scientist that was accepted via peer review at a top-tier machine learning conference workshop. The paper received peer-review scores of 6 (weak accept), 7 (accept) and 6 (weak accept) before meta-review and ranked among the top 45% of papers submitted for peer review. This demonstrates that a fully AI-generated paper can navigate the peer-review process successfully at a top-tier conference workshop. A full-sized version of this paper is available in Supplementary Information section D.2.1.
Fig. 2 Selected sections from a paper generated by The AI Scientist that was accepted via peer review at a top-tier machine learning conference workshop. The paper received peer-review scores of 6 (weak accept), 7 (accept) and 6 (weak accept) before meta-review and ranked among the top 45% of papers submitted for peer review. This demonstrates that a fully AI-generated paper can navigate the peer-review process successfully at a top-tier conference workshop. A full-sized version of this paper is available in Supplementary Information section D.2.1.CC BY · PubMed Central
Fig. 3 The phases and compute scaling of the AI Scientist. a, The research experimentation phase is visualized as a four-stage process. A preliminary baseline code implementation is first constructed (stage 1) and refined by tuning the hyperparameters (stage 2). The resultant code serves as a starting point for executing the research agenda through an agentic tree search (stage 3), followed by ablation experiments (stage 4). Full details of the agentic tree search process are provided in Methods. b, A real example of tree search by The AI Scientist with node annotations outlining the experiments conducted in the four different stages. c, Scores for The AI Scientist papers across compute budgets. Scaling the number of experimental nodes in the agentic tree search shows improvements for deeper test-time search budgets. Error bars represent the standard error. Full experimental details are provided in Supplementary Information section A.2.9.
Fig. 3 The phases and compute scaling of the AI Scientist. a, The research experimentation phase is visualized as a four-stage process. A preliminary baseline code implementation is first constructed (stage 1) and refined by tuning the hyperparameters (stage 2). The resultant code serves as a starting point for executing the research agenda through an agentic tree search (stage 3), followed by ablation experiments (stage 4). Full details of the agentic tree search process are provided in Methods. b, A real example of tree search by The AI Scientist with node annotations outlining the experiments conducted in the four different stages. c, Scores for The AI Scientist papers across compute budgets. Scaling the number of experimental nodes in the agentic tree search shows improvements for deeper test-time search budgets. Error bars represent the standard error. Full experimental details are provided in Supplementary Information section A.2.9.CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 2 Not valid · 9 Not checked · 50 Own results and interpretations

23 more claims

Create a free account to see all 63 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 41 references

Research works printed but not linked: 16

  1. Lenat, D. B. Automated theory formation in mathematics. In Proc. 5th International Joint Conference on Artificial Intelligence 833–842 (ed. Reddy, R.) (William Kaufmann, 1977).
  2. The Claude 3 Model Family: Opus, Sonnet, Haiku https://api.semanticscholar.org/CorpusID:268232499 (Anthropic, 2024).
  3. Chan, J. S. et al. MLE-bench: evaluating machine learning agents on machine learning engineering. In Proc. 13th International Conference on Learning Representation (ed. Yue, Y. et al.) (ICLR, 2025).
  4. Faldor, M., Zhang, J., Cully, A. & Clune, J. OMNI-EPIC: open-endedness via models of human notions of interestingness with environments programmed in code. In Proc. 13th International Conference on Learning Representations (eds Yue, Y. et al.) (ICLR, 2025).
  5. Hu, S., Lu, C. & Clune, J. Automated design of agentic systems. In Proc. 13th International Conference on Learning Representations (eds Yue, Y. et al.) (ICLR, 2025).
  6. Huang, Q., Vora, J., Liang, P. & Leskovec, J. MLAgentBench: evaluating language agents on machine learning experimentation. In Proc. 41st International Conference on Machine Learning Vol. 235 (eds Salakhutdinov, R. et al.) 20271–20309 (PMLR, 2024).
  7. Ma, Y. J. et al. Eureka: human-level reward design via coding large language models. In Proc. 12th International Conference on Learning Representations (eds Kim, B. et al.) (ICLR, 2024).
  8. González-Márquez, R. & Kobak, D. Learning representations of learning representations. In Proc. Data-centric Machine Learning Research (DMLR) Workshop at ICLR (ICLR, 2024).
  9. Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W. The NeurIPS 2021 consistency experiment. NeurIPS Blog https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment (2021).
  10. Lehmann, E. L. Testing Statistical Hypotheses 1st edn (Wiley, 1959).
  11. I Can’t Believe it’s Not Better: Challenges in Applied Deep Learning. ICLR Workshop at ICLR 2025 http://openreview.net/group?id=ICLR.cc/2025/Workshop/ICBINB (OpenReview, 2025).
  12. 13th Annual International Conference on Learning Representations (ICLR) 2025 Fact Sheet https://media.iclr.cc/Conferences/ICLR2025/ICLR2025_Fact_Sheet.pdf (ICLR, 2025).
  13. Szegedy, C. et al. Intriguing properties of neural networks. In Proc. 2nd International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (ICLR, 2014).
  14. Brown, T. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems Vol. 33 (eds Larochelle, H. et al.) 1877–1901 (Curran Associates, 2020).
  15. Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) Vol. 30, 5998–6008 (Curran Associates, 2017).
  16. El-Kishky, A. OpenAI o1 System Card https://api.semanticscholar.org/CorpusID:272648256 (OpenAI, 2024).

Cited by

See all 116

Metadata

Authors
Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J.
Affiliations
  • University of British Columbia (CA)
  • Sakana AI (Japan) (JP)
  • University of Oxford (GB)
Corresponding author
  • Yamada, Yutaro
  • Ha, David
  • Clune, Jeff
Venue
Nature
Published
2026-03-25
Volume
651
Issue
8107
Pages
914-919
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W7140287209
Cited by
116
References
41
Citations by year
2026: 115
FWCI
387.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Canadian Institute for Advanced Research
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
Keywords
research idea generation, code generation, manuscript preparation, peer review, foundation models, agentic systems, agentic search, research lifecycle, machine learning conferences
MeSH terms
Machine Learning, Artificial Intelligence, Automation, Humans, Research, Research Personnel, Peer Review, Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards end-to-end automation of AI research. https://app.cochat.ai/papers/10.1038/s41586-026-10265-5
The paper
Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J. (2026). Towards end-to-end automation of AI research. Nature. https://doi.org/10.1038/s41586-026-10265-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.