Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Open accessAbstract
Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into operational code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, particularly from the authors of those papers, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins. Code is available at: https://github.com/going-doer/Paper2Code.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PaperCoder achieves a score of 51.1 ± 1.4 on PaperBench.
“PaperCoder, a specialized multi-agent framework designed for transforming machine learning papers into executable code, achieves a score of 51.1 ± 1.4, outperforming all LLM agents baselines.”
From DeepCode: Open Agentic Coding (2025) · cites this paper as [8]
Table 2 (Data Processing; page not provided) reports a PaperBench Code-Dev replication score of 44.26% for PaperCoder, not 51.1 ± 1.4. It does show PaperCoder outperforming the two agent baselines listed. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPrior benchmarks in manuscript error detection and AI-assisted science have relied on raw PDFs or text-only inputs, offloading document understanding to OCR and parsing modules rather than the LLM itself, thereby conflating upstream parser failures with downstream model errors.
“While prior benchmarks in manuscript error detection [24] and AI-assisted science [38] have relied on raw PDFs or text-only inputs, this approach offloads document understanding to OCR and parsing modules rather than the LLM itself, thereby conflating upstream parser failures with downstream model errors.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [38]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredPaperCoder employs a multi-agent pipeline spanning planning, analysis, and generation, reaching 51.14% reproduction rate on PaperBench.
“PaperCoder employs a multi-agent pipeline spanning planning, analysis, and generation, reaching 51.14% reproduction rate on PaperBench [8].”
From DeepCode: Open Agentic Coding (2025) · cites this paper as [8]
The Abstract describes PaperCoder as a multi-agent framework with planning, analysis, and generation stages. Table 2 (Data Processing; page unavailable) reports a 44.26% replication score on PaperBench Code-Dev, not 51.14%. (LLM, conf=0.93)
Experimental — accuracy not yet measuredPaperBench and Paper2CodeBench rely substantially on manual assessment criteria and LLM-based correctness judgments, introducing potential inconsistencies and reliability concerns.
“While recent parallel efforts such as PaperBench (Starace et al., 2025) and Paper2CodeBench (Seo et al., 2025) have tackled related problems by exploring full codebase reconstruction, these evaluation approaches rely substantially on manual assessment criteria and LLM-based correctness judgments, introducing potential inconsistencies and reliability concerns.”
From SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers (2025) · cites this paper as Seo et al., 2025
Section 4.3.2 describes an author-written rubric and model-based assessment for PaperBench. The Model-Based Evaluation section describes language-model correctness scores and human-defined implementation criteria for Paper2Code. The paper does not conclude that these approaches introduce inconsistencies or reliability problems; its Detailed Analysis on Generated Repository section instead reports alignment with human judgments and describes model-based scores as reliable proxies. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv. org/abs/2108.07732. · arXiv
- Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. CoRR, abs/2404.07738, 2024. doi: 10.48550/ARXIV.2404.07738. URL https://doi.org/10. 48550/arXiv.2404.07738. · DOI · Paper page
- Monya Baker. 1,500 scientists lift the lid on reproducibility, 2016. URL https://www.nature. com/articles/533452a.
- Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum?id=ktrw68Cmu9c.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung, ed- itors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html.
- Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=VtmBAGCN7o.
- Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openreview.net/forum?id=1Fs1LvjYQW.
- Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R. Banaji. Chatgpt as research scientist: Probing gpt's capabilities as a research librarian, research ethicist, data generator and data predictor. CoRR, abs/2406.14765, 2024. doi: 10.48550/ARXIV.2406.14765. URL https://doi.org/10.48550/arXiv.2406.14765. · DOI
- Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. Chain of ideas: Revolutionizing research via novel idea development with LLM agents. CoRR, abs/2410.13185, 2024. doi: 10.48550/ARXIV.2410.13185. URL https://doi.org/10. 48550/arXiv.2410.13185. · DOI
- Tianyang Liu, Canwen Xu, and Julian J. McAuley. Repobench: Benchmarking repository- level code auto-completion systems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=pPjZIOuQuF.
- Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zheng Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark B. Gerstein. Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. 2023. URL https://api.semanticscholar.org/CorpusID:265221105.
- Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Hugo Larochelle. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res., 22:164:1-164:20, 2021. URL https://jmlr.org/papers/v22/20-303.html.
- Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Jo hannes Heidecke, Amelia Glaese, and Tejal A. Patwardhan. Paperbench: Evaluating ai's ability to replicate ai research. 2025. URL https://arxiv.org/pdf/2504.01848. · arXiv
- Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 -482, 2024. URL https://www.nature. com/articles/s41586-023-06747-5.
- Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=z8TW0ttBPp.
- Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Nicolaus Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. 2025. URL https://arxiv.org/pdf/2504.08066. · arXiv
- Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 -Volume 1: Long Papers, St. Julian's, Malta, March 17-22, 2024, pages 2931-2959. Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.eacl-long.179.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html.
Cited by
Sorted by citations
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5 · see citing works
“5 Related Works AI Co-Scientists Recent breakthroughs have pushed LLMs to PhD-level performance on STEM benchmarks [4], driving efforts to embed them as generators of the scientific forward pass, encompassing hypothesis generation [6], experimental planning [38], and manuscript drafting [9].”
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Kohler, B., Zollikofer, D., Einsiedler, J., Hoyle, A., & Ash, E. (2026) · arXiv · doi:10.48550/arxiv.2604.21965 · Cited by 1 · see citing works
“In Nguyen et al. (2026), the agents try System Domain Has Code New Data Evaluation CORE-Bench (Siegel et al., 2024) Multi ✓ Det. PaperBench (Starace et al., 2025) ML LLM Paper2Code (Seo et al., 2026) ML LLM AutoReproduce (Zhao et al., 2025) ML Both FIRE-Bench (Wang et al., 2026) ML LLM REPRO-Bench (Hu et al., 2025) So…”
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“While recent parallel efforts such as PaperBench (Starace et al., 2025) and Paper2CodeBench (Seo et al., 2025) have tackled related problems by exploring full codebase reconstruction, these evaluation approaches rely substantially on manual assessment criteria and LLM-based correctness judgments, introducing potential…”
ReproAgent: Contract-Guided Paper-to-Code Reproduction
Hu, X., Pan, Z., Wang, Z., Liu, Z., Su, Z., & Zhang, W. (2026) · arXiv · doi:10.48550/arxiv.2608.24291
“Following DLPa-per2Code (Sethi et al., 2018), recent systems extend reproduction to whole repositories: Paper-Coder, the system introduced by Seo et al. (Seo et al., 2026), stages planning, analysis, and coding; Sci-Reproducer (Xiang et al., 2025) highlights missing-algorithm bottlenecks; AutoP2C (Lin et al., 2025) an…”
InfluentialLi, Z., Li, Z., Guo, Z., Ren, X., & Huang, C. (2025) · arXiv · doi:10.48550/arxiv.2512.07921
- Authors
- Seo, M., Baek, J., Lee, S., & Hwang, S. J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.17192
- OpenAlex
- W4416872478
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- multi-agent LLM framework, large language models, reproducibility of research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
