CoChat
Sign Up
Cited by 75Topic ModelingNatural Language Processing TechniquesComputational and Text Analysis Methods

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Agent Laboratory: Using LLM Agents as Research Assistants

Open access
Schmidgall, Samuel, Su, Yusheng, Wang, Ze, Sun, Ximeng, Wu, Jialian, Yu, Xiaodong, Liu, Jiang, Moor, Michael, Liu, Zicheng, Barsoum, Emad·2025·Findings of the Association for Computational Linguistics: EMNLP 2025
DOI 10.18653/v1/2025.findings-emnlp.320
Open this paper in CoChat — free accountOpen PDF

Abstract

Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, Emad Barsoum. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.findings-emnlp.320

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Agent Laboratory's additional metrics include human and automated reviewer comparisons to ensure consistency and reliability in evaluations.

    “human and automated reviewer comparisons ensure consistency and reliability in evaluations.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Schmidgall et al. (2025)

    In “Automated versus Human Reviews” (page not provided), the paper compares human and automated scores but finds substantial discrepancies, not consistency. It says automated reviews do not align closely with human reviews and recommends reporting human evaluations alongside automated scores to better understand paper quality. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Agent Laboratory's performance dropped significantly in the literature review phase.

    “However, its performance dropped significantly in the literature review phase, reflecting the inherent challenges of automating structured literature reviews.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Schmidgall et al., 2025

    Section 6.2 (page unavailable) reports that models struggled during the literature review phase and sometimes reached the maximum number of steps, terminating the phase. That supports a performance problem, but the paper does not establish a significant drop in performance or attribute it to an inherent difficulty of automating structured literature reviews. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Agent Laboratory introduces additional evaluation metrics for agentic AI systems.

    “The recent proposed framework, Agent Laboratory Schmidgall et al. (2025), introduces additional evaluation metrics that provide a more comprehensive assessment of agentic AI systems.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Schmidgall et al. (2025)

    Sections 4.1 and 4.2 describe multiple measures used to evaluate Agent Laboratory, including research quality, usefulness, and co-pilot usability. The paper does not present these as newly introduced metrics for assessing agentic AI systems generally, or establish that they provide a more comprehensive assessment of such systems. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Schmidgall et al. (2025) reported that among the phases of data preparation, experimentation, report writing, and research report generation, the literature review phase exhibited the highest failure rate.

    “For instance, Schmidgall et al. (2025) reported that among the phases of data preparation, experimentation, report writing, and research report generation, the literature review phase exhibited the highest failure rate.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Schmidgall et al. (2025)

    Section 6.2 (Common failure modes; page unavailable) reports that models sometimes reached the step limit and terminated during literature review. The Figure 6 caption says phase-level success rates were evaluated, but the passages in our copy do not establish that literature review had the highest failure rate compared with data preparation, experimentation, or report writing. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Our analysis shows that while previous systems have performed well in fields such as chemistry, biology, and general science, literature review remains a significant challenge across nearly all approaches, especially in tasks like research idea generation and scientific discovery.

    “Our analysis shows that while previous systems have performed well in fields such as chemistry, biology, and general science, literature review remains a significant challenge across nearly all approaches, especially in tasks like research idea generation(Baek et al., 2024) and scientific discovery (Schmidgall et al., 2025).”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Schmidgall et al., 2025

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 85

  1. Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Janch- eska, John Yang, Carlos E Jimenez, Farshad Khor- rami, and 1 others. 2024. Enigma: Enhanced interac- tive generative model agent for ctf challenges. arXiv preprint arXiv:2409.16165. · arXiv
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. · arXiv
  3. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Lit- search: A retrieval benchmark for scientific literature search. arXiv preprint arXiv:2407.18940. · arXiv
  4. Altera AL, Andrew Ahn, Nic Becker, Stephanie Car- roll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, and 1 others. 2024. Project sid: Many-agent simulations toward ai civilization. arXiv preprint arXiv:2411.00114. · arXiv
  5. Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. 2024. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, pages 413-425.
  6. Joshua Ashkinaze, Julia Mendelsohn, Li Qiwei, Ceren Budak, and Eric Gilbert. 2024. How ai ideas affect the creativity, diversity, and evolution of human ideas: Evidence from a large, dynamic experiment. arXiv preprint arXiv:2401.13481. · arXiv
  7. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, and 1 others. 2023. Agent- verse: Facilitating multi-agent collaboration and ex- ploring emergent behaviors. In The Twelfth Interna- tional Conference on Learning Representations.
  8. Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Mingchen Zhuge, Jür- gen Schmidhuber, Xin Gao, and Xiangliang Zhang. 2024a. Scholarchemqa: Unveiling the power of lan- guage models in chemical research question answer- ing. arXiv preprint arXiv:2407.16931. · arXiv
  9. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, and 1 others. 2024b. Scienceagent- bench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080. · arXiv
  10. Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259. · arXiv
  11. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  12. Ning Ding, Shang Qu, Linhai Xie, Yifei Li, Zaoqu Liu, Kaiyan Zhang, Yibai Xiong, Yuxin Zuo, Zhangren Chen, Ermo Hua, and 1 others. 2024. Automating exploratory proteomics research via language models. arXiv preprint arXiv:2411.03743. · arXiv
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. · arXiv
  14. Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664. · arXiv
  15. Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hu- bert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, and 1 oth- ers. 2022. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53.
  16. Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. 2024. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36.
  17. Alireza Ghafarollahi and Markus J Buehler. 2024b. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning. arXiv preprint arXiv:2409.05556. · arXiv
  18. Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khy- ati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, and 1 others. 2024. Large lan- guage models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprint arXiv:2411.03562. · arXiv
  19. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, and 1 others. 2024. Blade: Benchmarking language model agents for data-driven science. arXiv preprint arXiv:2408.09667. · arXiv
  20. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. Ds-agent: Auto- mated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402.17453. · arXiv
  21. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksan- dra Faust. 2023. A real-world webagent with plan- ning, long context understanding, and program syn- thesis. arXiv preprint arXiv:2307.12856. · arXiv
  22. Nam Le Hai, Dung Manh Nguyen, and Nghi DQ Bui. 2024. Repoexec: Evaluate code generation with a repository-level executable benchmark. arXiv preprint arXiv:2406.11927. · arXiv
  23. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2024. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Ad- vances in neural information processing systems, 36.
  24. Andreas Happe and Jürgen Cito. 2023. Getting pwn'd by ai: Penetration testing with large language mod- els. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 2082-2086.
  25. Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, and 1 others. 2024. Simulating 500 million years of evolution with a language model. bioRxiv, pages 2024-07.
  26. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to- end web agent with large multimodal models. arXiv preprint arXiv:2401.13919. · arXiv
  27. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, and 1 others. 2024. Infiagent- dabench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.05507. · arXiv
  28. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.11610. · arXiv
  29. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning.
  30. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. · arXiv
  31. Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. 2024. Autonomous llm-driven re- search from data to human-verifiable research papers. arXiv preprint arXiv:2404.17605. · arXiv
  32. Junfeng Jiao, Saleh Afroogh, Yiming Xu, and Connor Phillips. 2024. Navigating llm ethics: Advancements, challenges, and future directions. arXiv preprint arXiv:2406.18841. · arXiv
  33. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770. · arXiv
  34. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wen- lin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2024. Dsbench: How far are data science agents to becoming data science ex- perts? arXiv preprint arXiv:2409.07703. · arXiv
  35. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, and 1 others. 2021. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-589.
  36. Hao Kang and Chenyan Xiong. 2024. Researcharena: Benchmarking llms' ability to collect and organize information as research agents. arXiv preprint arXiv:2406.10291. · arXiv
  37. Ji Woong Kim, Tony Z Zhao, Samuel Schmidgall, An- ton Deguet, Marin Kobilarov, Chelsea Finn, and Axel Krieger. 2024. Surgical robot transformer (srt): Im- itation learning for surgical tasks. In 8th Annual Conference on Robot Learning.
  38. Jakub Lála, Odhran O'Donoghue, Aleksandar Shtedrit- ski, Sam Cox, Samuel G Rodriques, and Andrew D White. 2023. Paperqa: Retrieval-augmented gener- ative agent for scientific research. arXiv preprint arXiv:2312.07559. · arXiv
  39. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52008.
  40. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, and 1 others. 2024a. Chain of ideas: Revolutionizing research via novel idea development with llm agents. arXiv preprint arXiv:2410.13185. · arXiv
  41. Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xi- aochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, and Hengxing Cai. 2024b. Scil- itllm: How to adapt llms for scientific literature un- derstanding. arXiv preprint arXiv:2408.15545. · arXiv
  42. Xinna Lin, Siqi Ma, Junjie Shan, Xiaojing Zhang, Shell Xu Hu, Tiannan Guo, Stan Z Li, and Kaicheng Yu. 2024. Biokgbench: A knowledge graph checking benchmark of ai agent for biomedical science. arXiv preprint arXiv:2407.00466. · arXiv
  43. Yiren Liu, Si Chen, Haocong Cheng, Mengxia Yu, Xiao Ran, Andrew Mo, Yiliu Tang, and Yun Huang. 2024. How ai processing delays foster creativity: Explor- ing research question co-creation with an llm-based agent. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1-25.
  44. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foer- ster, Jeff Clune, and David Ha. 2024a. The ai scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. · arXiv
  45. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Fo- erster, Jeff Clune, and David Ha. 2024b. The AI Scientist: Towards fully automated open-ended sci- entific discovery. arXiv preprint arXiv:2408.06292. · arXiv
  46. Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, An- ton Pashkov, and 1 others. 2024. Large language models surpass human experts in predicting neuro- science results. Nature Human Behaviour, pages 1-11.
  47. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Bal- dassari, Andrew D White, and Philippe Schwaller. 2024. Augmenting large language models with chem- istry tools. Nature Machine Intelligence, pages 1-11.
  48. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2024. Discov- erybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725. · arXiv
  49. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, and 1 others. 2023. Towards accurate differential diag- nosis with large language models. arXiv preprint arXiv:2312.00164. · arXiv
  50. Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. 2023. Scaling deep learning for materials discovery. Nature, 624(7990):80-85.
  51. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. · arXiv
  52. Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations.
  53. Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. 2024. Hyperagent: Generalist software engineering agents to solve coding tasks at scale. arXiv preprint arXiv:2409.16299. · arXiv
  54. Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. 2024. Citeme: Can language models accurately cite scientific claims? arXiv preprint arXiv:2407.12861. · arXiv
  55. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199. · arXiv
  56. Edward O Pyzer-Knapp, Jed W Pitera, Peter WJ Staar, Seiji Takeda, Teodoro Laino, Daniel P Sanders, James Sexton, John R Smith, and Alessandro Cu- rioni. 2022. Accelerating materials discovery using artificial intelligence, high performance computing and robotics. npj Computational Materials, 8(1):84.
  57. Chen Qian, Yufan Dang, Jiahao Li, Wei Liu, Zihao Xie, Yifei Wang, Weize Chen, Cheng Yang, Xin Cong, Xiaoyin Che, and 1 others. 2023. Experien- tial co-learning of software-developing agents. arXiv preprint arXiv:2312.17025. · arXiv
  58. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, and 1 others. 2024. Chatdev: Com- municative agents for software development. In Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 15174-15186.
  59. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. · arXiv
  60. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, and 1 others. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468-475.
  61. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems.
  62. Samuel Schmidgall and Michael Moor. 2025. Agen- trxiv: Towards collaborative autonomous research. arXiv preprint arXiv:2503.18102. · arXiv
  63. Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. Agentclinic: a multimodal agent benchmark to eval- uate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960. · arXiv
  64. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Her- nandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In In- ternational Conference on Machine Learning, pages 3135-3144. PMLR.
  65. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Process- ing Systems, 36.
  66. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large- scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109. · arXiv
  67. Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, and 1 others. 2024. Cs-bench: A comprehensive bench- mark for large language models towards computer science mastery. arXiv preprint arXiv:2406.08587. · arXiv
  68. Kyle Swanson, Wesley Wu, Nash L Bulaong, John E Pak, and James Zou. 2024. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, pages 2024-11.
  69. Nathan J Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E Kumar, Tanjin He, David Milsted, Matthew J McDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, and 1 others. 2023. An autonomous labo- ratory for the accelerated synthesis of novel materials. Nature, 624(7990):86-91.
  70. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  71. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, and 1 others. 2024. Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654. · arXiv
  72. A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.
  73. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, and 1 others. 2024. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv:2408.01605. · arXiv
  74. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291.
  75. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, and 1 others. 2024a. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.
  76. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xi- angru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024b. Open- devin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741. · arXiv
  77. Ryan Watkins. 2024. Guidance for researchers and peer- reviewers on the ethical use of large language models (llms) in scientific research workflows. AI and Ethics, 4(4):969-974.
  78. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- its reasoning in large language models. Advances in neural information processing systems, 35:24824- 24837.
  79. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2024. Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816. · arXiv
  80. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Auto- gen: Enabling next-gen llm applications via multi- agent conversation framework. arXiv preprint arXiv:2308.08155. · arXiv
  81. Jiacen Xu, Jack W Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swami- nathan, and Zhou Li. 2024. Autoattacker: A large lan- guage model guided system to implement automatic cyber-attacks. arXiv preprint arXiv:2403.01038. · arXiv
  82. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shen- gran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066. · arXiv
  83. John Yang, Carlos E Jimenez, Alexander Wettig, Kil- ian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer inter- faces enable automated software engineering. arXiv preprint arXiv:2405.15793. · arXiv
  84. Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Er- gen, Dongsub Shim, Honglak Lee, and 1 others. 2024. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows. arXiv preprint arXiv:2406.06357. · arXiv
  85. Yilun Zhou, Caiming Xiong, Silvio Savarese, and Chien- Sheng Wu. 2024. Shared imagination: Llms halluci- nate alike. arXiv preprint arXiv:2407.16604. · arXiv

Cited by

Sorted by citations

Related

Metadata

Authors
Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Moor, M., Liu, Z., & Barsoum, E.
Affiliations
  • Johns Hopkins University (US)
  • ETH Zurich (CH)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2025
Publisher
Association for Computational Linguistics
Published
2025
Pages
5977-6043
DOI
10.18653/v1/2025.findings-emnlp.320
OpenAlex
W4416035220
Cited by
59
References
0
Citations by year
2026: 40 · 2025: 19
FWCI
56.25
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
LLM agents, research assistants

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review