CoChat
Sign Up
PreprintCited by 33Information Retrieval and Search BehaviorTopic ModelingWeb Data Mining and Analysis

DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments

Open access
Zheng, Yuxiang, Fu, Dayuan, Hu, Xiangkun, Cai, Xiaojie, Ye, Lyumanshan, Lu, Pengrui, Liu, Pengfei·2025·arXiv
DOI 10.48550/arxiv.2504.03160arXiv 2504.03160
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) equipped with web search capabilities have demonstrated impressive potential for deep research tasks. However, current approaches predominantly rely on either manually engineered prompts (prompt engineering-based) with brittle performance or reinforcement learning within controlled Retrieval-Augmented Generation (RAG) environments (RAG-based) that fail to capture the complexities of real-world interaction. In this paper, we introduce DeepResearcher, the first comprehensive framework for end-to-end training of LLM-based deep research agents through scaling reinforcement learning (RL) in real-world environments with authentic web search interactions. Unlike RAG-based approaches that assume all necessary information exists within a fixed corpus, our method trains agents to navigate the noisy, unstructured, and dynamic nature of the open web. We implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures and overcoming significant technical challenges. Extensive experiments on open-domain research tasks demonstrate that DeepResearcher achieves substantial improvements of up to 28.9 points over prompt engineering-based baselines and up to 7.2 points over RAG-based RL agents. Our qualitative analysis reveals emergent cognitive behaviors from end-to-end RL training, including the ability to formulate plans, cross-validate information from multiple sources, engage in self-reflection to redirect research, and maintain honesty when unable to find definitive answers. Our results highlight that end-to-end training in real-world web environments is not merely an implementation detail but a fundamental requirement for developing robust research capabilities aligned with real-world applications. We release DeepResearcher at https://github.com/GAIR-NLP/DeepResearcher.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.03160

Cites

0 citation errors · 3 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • It is common to incorporate format reward when training reasoning models and multi-turn reasoning agents.

    “It is common to incorporate format reward when training reasoning models and multi-turn reasoning agents [15, 69, 24].”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [69]

    Section 3.4 (Reward; page unavailable) says DeepResearcher penalizes incorrectly formatted output during agent training. Section 5.1 describes its multi-turn reasoning, but the paper does not establish that format rewards are common across reasoning models and multi-turn agents generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Deep Researcher adopts the F1 score as the reward function.

    “Deep Researcher adopts the F1 score as the reward function.”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [69]

  • Many of these behaviors, including verification, making a plan, and iterative search, are also reported in recent studies on deep research agents.

    “Notably, many of these behaviors, including verification, making a plan, and iterative search, are also reported in recent studies on deep research agents [24, 69].”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [69]

  • Existing works on tool-using agent systems trained with RL leave memory management unsolved, letting the prompt length grow unboundedly.

    “Meanwhile, existing works on tool-using agent systems trained with reinforcement learning leave memory management unsolved, letting the prompt length grow unboundedly [24, 69].”

    From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [69]

    Section 2.2 discusses RL-trained search agents, and Section 3.1 describes DeepResearcher’s short-term browsing memory. Neither says that existing RL-trained tool-using agents leave memory management unsolved or allow prompts to grow unboundedly. Searches for memory management, prompt growth, and context-length limits found no passage supporting that assertion; page numbers are unavailable in our copy. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • Deep Research Agents are the most widely utilized LLM-based agents today.

    “Deep research represents one such well-defined task domain, with Deep Research Agents (DRAs) [13, 37, 24] emerging as the most widely utilized LLM-based agents today.”

    From DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) · cites this paper as [37]

    The Introduction names commercial and open-source deep-research agents, but does not compare their usage with other LLM-based agents. Searches for the claimed usage ranking and for adoption found no passage addressing whether DRAs are the most widely utilized. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 6

  1. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
  2. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta pro- gramming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations.
  3. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics.
  4. Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, and 1 oth- ers. 2024a. Searching for best practices in retrieval- augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 17716-17736.
  5. Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neu- big, and Pengfei Liu. 2024. Alignment for honesty. Advances in Neural Information Processing Systems, 37:63565-63598.
  6. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Conference on Empirical Methods in Natural Language Processing (EMNLP).

Cited by

Sorted by citations

Related

Authors
Zheng, Y., Fu, D., Hu, X., Cai, X., Ye, L., Lu, P., & Liu, P.
Affiliations
  • Shanghai Innovation Institute (CN)
  • Shanghai Jiao Tong University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
414-431
DOI
10.48550/arxiv.2504.03160
arXiv
2504.03160
OpenAlex
W4416037139
Cited by
25
References
0
Citations by year
2026: 21 · 2025: 4
FWCI
29.83
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (2015AA015408)
  • Shanghai Jiao Tong University
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
Keywords
large language models, deep research agents, reinforcement learning, web search, multi-agent systems, end-to-end learning, prompt engineering, retrieval-augmented generation, emergent behavior, self-reflection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review