CoChat
Sign Up
PreprintCited by 19Information Retrieval and Search BehaviorTopic ModelingArtificial Intelligence Applications

DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

Open access
Du, Mingxuan, Xu, Benfeng, Zhu, Chiwei, Wang, Xiaorui, Mao, Zhendong·2025·arXiv
DOI 10.48550/arxiv.2506.11763arXiv 2506.11763
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Research Agents are a prominent category of LLM-based agents. By autonomously orchestrating multistep web exploration, targeted retrieval, and higher-order synthesis, they transform vast amounts of online information into analyst-grade, citation-rich reports--compressing hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we present DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. Evaluating DRAs is inherently complex and labor-intensive. We therefore propose two novel methodologies that achieve strong alignment with human judgment. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The other framework is introduced to evaluate DRA's information retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com/Ayanami0730/deep_research_bench to accelerate the development of practical LLM-based agents.

AI snapshot

Written from the paper’s full text.

TL;DR

DeepResearch Bench is a 100-task benchmark for evaluating Deep Research Agents, built from real user-query distributions across 22 domains and paired with two evaluation frameworks, RACE and FACT. RACE assesses report quality via reference-based adaptive criteria, while FACT measures citation accuracy and effective citations, with human-consistency experiments supporting both.

Key findings

  • The benchmark contains 100 PhD-level research tasks across 22 fields, balanced as 50 Chinese and 50 English tasks, derived from 96,147 raw user queries filtered to 44,019 deep-research queries.
  • RACE(Full) achieved the strongest human consistency among automated methods: Pairwise Agreement Rate 71.33, Overall Pearson Correlation 99.54, Filtered Average Pearson 60.24, Filtered Average Spearman 59.12, and Overall Score 72.56, exceeding the vanilla prompt baseline (58.89, 98.89, 40.30, 43.75, 60.46) and human inter-agreement PAR of 68.44.
  • In main evaluation, Gemini-2.5-Pro Deep Research led RACE overall at 48.88 and FACT effective citations at 111.21; OpenAI Deep Research scored 46.98 overall and led Instruction-Following at 49.27; Perplexity Deep Research scored 42.25 overall with Citation Accuracy 90.24; Grok Deeper Search scored 40.24 overall with Citation Accuracy 83.59 and 8.15 effective citations.
  • Among LLMs with search tools, Claude-3.7-Sonnet w/Search scored 40.67 overall, 93.68 Citation Accuracy, and 32.48 effective citations; Gemini-2.5-Flash’s FACT support judgments aligned with humans in 96% of “support” cases and 92% of “not support” cases on 100 sampled statement-URL pairs.

Methods

  • Benchmark construction: collected 96,147 anonymized raw user queries, used DeepSeek-V3-0324 to filter to 44,019 deep-research queries, classified them into 22 WebOrganizer topic domains, then proportionally compressed the distribution to 100 tasks authored by PhD or senior practitioners with more than five years of domain experience and manually screened.
  • Evaluation frameworks: RACE uses LLM-as-a-Judge with four dimensions—Comprehensiveness, Insight/Depth, Instruction-Following, and Readability—dynamically generating task-specific dimension weights and criteria, then scoring target reports relative to high-quality reference reports. FACT extracts statement-URL pairs, deduplicates them, retrieves webpage content via the Jina Reader API, judges whether each source supports its statement, and computes Citation Accuracy and Average Effective Citations per Task.
  • Experiments: evaluated four DRAs (Gemini-2.5-pro Deep Research, OpenAI Deep Research, Grok Deeper Search, Perplexity Deep Research) and several LLMs with search tools; used Gemini-2.5-pro as RACE judge and Gemini-2.5-flash as FACT judge; human consistency used 50 Chinese tasks, four agents, three domain experts per task, 70+ annotators, and 225 person-hours.

Limitations

  • Stated: benchmark scale is constrained by the expertise and effort required to create authentic tasks; the 100-task size limits statistical robustness and topic coverage.
  • Stated: domain coverage bias may remain despite multiple review layers and explicit acceptance criteria.
  • Stated: human evaluation throughput was limited to three experts per task, 50 tasks, and four agents (600 reports), making the sample modest and susceptible to individual variability. Evident from the text: RACE reference reports came from Gemini-2.5-pro-based Deep Research as available in April 2025, and the authors cannot confirm whether model iterations occurred during reference collection; commercial model outputs were collected in specific 2025 time windows.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.11763

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

19 of 19 cited sources checked · 7 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

    A comprehensive benchmark for systematically evaluating the capabilities of Deep Research Agents remains absent.

    “However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent.”
    • Also stated in: § 1 Introduction · plain
      “These demanding evaluation requirements for DRAs pose a significant challenge for existing evaluation frameworks, which often fall short of offering a dedicated assessment of the multifaceted capabilities of such agents [14].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [14] [14] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, October 2023. arXiv:2308.03688 [cs]. - source

      The Introduction (p. 2) says that a systematic, standard benchmark for evaluating LLMs as agents was lacking. It does not make that claim specifically about Deep Research Agents, and the paper goes on to introduce AgentBench as a comprehensive benchmark for LLMs as agents (Introduction, pp. 2–3). It therefore does not establish that such a benchmark remains absent. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 3.1 RACE: A Framework for Report Quality Evaluation

    Existing approaches using fixed checklists or static rubrics struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks.

    “Existing approaches using fixed checklists [23] or static rubrics [25, 1] struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [23] [23] Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models, September 2024. arXiv:2409.16191 [cs]. - source

      Section 3.1 (page unavailable) says earlier checklist-based approaches can give every quality aspect equal weight, causing evaluation bias. But the paper does not establish that static rubrics struggle with deep-research tasks or specialized domains; Appendix D (page unavailable) says domain-specific checklists were unnecessary for its open-ended QA evaluation. (LLM, conf=0.81)

      Experimental — accuracy not yet measured
    • cites [25] [25] Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models, April 2024. arXiv:2402.14207 [cs]. - source

      Section 4.2 (page unavailable) describes a five-point rubric for evaluating Wikipedia-like articles, so the paper supports the reference to a rubric-based approach. Neither that section nor Appendix C.2 says such rubrics struggle to adapt to diverse tasks, specialized domains, or nuanced quality aspects of deep research tasks. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [1] [1] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs, August 2024. arXiv:2408.07055 [cs]. - source

      The LongBench-Write section and Appendix C (page not provided) describe a fixed, six-dimension rubric for evaluating writing responses. They do not say that static rubrics struggle to adapt to diverse tasks, specialized domains, or nuanced aspects of deep research, so the claimed limitation is unsupported. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1 Introduction

    Existing evaluation frameworks often fall short of offering a dedicated assessment of the multifaceted capabilities of DRAs.

    “These demanding evaluation requirements for DRAs pose a significant challenge for existing evaluation frameworks, which often fall short of offering a dedicated assessment of the multifaceted capabilities of such agents [14].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [14] [14] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, October 2023. arXiv:2308.03688 [cs]. - source

      The Introduction (p. 2) says most agent benchmarks focus on a single environment and fail to give a comprehensive view across application scenarios; Related Work (p. 9) says many benchmarks do not evaluate multi-round interaction or agents' ability to act. This supports the general shortfall in evaluating varied agent capabilities, but the cited paper does not specifically assess frameworks for DRAs or their dedicated evaluation requirements. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data
    § 4.2.2 Evaluation on FACT Framework

    Deep Research Agents, except Grok, tend to include more Effective Citations than LLMs with Search Tools.

    “Viewing evaluation results by FACT in Table 1, Deep Research Agents, except Grok, tend to include more Effective Citations than LLMs with Search Tools.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 4.2.1 Evaluation on RACE Framework

    OpenAI Deep Research surpasses Gemini-2.5-Pro Deep Research in the Instruction-Following dimension of RACE.

    “OpenAI Deep Research also exhibits strong capabilities, surpassing Gemini-2.5-Pro Deep Research in the Instruction-Following dimension.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 22
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation
    § 5 Related Work

    Existing efforts employing the LLM-as-a-judge methodology lack both a comprehensive framework design and verification of human consistency.

    “Others employ the LLM-as-a-judge methodology [36], yet these efforts lack both a comprehensive framework design and verification of human consistency.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [36] [36] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, December 2023. arXiv:2306.05685 [cs]. - source

      The paper does employ LLM-as-a-judge, but the claim that such efforts lack verification of human consistency conflicts with its reported comparison against human evaluations. The Introduction (p. 2) reports over 80% agreement, at the level of human–human agreement, and proposes a hybrid evaluation framework; the Conclusion (p. 10) reiterates the human-agreement result. Whether that framework is ‘comprehensive’ is a separate judgment the source does not establish. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.1 Topic Distribution Analysis

    The topic taxonomy proposed by WebOrganizer was adopted, selecting 22 distinct topic domains.

    “To categorize the deep research queries, we adopt the topic taxonomy proposed by WebOrganizer [29], selecting 22 distinct topic domains for this classification.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] [29] Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation, February 2025. arXiv:2502.10341 [cs]. - source

      Section 2.1 (p. 3) confirms that WebOrganizer proposes a topic taxonomy, but says it contains 24 categories, not 22. The citing authors may have selected 22 of those categories for their own classification; the source does not document that selection. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisecitation
    § 1 Introduction

    Deep Research Agents are the most widely utilized LLM-based agents today.

    “Deep research represents one such well-defined task domain, with Deep Research Agents (DRAs) [13, 37, 24] emerging as the most widely utilized LLM-based agents today.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [13] [13] Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. WebThinker: Empowering Large Reasoning Models with Deep Research Capability, April 2025. arXiv:2504.21776 [cs]. - source

      The Introduction describes deep research as a research task and presents WebThinker as a deep research agent. It does not say that Deep Research Agents are the most widely utilized LLM-based agents or compare their usage with other agents. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [37] [37] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments, April 2025. arXiv:2504.03160 [cs]. - source

      The Introduction names commercial and open-source deep-research agents, but does not compare their usage with other LLM-based agents. Searches for the claimed usage ranking and for adoption found no passage addressing whether DRAs are the most widely utilized. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [24] [24] Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants, January 2025. - source

      The LLM Agents and Background & Related Work sections describe research agents and their applications, but do not compare how widely different kinds of LLM agents are used. Searches for “Deep Research Agents,” “most widely utilized,” and adoption or prevalence found no passage supporting the claimed ranking; page numbers were unavailable in our copy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § 1 Introduction

    Current benchmarks typically focus on isolated capabilities such as web browsing and information retrieval, or generative abilities disconnected from real-time information acquisition.

    “Current benchmarks typically focus on assessing isolated capabilities-such as web browsing and information retrieval [28, 38, 39], or generative abilities disconnected from real-time information acquisition [23, 1, 31].”
    Quote in the paperSubstantial claim6 of 6 sources checked
    • cites [28] [28] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents, April 2025. arXiv:2504.12516 [cs]. - source
    • cites [38] [38] Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese, May 2025. arXiv:2504.19314 [cs]. - source
    • cites [39] [39] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. arXiv:2307.13854 [cs]. - source

      The Introduction and §3 identify WebArena as a benchmark for web-based tasks, so it fits the web-browsing portion of the cited clause. But they describe diverse, long-horizon tasks evaluated by whether agents achieve their goals, not a benchmark focused on an isolated browsing or information-retrieval capability. The sentence attaches separate citations to the claim about generative abilities, so that clause is not attributed to WebArena. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [23] [23] Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models, September 2024. arXiv:2409.16191 [cs]. - source
    • cites [1] [1] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs, August 2024. arXiv:2408.07055 [cs]. - source
    • cites [31] [31] Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A Comprehensive Benchmark for Generative Writing, March 2025. arXiv:2503.05244 [cs]. - source

  • May combine two claims.

    Premisenone given
    § 5 Related Work

    Using QA datasets as evaluation metrics for DRAs neither aligns with real-world DRA applications nor comprehensively assesses their broader capabilities.

    “Among these works, some utilize QA datasets [21, 16, 30] as evaluation metrics; however, this approach neither aligns with real-world DRA applications nor comprehensively assesses their broader capabilities.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [21] [21] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jaeho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Serguei Popov, Robert Gerbicz, Geoff Galgon, source
    • cites [16] [16] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. arXiv:2311.12983 [cs]. - source

      In “Evaluating General Assistants” (page unavailable), GAIA says repurposed evaluation datasets may fail to test tool use, while closed-environment evaluations may miss results grounded in real-world interactions. This supports the concern about assessing broader capabilities, but the paper does not assess QA datasets for deep research agents specifically; its own benchmark uses real-world questions. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [30] [30] Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. WebWalker: Benchmarking LLMs in Web Traversal, January 2025. arXiv:2501.07572 [cs]. -

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 3.1 RACE: A Framework for Report Quality Evaluation

    Directly prompting LLMs to generate task-specific criteria from scratch can lead to results that deviate significantly from the intended assessment goals.

    “Directly prompting LLMs to generate task-specific criteria from scratch can lead to results that deviate significantly from the intended assessment goals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2.1 Evaluation on RACE Framework

    RACE scores show a strong linear correlation with human judgments, suggesting the framework captures meaningful performance differences between models.

    “However, this should not be concerning, as subsequent experiments 4.3 reveal a strong linear correlation between these RACE scores and human judgments, suggesting the framework effectively captures meaningful performance differences between models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • May combine two claims.

    Findingnone given
    § 1 Introduction

    RACE and FACT are two novel evaluation frameworks that assess report generation quality and information retrieval abilities of Deep Research Agents, respectively.

    “We further propose RACE and FACT , two novel evaluation frameworks that respectively assess the report generation quality and the information retrieval abilities of Deep Research Agents.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

    DeepResearch Bench consists of 100 PhD-level research tasks, each crafted by domain experts across 22 distinct fields.

    “we present DeepResearch Bench , a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2.2 Evaluation on FACT Framework

    Among LLMs with Search Tools, Claude-3.7-Sonnet achieves the second highest number of Effective Citations and demonstrates strong Citation Accuracy.

    “Among the LLMs with Search Tools, Claude-3.7-Sonnet achieves the second highest number of Effective Citations and demonstrates strong Citation Accuracy; this also aligns with Claude-3.7-Sonnet w/Search achieving the best overall score in the RACE framework.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 1 Introduction

    DeepResearch Bench is a 100-task benchmark spanning 22 domains, with each task crafted and iteratively refined by domain experts.

    “we introduce DeepResearch Bench , a 100-task benchmark spanning 22 domains, with each task meticulously crafted and iteratively refined by domain experts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3.3 Comparison of Different Evaluation Methods

    RACE(Full) exhibits the strongest overall performance, significantly outperforming both the vanilla baseline and other RACE variants.

    “As shown in Table 2, RACE(Full) exhibits the strongest overall performance, significantly outperforming both the vanilla baseline and other RACE variants.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • May combine two claims.

    Findingreasoning
    § 4.3.2 Evaluation Metrics

    After filtering out tasks with negative ICC, 37 tasks remained, forming a subset with higher expert consensus.

    “After applying this filtering criterion, 37 tasks (out of the original set) remained in our experiment, forming a subset with demonstrably higher expert consensus.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.1 RACE: A Framework for Report Quality Evaluation

    Preliminary experiments suggest that scoring reports in isolation often yields insufficiently discriminative results; models tend to assign uniformly high scores, masking genuine quality variations.

    “Preliminary experiments suggest that scoring reports in isolation often yields insufficiently discriminative results; models tend to assign uniformly high scores, thereby masking genuine quality variations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2.1 Evaluation on RACE Framework

    OpenAI Deep Research and Gemini-2.5-Pro Deep Research show comparable performance, both significantly outperforming the other two Deep Research Agents.

    “It can be observed that OpenAI Deep Research and Gemini-2.5-Pro Deep Research show comparable performance, both significantly outperforming the other two Deep Research Agents.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 2.1 Topic Distribution Analysis

    Filtering the 96,147 raw queries yielded 44,019 queries conforming to the deep research task definition.

    “This process ultimately yields a dataset of 44,019 queries conforming to our deep research task definition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2.2 Evaluation on FACT Framework

    Gemini-2.5-Pro Deep Research achieves an average of 111.21 effective citations in its final reports, significantly outperforming other models.

    “Notably, Gemini-2.5-Pro Deep Research achieves an average of 111.21 effective citations in its final reports, significantly outperforming other models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2.1 Evaluation on RACE Framework

    Individual models maintain relatively stable performance across different topic tasks, demonstrating the robustness of the RACE evaluation framework.

    “When viewing scores of different models across topics in Figure 5, we can observe that individual models maintain relatively stable performance across different topic tasks, demonstrating the robustness of our RACE evaluation framework.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 2.2 Benchmark Task Collection

    The final DeepResearch Bench dataset consists of 100 tasks, 50 in Chinese and 50 in English, proportionally compressed from the determined distribution.

    “we proportionally compress this determined distribution to form a final dataset of 100 tasks, 50 in Chinese and 50 in English.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The in-house dataset of 96,147 user queries is representative of the real-world distribution of human research task demands.

  • May combine two claims.

    Findingown data
    § 4.3.4 Comparison of Different Judge LLM

    Gemini 2.5 Pro Preview achieves the best overall performance as Judge LLM and maintains a competitive average cost ($0.13 per query), only higher than o4-mini.

    “As detailed in Table 3, Gemini 2.5 Pro Preview not only achieves the best overall performance but also maintains a competitive average cost ($0.13 per query), only higher than that of o4-mini.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 5 Related Work

    DeepResearch Bench addresses the gap by providing a systematic and unified evaluation method with strong human consistency.

    “In contrast, our DeepResearch Bench addresses this gap by providing a systematic and unified evaluation method with strong human consistency, thereby supporting subsequent DRA development and assessment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § C Judge LLM Selection for the FACT Framework

    Gemini-2.5-Flash demonstrated comparable accuracy to Gemini-2.5-Pro for FACT-specific tasks but at a more advantageous cost.

    “Since Gemini-2.5-Flash demonstrated comparable accuracy to Gemini-2.5-Pro for these specific tasks but at a more advantageous cost, we select Gemini-2.5-Flash as the Judge LLM for the FACT framework.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The operations within the FACT framework (such as extracting statements from full reports and analyzing webpage content for support) are token-intensive, making cost-effectiveness a critical consideration.

  • May combine two claims.

    Findingown data
    § 6 Conclusion

    RACE and FACT have demonstrated high consistency with human judgments, affirming their reliability.

    “Our key evaluation frameworks, RACE and FACT, have demonstrated high consistency with human judgments, affirming their reliability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § G.1 Human Evaluation Effort

    The human evaluation process culminated in a total of 225 person-hours across all tasks and annotators.

    “This meticulous process culminated in a total of 225 person-hours of human evaluation across all tasks and annotators.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § G.1 Human Evaluation Effort

    On average, each expert annotator spent approximately 1.5 hours per query to evaluate the reports from the four agents.

    “On average, each expert annotator spent approximately 1.5 hours per query to evaluate the reports from the four agents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 6 Conclusion

    DeepResearch Bench is the first comprehensive benchmark for evaluating the report generation and web retrieval capabilities of Deep Research Agents.

    “In this work, we introduce DeepResearch Bench, the first comprehensive benchmark for evaluating the report generation and web retrieval capabilities of Deep Research Agents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 6 Conclusion

    DeepResearch Bench comprises 100 high-quality research tasks across 22 distinct domains, curated to reflect authentic user needs.

    “Comprising 100 high-quality research tasks across 22 distinct domains, this benchmark is meticulously curated to reflect authentic user needs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning
    § 1 Introduction

    Evaluating the quality of extensive research reports remains an open-ended problem because establishing a definitive ground truth for complex research queries is almost impossible.

    “Moreover, evaluating the quality of these extensive research reports remains an open-ended problem, as establishing a definitive 'ground truth' or standard answer for complex research queries is almost impossible.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 2.2 Benchmark Task Collection

    All submitted tasks underwent manual screening by the team to verify quality, clarity, complexity, and alignment with the deep research definition.

    “All submissions undergo a manual screening by our team to verify their quality, clarity, complexity, and alignment with our definition of deep research.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 4.3 Human Consistency

    For each task, three domain-expert annotators scored the reports.

    “For each task, three domain-expert annotators score these reports.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.1 RACE: A Framework for Report Quality Evaluation

    Four top-level orthogonal dimensions were established based on domain expertise: Comprehensiveness, Insight/Depth, Instruction-Following, and Readability.

    “Therefore, we first establish four top-level orthogonal dimensions based on domain expertise: Comprehensiveness (COMP) , Insight/Depth (DEPTH) , Instruction-Following (INST) , and Readability (READ) .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four top-level dimensions (Comprehensiveness, Insight/Depth, Instruction-Following, Readability) are orthogonal and comprehensively cover report quality.

  • May combine two claims.

    Interpretationown data
    § 4.3.3 Comparison of Different Evaluation Methods

    RACE can reliably and accurately evaluate deep research reports, achieving high human consistency efficiently.

    “These results collectively demonstrate that RACE can reliably and accurately evaluate deep research reports, achieving high human consistency efficiently.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • May combine two claims.

    Interpretationreasoning
    § 1 Introduction

    The final generated report is the primary interface for assessing DRA overall performance because internal processes are not transparent.

    “Because their internal reasoning and information retrieval processes are not transparent, the final generated report becomes the primary interface through which their overall performance can be assessed.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 4.3 Human Consistency

    Human consistency experiments used 50 Chinese tasks from DeepResearch Bench, with reports generated by four distinct agents.

    “We conduct experiments using 50 Chinese tasks from DeepResearch Bench, with reports generated by four distinct agents.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.
    • Rests on an assumption: The 50 Chinese tasks used in the human consistency experiments are representative of the full DeepResearch Bench.

  • May combine two claims.

    Interpretationreasoning
    § 4.2.1 Evaluation on RACE Framework

    The different RACE dimensions are somewhat decoupled, and a sufficiently powerful Judge LLM can capture performance variations across these dimensions.

    “This indicates that the different dimensions are somewhat decoupled and that a sufficiently powerful Judge LLM can capture performance variations across these various dimensions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 2.2 Benchmark Task Collection

    Ph.D. or senior practitioners with more than five years of relevant domain experience were invited to propose candidate tasks.

    “We invite Ph.D. or senior practitioners with more than five years of relevant domain experience to propose candidate tasks, as shown in Figure 4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 4.3.1 Human Data Collection

    Each annotator was limited to a maximum of three queries to ensure diverse perspectives.

    “Each annotator is limited to a maximum of three queries to ensure diverse perspectives.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 4.3.1 Human Data Collection

    Annotators evaluated reports across four dimensions and overall performance using a custom interface, guided only by basic scoring criteria to minimize bias.

    “Using a custom interface, they evaluate reports across four dimensions and overall performance, guided only by basic scoring criteria to minimize bias.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductreasoning
    § 2.2 Benchmark Task Collection

    The compression process ensures the benchmark maintains the same topical balance as the observed real-world distribution.

    “This compression process meticulously ensures that the benchmark maintains the same topical balance as the observed real-world distribution.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The in-house dataset of 96,147 user queries is representative of the real-world distribution of human research task demands.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 4.1 Experimental Setup

    An initial pre-processing step cleaned citation formatting from generated reports because overly lengthy or complex citation styles can adversely affect the Judge LLM's scoring process.

    “When employing the RACE framework, an initial pre-processing step involves cleaning citation formatting from the generated reports, as overly lengthy or complex citation styles can adversely affect the Judge LLM's scoring process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 2.1 Topic Distribution Analysis

    All raw query logs underwent rigorous anonymization to remove personally identifiable information.

    “To ensure user privacy, all raw query logs underwent rigorous anonymization, removing personally identifiable information such as user IDs, IP addresses, and session metadata.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 4.3.1 Human Data Collection

    70+ annotators with Master's degrees and relevant domain expertise were recruited to gather human judgments.

    “To gather human judgments, we recruit 70+ annotators with Master's degrees and relevant domain expertise.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § G.2 Configuration of LLMs with Web Search Tools

    The standardized settings for LLMs with web search tools were implemented to minimize variability and enable more direct comparison of models' capabilities.

    “These standardized settings were implemented to minimize variability arising from differing default configurations and to enable a more direct comparison of the models' capabilities in the context of deep research tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § A Limitations

    Each of the 100 tasks in DeepResearch Bench was developed by a verified PhD-level expert who underwent orientation on task specifications and participated in multiple rounds of refinement.

    “Each of our 100 tasks was developed by a verified PhD-level expert who underwent orientation on our task specifications and participated in multiple rounds of refinement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § A Limitations

    The human evaluation for the current study was limited to three experts per task, covering 50 tasks across four agents (totaling 600 reports).

    “For the current study, we limited human evaluation to three experts per task, covering 50 tasks across four agents (totaling 600 reports).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given
    § 5 Related Work

    The field still lacks a standardized evaluation methodology for DRAs, preventing meaningful comparative analysis of their capabilities.

    “However, the field still lacks a standardized evaluation methodology for these DRAs, preventing meaningful comparative analysis of their capabilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § A Limitations

    The human evaluation sample remains modest and susceptible to individual variability.

    “While sufficient to validate the RACE framework, this sample remains modest and susceptible to individual variability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § A Limitations

    DeepResearch Bench and the RACE & FACT evaluation framework offer the first comprehensive assessment of Deep Research Agents.

    “While DeepResearch Bench and the RACE & FACT evaluation framework offer the first comprehensive assessment of Deep Research Agents, several limitations remain.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § G.1 Human Evaluation Effort

    The human evaluation effort yielded a robust and reliable dataset of human judgments.

    “This substantial effort yielded a robust and reliable dataset of human judgments, serving as the basis for our analysis of human consistency.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § G.2 Configuration of LLMs with Web Search Tools

    The citation formatting and standardization was essential for equitable and rigorous factual verification.

    “This uniform approach to citation structure was essential for equitable and rigorous factual verification.”
    Quote in the paperSubstantial claim
50 of 50

  • The paper relies on this without support.

    The in-house dataset of 96,147 user queries is representative of the real-world distribution of human research task demands.

    “To obtain this distribution, we collect an in-house dataset of 96,147 raw user queries from interactions with web search-enabled LLM Chatbot.”
    “This distribution, visualized in Figure 3, indicates the real-world user demand for deep research within these domains.”

  • The paper relies on this without support.

    Human expert judgments are a valid gold standard for evaluating the quality of deep research reports.

    “to validate the effectiveness of our proposed RACE framework, we must assess its human consistency.”
    “To gather human judgments, we recruit 70+ annotators with Master's degrees and relevant domain expertise.”

  • The paper relies on this without support.

    The reference reports selected from Gemini-2.5-pro-based Deep Research are high-quality and suitable as references for RACE scoring.

    “The reference reports ( R ref ) used in RACE's scoring methodology were selected from deep research articles generated by the Gemini-2.5-pro-based Deep Research, as available in April 2025 3 .”
    “For each task t , a high-quality report R ref is selected as a reference.”

  • The paper relies on this without support.

    Citation Accuracy and Average Effective Citations per Task are valid measures of an agent's information retrieval and collection capabilities.

    “Citation Accuracy (C. Acc.) measures the precision of an agent's citations, reflecting its ability to ground statements with appropriate sources correctly. And Average Effective Citations per Task (E. Cit.) quantifies the average amount of valuable, verifiably supported information an agent retrieves and presents per task.”

  • The paper relies on this without support.

    The Judge LLM's binary support judgment accurately determines whether a cited webpage provides sufficient evidence to support a statement.

    “Then, the Judge LLM assesses whether this content provides sufficient evidence to support the statement. This results in a binary judgment ('support' or 'not support') for each pair, determining whether the citation accurately grounds the claim.”

  • The paper relies on this without support.

    The 50 Chinese tasks used in the human consistency experiments are representative of the full DeepResearch Bench.

    “We conduct experiments using 50 Chinese tasks from DeepResearch Bench, with reports generated by four distinct agents.”

  • The paper relies on this without support.

    The four top-level dimensions (Comprehensiveness, Insight/Depth, Instruction-Following, Readability) are orthogonal and comprehensively cover report quality.

    “we first establish four top-level orthogonal dimensions based on domain expertise: Comprehensiveness (COMP) , Insight/Depth (DEPTH) , Instruction-Following (INST) , and Readability (READ) .”

  • The paper relies on this without support.

    Creating tasks that authentically reflect real research challenges requires substantial expertise and effort.

    “Creating tasks that authentically reflect real research challenges requires substantial expertise and effort.”

  • The paper relies on this without support.

    The assessment of research reports presents inherent challenges: a single report may contain dozens of pages with dense technical content and sophisticated analysis.

    “The assessment of research reports presents inherent challenges: a single report may contain dozens of pages with dense technical content and sophisticated analysis.”

  • The paper relies on this without support.

    Even for PhD-level evaluators, a thorough review typically requires 30-60 minutes, with the evaluation of multiple reports multiplying this burden.

    “Even for PhD-level evaluators, a thorough review typically requires 30-60 minutes, with the evaluation of multiple reports multiplying this burden.”

  • The paper relies on this without support.

    The operations within the FACT framework (such as extracting statements from full reports and analyzing webpage content for support) are token-intensive, making cost-effectiveness a critical consideration.

    “The operations within the FACT framework (such as extracting statements from full reports and analyzing webpage content for support) are known to be token-intensive, making cost-effectiveness a critical consideration.”

  • The paper relies on this without support.

    Human scores from a limited number of experts (k=3) for n=4 reports per task can affect task-level metric stability.

    “Given that human scores are from a limited number of experts ( k = 3 ) for the n = 4 reports per task, expert inconsistencies can affect task-level metric stability.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DeepResearch Bench's scoring approach leans heavily on overlap with reference solutions and simple citation counts.

    “This benchmark confirmed the difficulty of deep research tasks, where no model exceeded roughly 30% on their overall metrics, yet its scoring approach leans heavily on overlap with reference solutions and simple citation counts.”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [8]

    Section 3.1 describes RACE as comparing reports with a reference against task-specific, weighted criteria; it does not describe scoring by simple overlap. Section 3.2 and Appendix E say FACT counts citations only after checking whether the cited source supports each statement, and also measures citation accuracy—not simple citation counts. The cited sentence’s roughly 30% ceiling is also inconsistent with Table 1, which reports RACE overall scores above 40 for several models. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • Existing benchmarks face two primary limitations: reliance on static datasets or answer keys, and use of non-expert or automated evaluation including coarse metrics or auto-generated rubrics.

    “To summarize, existing benchmarks thus face two primary limitations: a reliance on static datasets or answer keys [15, 16], and the use of non-expert or automated evaluation, including coarse metrics [20] or auto-generated rubrics [8, 11, 24].”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [8]

  • Some recent deep research benchmarks introduce LLM-generated rubrics and evaluation metrics reliant upon LLM-generated reference reports, raising concerns about circularity and limited oversight.

    “some benchmarks introduce LLM-generated rubrics and evaluation metrics reliant upon LLM-generated reference reports [8], thus raising concerns about circularity and limited oversight [7]”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [8]

  • DeepResearch Bench introduced 100 PhD-level research problems across 22 fields, each requiring a long-form report.

    “Closer to our setting, DeepResearch Bench [8] introduced 100 PhD-level research problems across 22 fields (e.g., scientific analysis, legal reasoning), each requiring a long-form report.”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [8]

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Du, M., Xu, B., Zhu, C., Wang, X., & Mao, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.11763
arXiv
2506.11763
OpenAlex
W4415069437
Semantic Scholar
cca73506ab839718879a49ccce389d33907aa053
Cited by
2
References
38
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
deep research agents, LLM agents, benchmark, citation accuracy, information retrieval evaluation, web exploration, reference-based evaluation, human judgment alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review