CoChat
Sign Up
PreprintCited by 5Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research

Open access
Coelho, João, Ning, Jingjie, He, Jingyuan, Mao, Kangrui, Paladugu, Abhijay, Setlur, Pranav, Jin, Jiahe, Callan, Jamie +3 more·2025·arXiv
DOI 10.48550/arxiv.2505.19253arXiv 2505.19253
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce \textsc{DeepResearchGym} as an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is free for research use. To evaluate deep research systems' outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge to measure alignment with users' information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with~\textsc{DeepResearchGym} achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A case study on short-answer search agents further demonstrates the sandbox's utility for cost-effective training, showing that models trained within the sandbox can generalize to commercial search.

AI snapshot

Written from the paper’s full text.

TL;DR

DeepResearchGym is an open-source sandbox combining a free, reproducible search API over ClueWeb22 and FineWeb with an LLM-as-judge evaluation protocol for deep research systems. It reports comparable system performance to commercial search APIs, stable rankings across metrics, and cost-effective agent training that generalizes to commercial search.

Key findings

  • The search API indexes ClueWeb22-A/B and FineWeb CC-MAIN-2024-51; on Researchy Questions with ClueWeb22-B, retrieval at L=100 gives MRR@10 48.34, nDCG@10 39.40, R@100 78.06, rising to 48.45, 39.55, and 78.43 at L=500, while ANN recall R@10 rises from 90.01 to 95.39, indicating minimal approximate-search loss.
  • The API responds in under half a second, the /fetch endpoint has median latency 0.09 seconds per request, and during its first four months the public service recorded over 12 million search requests from 384 unique IP addresses across 13 countries.
  • Systems generally maintain relative rankings when moved from commercial search APIs to DeepResearchGym: GPT-Researcher KPR 60.61 commercial vs 64.67 ours, OpenDeepSearch 32.92 vs 42.81, HuggingFace-DeepSearch 33.00 vs 35.22; top systems score higher on clarity/insight than on KPR coverage.
  • Human validation over 210 queries with three annotators from seven co-authors found average pairwise Cohen’s κ 0.87 and LLM-human agreement κ=0.72 for KPR, 0.86 for citation precision and recall, 0.89 for clarity, and 0.84 for insightfulness; in the training case, 640,000 API calls would cost up to US$640 with SERPER or US$5,000 with Tavily, and the ClueWeb-trained agent generalized to Serper, e.g. GAIA 20.3 vs 18.4 and HLE 7.0 vs 6.9 for the Serper-trained agent, while trailing on WebWalkerQA 29.7 vs 35.4.

Methods

  • The sandbox uses dense retrieval with MiniCPM-Embedding-Light, 1024-dimensional embeddings, up to 8192 tokens, BEIR nDCG@10 55.27, DiskANN approximate nearest neighbor search, and 25-million-document shards, exposing /search and /fetch endpoints over ClueWeb22-A/B and FineWeb.
  • Evaluation extends Researchy Questions, about 96,000 real-world queries, with LLM-as-judge metrics: KPR/KPC for relevance, citation precision/recall for retrieval faithfulness, and clarity/insight for report quality; benchmarking uses the top 1,000 test queries, gpt-4.1-mini-2025-04-14 as judge, and ClueWeb22-B by default.
  • A case study trains a Qwen3-1.7B search agent with GRPO on 5,000 synthesized ClueWeb22-grounded queries, compares it to a Serper-trained agent, and evaluates on GAIA, WebWalkerQA, HLE, and a 53-query ClueWeb-Test subset.

Limitations

  • Stated: static web corpora limit time-sensitive queries; among 1,000 test queries, 81.6% were classified temporally stable and 18.4% temporally evolving, and the framework is not intended to substitute real-time evaluation for post-cutoff developments.
  • Stated: LLM-as-judge evaluation introduces variability and uses a proprietary LLM, limiting reproducibility; ClueWeb22 also requires a license, though FineWeb is immediately accessible.
  • Evident from the text: perplexity-sonar-deepsearch and gpt4-search-preview were not evaluated with DeepResearchGym’s API in Table 2, so cross-setting comparison is incomplete; ClueWeb-Test is only 53 queries; and human annotators were drawn from the co-author pool.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.19253

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 19 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 3.2.1 RESEARCHY QUESTIONS · p. 6

    The Researchy Questions dataset consists of approximately 96,000 real-world information-seeking queries that led users to engage with multiple documents during search sessions.

    “the dataset consists of approximately 96,000 real-world information-seeking queries that led users to engage with multiple documents during search sessions, as measured by aggregated click distributions over ClueWeb22.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Rosset et al., 2024 source
  • Premisecitation
    § 3.2.1 RESEARCHY QUESTIONS · p. 6

    The Researchy Questions dataset was curated specifically to capture queries that drive extensive information exploration and synthesis.

    “The Researchy Questions dataset (Rosset et al., 2024) was curated specifically to capture such queries.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Rosset et al., 2024 source
  • Findingown data
    § ABSTRACT · p. 1

    Performance rankings of systems remain consistent across evaluation metrics when integrated with DEEPRESEARCHGYM.

    “with performance rankings remaining consistent across evaluation metrics.”
    • Also stated in: § 4.2 SYSTEM-LEVEL EVALUATION · p. 7 · hedged
      “First, systems generally maintain their relative performance rankings across both retrieval settings, confirming that DEEPRESEARCHGYM's search API provides sufficient retrieval quality to support effective report generation.”
    Quote in the paperSubstantial claim
  • Findingown data
    § ABSTRACT · p. 1

    Systems integrated with DEEPRESEARCHGYM achieve performance comparable to those using commercial APIs.

    “Experimental results show that systems integrated with DEEPRESEARCHGYM achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics.”
    • Also stated in: § 4.2 SYSTEM-LEVEL EVALUATION · p. 7 · plain
      “systems generally maintain their relative performance rankings across both retrieval settings, confirming that DEEPRESEARCHGYM's search API provides sufficient retrieval quality to support effective report generation.”
    Quote in the paperSubstantial claim
  • Findingown data
    § ABSTRACT · p. 1

    The DEEPRESEARCHGYM search API achieves lower latency than popular commercial APIs.

    “It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is free for research use.”
    • Also stated in: § 3.1.3 RETRIEVAL API · p. 5 · plain
      “Our API consistently responds in under half a second, outperforming commercial services.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 45
  • May combine two claims. 34
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2.2 LONG-FORM REPORT EVALUATION METRICS · p. 7

    The dimensions of clarity and insightfulness are commonly used in long-form generation evaluation.

    “These dimensions are commonly used in long-form generation evaluation (Liu et al., 2023; Saha et al., 2024) and provide evidence of the presentation quality of the generated content.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Liu et al., 2023 source

      The Evaluation Criteria section discusses coherence and engagingness, but the passages found do not identify clarity and insightfulness as evaluation dimensions. Searches for both terms and for their use in long-form generation evaluation found no passage establishing the attributed claim. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites Saha et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § 1 INTRODUCTION · p. 2

    The DEEPRESEARCHGYM system achieves strong retrieval quality with minimal loss from approximate search.

    “Empirical evaluations show that the system achieves strong retrieval quality with minimal loss from approximate search, while maintaining response times below those attained by commercial APIs.”
    • Also stated in: § 3.1.2 SEARCH INDEXES · p. 4 · hedged
      “The results with increased L indicate that the error introduced by ANN search is minimal, solidifying the retrieval quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 SYSTEM-LEVEL EVALUATION · p. 8

    Commercial deep research systems tend to produce strong narrative quality but sometimes with reduced citation precision.

    “The evaluation also reveals a trade-off in commercial systems, which tend to produce strong narrative quality but sometimes with reduced citation precision.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4 HUMAN EVALUATION · p. 9

    Agreement between LLM-based automatic judgments and human preferences was strong: κ = 0.72 for KPR, 0.86 for both citation precision and recall, 0.89 for clarity, and 0.84 for insightfulness.

    “Agreement between LLMbased automatic judgments and human preferences was similarly strong: κ = 0 . 72 for KPR, 0 . 86 for both citation precision and recall, 0 . 89 for clarity, and 0 . 84 for insightfulness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 3.1.3 RETRIEVAL API · p. 5

    The DEEPRESEARCHGYM /fetch endpoint maintains a median latency of 0.09 seconds per single request.

    “The endpoint maintains a median latency of 0.09 seconds per single request.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning
    § 5 CASE STUDY: DEEPRESEARCHGYM API FOR SEARCH AGENT TRAINING · p. 9

    An experiment with 10,000 queries, 16 trajectories per query, and 4 searches per trajectory results in 640,000 API calls.

    “For instance, an experiment with 10,000 queries, 16 trajectories per query, and 4 searches per trajectory results in 640,000 API calls”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § A QUERY LOG ANALYSIS · p. 16

    The query log comprises roughly 12 million queries submitted by 384 unique IP addresses across 13 countries over the first four months.

    “we analyzed a query log comprising roughly 12 million queries submitted by 384 unique IP addresses across 13 countries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 6 CONCLUSION AND FUTURE WORK · p. 10

    Systems maintain comparable performance when transitioning from proprietary APIs to DEEPRESEARCHGYM's transparent environment, confirming preserved retrieval fidelity.

    “Evaluation-wise, Systems maintain comparable performance when transitioning from proprietary APIs to our transparent environment, confirming preserved retrieval fidelity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 RESULTS · p. 10

    The ClueWeb-trained agent generalizes effectively to commercial search at inference, matching or exceeding the Serper-trained baseline on GAIA and HLE, though trailing on WebWalkerQA.

    “Furthermore, the ClueWeb-trained agent generalizes effectively to commercial search at inference, matching or exceeding the Serper-trained baseline on GAIA and HLE, though trailing on WebWalkerQA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 5 CASE STUDY: DEEPRESEARCHGYM API FOR SEARCH AGENT TRAINING · p. 9

    Such an experiment costs up to US$640 with SERPER or US$5,000 with Tavily.

    “costing up to US$640 with SERPER or US$5,000 with Tavily.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § Insightfulness (Score: 9/10) · p. 31

    The example report received an insightfulness score of 9/10.

    “Insightfulness (Score: 9/10)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § E.2 KEY-POINT RECALL AND CONTRADICTION · p. 29

    The example report does not contradict any of the keypoints, yielding a key-point recall (KPR) of 6/13 and a key-point contradiction (KPC) of 0/13.

    “The report does not contradict any of the keypoints, hence KPR for this report would be computed as 6/13, and KPC as 0/13.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A QUERY LOG ANALYSIS · p. 17

    The long tail of repeating queries contributes substantially to overall query volume.

    “the long tail of repeating queries contributes substantially to overall query volume”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § Clarity (Score: 9/10) · p. 30

    The example report received a clarity score of 9/10.

    “Clarity (Score: 9/10)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A QUERY LOG ANALYSIS · p. 17

    Singleton queries account for roughly 27% of all queries.

    “Singleton queries account for roughly 27% of all queries”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § A QUERY LOG ANALYSIS · p. 17

    PERSON, DESCRIPTION, and LOCATION queries are the most frequent MS-MARCO types in the query log.

    “Figure 4 (right) further characterizes them by MS-MARCO types (Nguyen et al., 2016), with PERSON, DESCRIPTION, and LOCATION queries being the most frequent.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nguyen et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data
    § F ADDITIONAL BENCHMARKING RESULTS · p. 31

    For both deep research systems, Gemini-2.5-Pro and GPT-OSS-20B show strong agreement on relevance metrics (ρ≈0.8) and moderate agreement on faithfulness and quality (ρ≈0.4-0.6) with gpt-4.1-mini.

    “For both deep research systems, Gemini-2.5-Pro and GPT-OSS-20B both show strong agreement on relevance metrics ( ρ ≈ 0 . 8 ) and moderate agreement on faithfulness and quality ( ρ ≈ 0 . 4 -0 . 6 ).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § E.3 RETRIEVAL FAITHFULNESS · p. 30

    The claims shown in Table 6 were rated as fully supported by the source URLs.

    “Shown claims were rated as being fully supported by the source URLs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A QUERY LOG ANALYSIS · p. 16

    The top-25 most frequent first words in queries show a dominance of WH-questions and geo-spatial terms, another pattern consistent with information-seeking behavior.

    “Figure 3 (right) displays the top-25 most frequent first words in queries, highlighting a dominance of WH-questions (e.g., who, when) and geo-spatial terms (e.g., location, country), another pattern consistent with information-seeking behavior.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 34

    On the query 'what role did indians play in the wars for empire?', GPTResearcher interpreted 'Indians' as referring to people from India and produced a report on Indian troops in British imperial wars.

    “On the query 'what role did indians play in the wars for empire?', GPTResearcher interpreted 'Indians' as referring to people from India and produced a report on Indian troops in British imperial wars.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Findingown data
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 33

    The report is thematically correct but doesn't align with how the gold key points are structured.

    “The report is thematically correct but doesn't align with how the gold key points are structured.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 32

    On the query 'do the american population wish more autonomy in their work?', GPTResearcher generated a comprehensive report on workplace autonomy but missed gold key points addressing autonomy in other contexts.

    “on the query 'do the american population wish more autonomy in their work?', the system generated a comprehensive report on workplace autonomy, covering flexibility, remote work, and work-life balance. But it missed gold key points addressing autonomy in other contexts”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 33

    On the query 'what makes an athlete elite', GPTResearcher discussed psychological traits, training environments, coaching relationships, and genetic factors, but the gold key points emphasized neuromuscular mechanisms, so it failed to capture those key points.

    “Because the report never engaged with this mechanistic level, it failed to capture these key points despite being broadly accurate”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 32

    GPTResearcher produces a coherent report focused on the most salient interpretation but fails to cover additional sub-facets or long-tail details.

    “GPTResearcher produces a coherent report focused on the most salient interpretation but fails to cover additional sub-facets or long-tail details.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.1.2 SEARCH INDEXES · p. 4

    The MiniCPM-Embedding-Light model achieves a zero-shot performance of 55.27 in nDCG@10 on the BEIR benchmark, outperforming bge-large-en-v1.5 and jina-embeddings-v3.

    “It achieves competitive performance on multiple benchmarks, and shows good generalization ability given a zero-shot performance of 55.27 in nDCG@10 on the BEIR benchmark (Thakur et al., 2021), outperforming other popular alternatives such as bge-large-en-v1.5 (BAAI, 2024) and jina-embeddings-v3 (JinaAI, 2024), which achieve 54.29 and 53.88 in nDCG@10, respectively.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites OpenBMB, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Thakur et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    ClueWeb22-B includes around 200 million web pages, with approximately 87 million in English.

    “It includes around 200 million web pages, with approximately 87 million in English.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Overwijk et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    ClueWeb22 was collected in 2022 and comprises approximately 10 billion web pages.

    “ClueWeb22 was collected in 2022 and comprises approximately 10 billion web pages.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Overwijk et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    FineWeb comprises approximately 15 trillion tokens of cleaned and deduplicated web data.

    “It comprises approximately 15 trillion tokens of cleaned and deduplicated web data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Penedo et al., 2024 source

  • May combine two claims.

    Interpretationreasoning
    § 1 INTRODUCTION · p. 2

    This setup allows researchers to audit system behavior, analyze the influence of retrieved evidence, and rerun deep research experiments under reproducible search conditions.

    “This setup allows researchers to audit system behavior, analyze the influence of retrieved evidence, and rerun deep research experiments under reproducible search conditions, since retrieval results remain stable over time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § 4.4 HUMAN EVALUATION · p. 9

    The human study was conducted double-blind, with randomization of system assignment and report order, and ties were disallowed.

    “The study was conducted double-blind, with randomization of system assignment and report order, and ties were disallowed to enforce binary preferences.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 3.2 DEEP RESEARCH EVALUATION METHODS · p. 6

    The DEEPRESEARCHGYM evaluation protocol is agnostic to the specific evaluation task and compatible with a broad range of use cases.

    “While the sandbox is agnostic to the specific evaluation task and compatible with a broad range of use cases, we introduce this protocol to fill a gap in the evaluation landscape, and to provide initial empirical observations using our API.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 1 INTRODUCTION · p. 2

    The DEEPRESEARCHGYM evaluation protocol leverages Researchy Questions' ground-truth documents to yield more reliable judgments.

    “leveraging Researchy Questions' ground-truth documents to yield more reliable judgments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 4.3 QUERY-LEVEL ANALYSIS · p. 8

    The variability in retrieval faithfulness underscores the importance of using a standard retrieval API when benchmarking deep research systems.

    “This variability underscores the importance of using a standard retrieval API when benchmarking deep research systems, as it helps control for retrieval effects and ensures that observed differences stem from model behavior rather than different access to evidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 4.4 HUMAN EVALUATION · p. 9

    The KPC metric was excluded due to insufficient non-tied comparisons.

    “The KPC metric was excluded due to insufficient non-tied comparisons.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § 1 INTRODUCTION · p. 2

    Retrieval results remain stable over time.

    “since retrieval results remain stable over time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 3.1.3 RETRIEVAL API · p. 5

    FineWeb can be accessed immediately, while ClueWeb22 requires users to first obtain a license through ClueWeb's official channels.

    “FineWeb can be accessed immediately, while ClueWeb22 requires users to first obtain a license through ClueWeb's official channels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.1 EXPERIMENTAL SETUP · p. 7

    Search-o1 and Search-R1 serve as lower-bound references.

    “Although not designed for deep research tasks, these last two systems serve as lower-bound references and help verify that our evaluation metrics capture meaningful differences in generative capabilities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 3.2.1 RESEARCHY QUESTIONS · p. 6

    The heavy engagement with diverse sources reflects the essential challenges deep research systems are designed to address.

    “The heavy engagement with diverse sources reflects the essential challenges deep research systems are designed to address: synthesizing information across multiple perspectives, reconciling conflicting evidence, and constructing comprehensive responses.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 4.1 EXPERIMENTAL SETUP · p. 7

    OpenDeepSearch produces similarly comprehensive outputs, while Search-o1 and Search-R1 focus on concise, short-form answers.

    “OpenDeepSearch produces similarly comprehensive outputs, while Search-o1 and Search-R1 focus on concise, short-form answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    ClueWeb22-B approximates the super head of the web, encompassing the most frequently visited pages.

    “Category B, known as ClueWeb22-B, approximates the super head of the web, encompassing the most frequently visited pages (e.g., pages from Wikipedia, major news outlets, and other top domains).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Overwijk et al., 2022 source

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § 4.1 EXPERIMENTAL SETUP · p. 7

    The top 1,000 queries by number of clicked documents align with the goals of deep research systems.

    “This ranking naturally favors queries that drive extensive exploration, aligning with the goals of deep research systems.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The top 1,000 queries ranked by number of clicked documents align with the goals of deep research systems.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    The most recent FineWeb crawl includes over 180 million documents.

    “we focus on the most recent crawl, which includes over 180 million documents capturing more recent trends compared to earlier data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Penedo et al., 2024 source

  • Probably not a substantial claim: may not pass S4 (based). The quote only partly states this claim.

    Interpretationnone given
    § 4.1 EXPERIMENTAL SETUP · p. 7

    The evaluation metrics capture meaningful differences in generative capabilities.

    “help verify that our evaluation metrics capture meaningful differences in generative capabilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 5 CASE STUDY: DEEPRESEARCHGYM API FOR SEARCH AGENT TRAINING · p. 9

    Training search agents through reinforcement learning needs thousands of search API calls.

    “Training search agents through reinforcement learning needs thousands of search API calls.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    FineWeb was collected from 96 Common Crawl snapshots between 2013 and 2024.

    “FineWeb is a large-scale English web corpus collected from 96 Common Crawl snapshots between 2013 and 2024.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Penedo et al., 2024 source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given
    § 3.2 DEEP RESEARCH EVALUATION METHODS · p. 6

    The DEEPRESEARCHGYM evaluation protocol fills a gap in the evaluation landscape.

    “we introduce this protocol to fill a gap in the evaluation landscape, and to provide initial empirical observations using our API.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1.1 WEB CORPORA · p. 4

    ClueWeb22-A encompasses approximately 1 billion English pages from the mostly head of the web.

    “This larger subset encompasses approximately 1 billion English pages from the mostly head of the web, offering a more diverse mix of frequently visited websites.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Overwijk et al., 2022 source

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 4.1 EXPERIMENTAL SETUP · p. 7

    All four systems are capable of generating long-form reports.

    “All four systems are capable of generating long-form reports.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5.2 RESULTS · p. 10

    DEEPRESEARCHGYM enables cost-effective and reproducible training, where learned search strategies transfer across both engines and evaluation distributions.

    “these results confirm that DEEPRESEARCHGYM enables cost-effective and reproducible training, where learned search strategies transfer across both engines and evaluation distributions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § 6 CONCLUSION AND FUTURE WORK · p. 11

    The embedding model used in the dense retrieval pipeline is publicly available, and instructions for replicating experiments are provided.

    “The embedding model used in our dense retrieval pipeline is publicly available, and instructions for replicating experiments are provided.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 6 CONCLUSION AND FUTURE WORK · p. 10

    DEEPRESEARCHGYM offers a reproducible sandbox for developing and benchmarking deep research systems, providing a stable cost-effective alternative to commercial search APIs.

    “DEEPRESEARCHGYM offers a reproducible sandbox for developing and benchmarking deep research systems, providing a stable cost-effective alternative to commercial search APIs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 6 CONCLUSION AND FUTURE WORK · p. 11

    The paper used a proprietary LLM to support the evaluation, which brings further problems in terms of reproducibility.

    “Also, we used a proprietary LLM to support the evaluation, which brings further problems in terms of reproducibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6 CONCLUSION AND FUTURE WORK · p. 11

    The retrieval API has been made available as an online service.

    “The retrieval API has also been made available as an online service”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A QUERY LOG ANALYSIS · p. 16

    Prior studies of general-purpose search engines report that around 50% of queries are informational.

    “Prior studies of general-purpose search engines report that around 50% of queries are informational (Broder, 2002).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Broder, 2002 source

  • May combine two claims.

    Interpretationown data
    § 6 CONCLUSION AND FUTURE WORK · p. 10

    Agents trained exclusively within DEEPRESEARCHGYM generalize effectively to commercial search at inference time, validating the framework as a practical training environment.

    “Beyond evaluation, our case study shows that agents trained exclusively within DEEPRESEARCHGYM generalize effectively to commercial search at inference time, validating the framework as a practical training environment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 6 CONCLUSION AND FUTURE WORK · p. 11

    The automated evaluation used in the manuscript leverages LLMs as judges, which introduces some inherent variability.

    “We do acknowledge that the automated evaluation used in the manuscript leverages LLMs as judges, which introduces some inherent variability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § F ADDITIONAL BENCHMARKING RESULTS · p. 32

    The results in Table 2 leverage ClueWeb22 for search because it has better alignment with the Researchy Questions dataset.

    “Finally, our results in Table 2 leverage the ClueWeb22 for search, given its better alignment with the Researchy Questions dataset.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given
    § F ADDITIONAL BENCHMARKING RESULTS · p. 31

    Reliance on a single LLM judge can introduce model-specific biases.

    “reliance on a single LLM judge can introduce model-specific biases.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § B RESEARCHY QUESTIONS · p. 17

    The framework is not intended to substitute real-time evaluation for tasks where post-cutoff developments are essential.

    “This framing aligns with the goal of providing a reproducible research sandbox, while clarifying that the framework is not intended to substitute real-time evaluation for tasks where post-cutoff developments are essential.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning
    § B RESEARCHY QUESTIONS · p. 17

    Questions like why is there a chip shortage are inherently time-sensitive and may require post-cutoff updates, while queries like can there be knowledge that is independent of culture rely on conceptual and historical knowledge that remains stable across time.

    “Questions like why is there a chip shortage are inherently time-sensitive and may require post-cutoff updates, while queries such as can there be knowledge that is independent of culture rely on conceptual and historical knowledge that remains stable across time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § F ADDITIONAL BENCHMARKING RESULTS · p. 31

    The correlations suggest that while different judges may apply different scoring scales or exhibit distinct biases, the relative ordering of system performance remains consistent.

    “These correlations suggest that while different judges may apply different scoring scales or exhibit distinct biases, the relative ordering of system performance remains consistent.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § B RESEARCHY QUESTIONS · p. 17

    The benchmark evaluates whether systems can identify and use the correct supporting evidence within the available time-frame.

    “The benchmark therefore evaluates whether systems can identify and use the correct supporting evidence within the available time-frame.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § B RESEARCHY QUESTIONS · p. 17

    The static setting still provides controlled access to all ground-truth evidence available up to the snapshot date, even for evolving queries.

    “The static setting still provides controlled access to all ground-truth evidence available up to the snapshot date, even for evolving queries.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The static corpus provides all ground-truth evidence available up to the snapshot date.

  • May combine two claims.

    Interpretationreasoning
    § B RESEARCHY QUESTIONS · p. 17

    DEEPRESEARCHGYM prioritizes stable and transparent experimentation, which requires fixed snapshots, yet this choice constrains queries that rely on information appearing after the corpus cutoff.

    “Our framework prioritizes stable and transparent experimentation, which requires fixed snapshots, yet this choice constrains queries that rely on information appearing after the corpus cutoff.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § A QUERY LOG ANALYSIS · p. 16

    The queries are more complex, consistent with the goals of a deep research system.

    “This further suggests that queries are more complex, consistent with the goals of a deep research system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § G QUALITATIVE ANALYSIS: FAILURE MODES · p. 34

    Domain misinterpretations represent more severe failures where ambiguous terminology leads the system to research the wrong topic entirely.

    “and represent more severe failures where ambiguous terminology leads the system to research the wrong topic entirely.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § H.1 DATA SYNTHESIS AND TRAINING CONFIGURATION · p. 34

    All generated training queries undergo rigorous validation via LLM judging to ensure they are answerable, have a single correct answer, and require search rather than relying solely on parametric knowledge.

    “All generated queries undergo rigorous validation via LLM judging to ensure they are (i) answerable, (ii) have a single correct answer, and (iii) require search rather than relying solely on parametric knowledge.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § H.2 OBTAINING THE CLUEWEB-TEST EVALUATION SET · p. 35

    A query is retained in ClueWeb-Test if it satisfies two criteria: the agent achieves pass@5 success, and the agent performs at least one search action during successful trajectories, confirming that the answer requires information retrieval rather than relying solely on parametric knowledge.

    “A query is retained in ClueWeb-Test if it satisfies two criteria: (i) the agent achieves pass@5 success, and (ii) the agent performs at least one search action during successful trajectories, confirming that the answer requires information retrieval rather than relying solely on parametric knowledge.”
    Quote in the paperSubstantial claim
68 of 68

  • The paper relies on this without support.

    The set of documents clicked by users in Researchy Questions search sessions serves as a valid proxy for ground-truth information targets for evaluating deep research reports.

    “Given that Researchy Questions are derived from real-world web search sessions, we leverage the set of documents clicked by users as a proxy for ground-truth information targets.”

  • The paper relies on this without support.

    The top 1,000 queries ranked by number of clicked documents align with the goals of deep research systems.

    “This ranking naturally favors queries that drive extensive exploration, aligning with the goals of deep research systems.”

  • The paper relies on this without support.

    ClueWeb22-B has higher alignment with the Researchy Questions benchmark than other corpora.

    “DEEPRESEARCHGYM's search API defaults to the ClueWeb22-B corpus given the higher alignment with the Researchy Questions benchmark.”

  • The paper relies on this without support.

    Evaluating systems in their default configurations is a fair comparison.

    “All systems are evaluated in their default configurations, and DEEPRESEARCHGYM's search API defaults to the ClueWeb22-B corpus given the higher alignment with the Researchy Questions benchmark.”

  • The paper relies on this without support.

    Short-form question answering maintains the core search behaviors of long-form report generation.

    “while maintaining the core search behaviors.”

  • The paper relies on this without support.

    ClueWeb22 and FineWeb are high-quality web corpora suitable for anchoring retrieval.

    “By anchoring retrieval to high-quality web corpora”

  • The paper relies on this without support.

    The curated subset of standard benchmarks covered by ClueWeb22 ensures both agents have access to relevant information.

    “ensuring both agents have access to relevant information.”

  • The paper relies on this without support.

    The representative sample of queries is representative of all queries in the log.

    “We first classified a representative sample of queries”

  • The paper relies on this without support.

    gpt-4.1-mini-2025-04-14 accurately classifies query intent and temporal sensitivity.

    “classified as as informational by gpt-4.1-mini-2025-04-14”
    “Using gpt-4.1-mini as a classifier”

  • The paper relies on this without support.

    LLM-as-a-judge soft-match rewards provide a valid training signal.

    “using LLM-as-a-judge soft-match rewards between agent responses and ground-truth answers.”

  • The paper relies on this without support.

    The static corpus provides all ground-truth evidence available up to the snapshot date.

    “The static setting still provides controlled access to all ground-truth evidence available up to the snapshot date”

  • The paper relies on this without support.

    The query log's 384 unique IP addresses across 13 countries are a valid measure of initial adoption.

    “To assess the initial adoption of this service over the first four months, we analyzed a query log comprising roughly 12 million queries submitted by 384 unique IP addresses across 13 countries.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing QA benchmarks, both general and deep research specific, focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR.

    “Typical QA benchmarks, both general [14, 17, 21, 29] and deep research specific [5, 13], focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [5]

Cites

Flagged first, then by citations

55 of the 106 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨ aschel, Sebastian Riedel, and Douwe Kiela. RetrievalAugmented Generation for Knowledge-Intensive NLP Tasks. In Annual Conference on Neural Information Processing Systems, NeurIPS , 2020.
  2. Suhas Jayaram Subramanya, Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnaswamy, and Rohan Kadekodi. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. In Annual Conference on Neural Information Processing Systems, NeurIPS , 2019.

Cited by

Sorted by citations

Authors
Coelho, J., Ning, J., He, J., Mao, K., Paladugu, A., Setlur, P., Jin, J., Callan, J., Magalhães, J., Martins, B., & Xiong, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.19253
arXiv
2505.19253
OpenAlex
W4414854692
Semantic Scholar
d44ff975b27dbfbf748ebb8fa815cb423c6fe53f
Cited by
1
References
106
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
benchmarking, LLM-as-a-judge, reporting quality, dense retrievers, approximate nearest neighbor search

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review