CoChat
Sign Up
PreprintCited by 10Information Retrieval and Search BehaviorTopic ModelingRecommender Systems and Techniques

BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

Open access
Chen, Zijian, Ma, Xueguang, Zhuang, Shengyao, Nie, Ping, Zou, Kai, Liu, Andrew, Green, Joshua, Patel, Kshama +12 more·2025·arXiv
DOI 10.48550/arxiv.2508.06600arXiv 2508.06600
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep-Research agents, which integrate large language models (LLMs) with search tools, have shown success in improving the effectiveness of handling complex queries that require iterative search planning and reasoning over search results. Evaluations on current benchmarks like BrowseComp relies on black-box live web search APIs, have notable limitations in (1) fairness: dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep research methods; (2) transparency: lack of control over the document corpus makes it difficult to isolate retriever contributions. In other words, the current evaluations may compare a complete deep research system at a given time, but they do not foster well-controlled experiments to provide insights into the capability of underlying deep research LLMs. To address these challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation. The benchmark is shown to be effective in distinguishing the performance of deep research systems. For instance, the open-source model Search-R1, when paired with the BM25 retriever, achieves 3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with fewer search calls. This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods, fostering insights into retrieval effectiveness, citation accuracy, and context engineering in Deep-Research system.

AI snapshot

Written from the paper’s full text.

TL;DR

BrowseComp-Plus is a new deep-research benchmark built from BrowseComp that replaces black-box live web search with a fixed, human-verified 100K-document corpus containing supporting evidence and hard negatives. It enables controlled, reproducible evaluation of retrieval and LLM agent components, showing that stronger retrievers substantially improve both accuracy and search efficiency.

Key findings

  • The benchmark contains 830 queries and 100,195 documents; on average each query has 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents, with documents averaging 5,179.2 words.
  • End-to-end accuracy varies widely: Search-R1 with BM25 reaches 3.86%, while GPT-5 with BM25 reaches 55.90% and GPT-5 with Qwen3-Embedding-8B reaches 70.12% with fewer search calls.
  • Stronger retrieval improves both effectiveness and efficiency: replacing BM25 with Qwen3-Embedding-8B raises GPT-5 accuracy from 55.90% to 70.12% and reduces search calls from 23.23 to 21.74; Sonnet 4 and Opus 4 more than double accuracy.
  • In an oracle setting where agents receive all labeled positive documents, GPT-4.1 reaches 93.49% accuracy, versus 14.58% with BM25, indicating large headroom tied to retrieval quality.

Methods

  • BrowseComp-Plus is derived from BrowseComp’s 1,266 question-answer pairs. Evidence documents were gathered using OpenAI o3 with web search, then human-verified by 14 university student annotators over 400+ hours; 830 queries passed verification.
  • Hard negatives were mined by prompting GPT-4o to decompose each question into about seven sub-queries, querying SerpAPI for up to 100 results per sub-query, scraping and deduplicating documents to form the final 100,195-document corpus.
  • Experiments pair commercial and open-source LLMs with BM25, Qwen3-Embedding (0.6B/4B/8B), and ReasonIR retrievers. Agents retrieve top k=5 results truncated to 512 tokens; metrics include accuracy, recall, search calls, calibration error, and retrieval Recall@k/nDCG@k.

Limitations

  • The paper states that the 100K corpus is smaller than web-scale; augmenting it with FineWeb-edu to 9,771,311 documents did not change overall conclusions, though neural retriever scores dropped on the larger corpus.
  • The paper states that retrieved documents are truncated to the first 512 tokens due to budget constraints; 86.5% of queries still contain the ground-truth answer in at least one gold document under this setting.
  • The paper states that some BrowseComp cases were removed, including 42 queries requiring Google Maps distances, 13 ambiguous or non-unique-answer queries, and flawed BrowseComp question-answer pairs; these exclusions limit coverage of dynamic or API-dependent tasks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.06600

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

1 of 1 cited sources checked · 1 supported · 0 flagged

  • Findingown data
    § 4.7 Effect of Retrieval Quality

    For most proprietary models, Qwen3-Embedding-8B reduces search calls by approximately 1-3 compared to BM25.

    “For most proprietary models, Qwen3-Embedding-8B reduces search calls by approximately 1-3 compared to BM25.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.3 Experiment Setup

    When documents are truncated to the first 512 tokens, 86.5% of queries still contain the ground-truth answer in at least one of their gold documents.

    “when documents are truncated to the first 512 tokens, 86.5% of queries still contain the ground-truth answer in at least one of their gold documents.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 4.8.2 Impact of Reasoning Effort

    Calibration error tends to decrease with higher reasoning effort.

    “Interestingly, calibration error tends to decrease with higher reasoning effort”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 4.7 Effect of Retrieval Quality

    Replacing the BM25 retriever with a stronger retriever leads to significant accuracy gains across all LLM agents.

    “Now, as indicated in Table 1, replacing the BM25 retriever with a stronger retriever leads to significant accuracy gains across all LLM agents.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction

    Even state-of-the-art retrieval models struggle to retrieve relevant documents for complex, reasoning-intensive queries on original BrowseComp queries.

    “even state-of-the-art retrieval models struggle to retrieve relevant documents for these complex, reasoning-intensive queries”
    Quote in the paperSubstantial claim
  • May combine two claims. 28
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 37

  • May combine two claims.

    Findingcitation
    § Abstract

    BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.

    “we introduce BrowseComp-Plus , a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] [1] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv:2504.12516 , 2025. URL https: //arxiv.org/abs/2504.12516 . - source

  • May combine two claims.

    Findingown data
    § 4.7 Effect of Retrieval Quality

    Within the Qwen3 embedding family, larger models consistently perform better, with Qwen3-8B surpassing ReasonIR-8B at the 8B scale.

    “Notably, we observe a model size scaling law within the Qwen3 embedding family; larger models consistently perform better, with Qwen3-8B surpassing ReasonIR-8B at the 8B scale.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.8.2 Impact of Reasoning Effort

    oss-20b with Qwen3-Embed-8B improves accuracy from 13.37% in low mode to 34.58% in high mode, accompanied by a recall jump from 17.37% to 49.29%.

    “For example, oss-20b with Qwen3-Embed-8B improves accuracy from 13.37% in low mode to 34.58% in high mode, accompanied by a recall jump from 17.37% to 49.29%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Effect of Retrieval Quality

    Compared to BM25, Qwen3-Embedding-8B and ReasonIR-8B achieve substantially higher recall and nDCG for both evidence document retrieval and gold document retrieval.

    “Compared to BM25, Qwen3Embedding-8B and ReasonIR-8B achieve substantially higher recall and nDCG for both evidence document retrieval and gold document retrieval.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Effect of Retrieval Quality

    Sonnet 4 and Opus 4 both more than double their accuracy when replacing BM25 with a stronger retriever.

    “For instance, OpenAI's gpt-5 accuracy improves from 55.9% to 70.12%, while Sonnet 4 and Opus 4 both more than double their accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.8.3 Effect of Document Reading Strategy

    For Qwen3-32B, accuracy improves slightly from 10.36% to 11.69%, and the number of get-document calls remains low (0.27 per query on average).

    “Accuracy improves slightly from 10.36% to 11.69%, and the number of get-document calls remains low (0.27 per query on average).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.4 Final Corpus Statistics

    The final corpus contains 100,195 documents and 830 queries.

    “we arrive at a corpus of 100,195 documents, along with 830 queries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Effect of Retrieval Quality

    Although agents using BM25 issue more search calls, nearly all citation metrics are lower than those achieved with Qwen3-Embedding-8B.

    “As the results show, although agents using BM25 issue more search calls, nearly all metrics are lower than those achieved with Qwen3-Embedding-8B.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § 1 Introduction

    Even when equipped with state-of-the-art retrievers, Deep-Research agents still face substantial challenges in consistently surfacing all necessary evidence for reasoning-intensive queries.

    “Deep-Research agents still face substantial challenges in consistently surfacing all necessary evidence, for reasoning-intensive queries.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.8.3 Effect of Document Reading Strategy

    For gpt-4.1, enabling the get-document tool improves answer accuracy from 35.42% to 43.61%, with a modest increase in search calls (from 8.67 to 10.03) and an average of 1.85 full-document reads per query.

    “For gpt-4.1, enabling the get-document tool improves answer accuracy from 35.42% to 43.61%, with a modest increase in search calls (from 8.67 to 10.03) and an average of 1.85 full-document reads per query.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.8.1 Oracle Retrieval

    For the remaining 6.51% of cases, human annotators confirmed that the answers are indeed answerable from the positive documents; the errors stem solely from gpt-4.1's failure to reason correctly.

    “For the remaining 6.51% of cases, human annotators reviewed each instance and confirmed that the answers are indeed answerable from the positive documents; the errors stem solely from gpt-4.1 's failure to reason correctly.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § Abstract

    Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives, enabling controlled experimentation.

    “Each query in BrowseComp-Plus includes human-verified supporting documents and mined challenging negatives”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.4 Final Corpus Statistics

    On average, each query contains 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents.

    “each query contains 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.6 End-to-End Deep-Research Performance

    Proprietary models demonstrate high answer accuracy, with gpt-5 achieving the highest accuracy (70.12%) when paired with the Qwen3-Embedding-8B retriever.

    “gpt-5 achieving the highest accuracy (70.12%) when paired with the Qwen3-Embedding-8B retriever.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Pro with BM25 costs $138.64.

    “| Gemini 2.5 Pro | BM25 | 19.04% | $138.64 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Flash with Qwen3-Embed-8B costs $41.29.

    “| Gemini 2.5 Flash | Qwen3-Embed-8B | 33.01% | $41.29 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Flash with BM25 achieves 15.54% accuracy.

    “| Gemini 2.5 Flash | BM25 | 15.54% | $47.32 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Flash with Qwen3-Embed-8B achieves 33.01% accuracy.

    “| Gemini 2.5 Flash | Qwen3-Embed-8B | 33.01% | $41.29 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    gpt-4.1 with BM25 achieves 14.58% accuracy.

    “| gpt-4.1 | BM25 | 14.58% | $106.96 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Flash with BM25 costs $47.32.

    “| Gemini 2.5 Flash | BM25 | 15.54% | $47.32 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Pro with BM25 achieves 19.04% accuracy.

    “| Gemini 2.5 Pro | BM25 | 19.04% | $138.64 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    o3 with Qwen3-Embed-8B costs $740.79.

    “| o3 | Qwen3-Embed-8B | 63.49% | $740.79 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Pro with Qwen3-Embed-8B achieves 28.67% accuracy.

    “| Gemini 2.5 Pro | Qwen3-Embed-8B | 28.67% | $99.92 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Opus4 with BM25 achieves 15.54% accuracy.

    “| Opus4 | BM25 | 15.54% | $2,043.95 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Sonnet 4 with Qwen3-Embed-8B costs $325.75.

    “| Sonnet 4 | Qwen3-Embed-8B | 36.75% | $325.75 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    GPT-5 with Qwen3-Embed-8B costs $360.71.

    “| GPT-5 | Qwen3-Embed-8B | 70.12% | $360.71 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Sonnet 4 with BM25 costs $352.04.

    “| Sonnet 4 | BM25 | 14.34% | $352.04 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § C Problematic Cases

    The ground-truth answer for that question is correct, but it is only one of three usernames credited as authors.

    “While the ground-truth answer is correct, it is only one of three usernames credited as authors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    o3 with BM25 costs $836.35.

    “| o3 | BM25 | 49.28% | $836.35 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Opus4 with BM25 costs $2,043.95.

    “| Opus4 | BM25 | 15.54% | $2,043.95 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Gemini 2.5 Pro with Qwen3-Embed-8B costs $99.92.

    “| Gemini 2.5 Pro | Qwen3-Embed-8B | 28.67% | $99.92 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Sonnet 4 with BM25 achieves 14.34% accuracy.

    “| Sonnet 4 | BM25 | 14.34% | $352.04 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Sonnet 4 with Qwen3-Embed-8B achieves 36.75% accuracy.

    “| Sonnet 4 | Qwen3-Embed-8B | 36.75% | $325.75 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Opus4 with Qwen3-Embed-8B achieves 36.14% accuracy.

    “| Opus4 | Qwen3-Embed-8B | 36.14% | $1,842.48 |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6 Conclusion

    Stronger retrievers improve final answer accuracy and reduce the number of search iterations required.

    “Stronger retrievers not only improve final answer accuracy but also reduce the number of search iterations required”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § C Problematic Cases

    One question asks for the username of an individual who authored a specific story on an internet forum.

    “For instance, one question asks for the username of an individual who authored a specific story on an internet forum.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    gpt-4.1 with BM25 costs $106.96.

    “| gpt-4.1 | BM25 | 14.58% | $106.96 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    Opus4 with Qwen3-Embed-8B costs $1,842.48.

    “| Opus4 | Qwen3-Embed-8B | 36.14% | $1,842.48 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    GPT-5 with BM25 costs $400.36.

    “| GPT-5 | BM25 | 55.9% | $400.36 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § F Evaluation Prompt

    gpt-4.1 with Qwen3-Embed-8B costs $89.81.

    “| gpt-4.1 | Qwen3-Embed-8B | 35.42% | $89.81 |”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning
    § C Problematic Cases

    We remove 13 such queries due to this kind of ambiguity.

    “We remove 13 such queries due to this kind of ambiguity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.8.4 Effect of Corpus Size

    The drop in neural retriever performance on Fineweb is theoretically expected because the relative ranking of documents from the original small corpus remains unchanged, but newly added Fineweb documents can appear in the top ranks.

    “This drop is theoretically expected: the relative ranking of documents from the original small corpus remains unchanged, but the newly added Fineweb documents can now appear in the top ranks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The relative ranking of documents from the original small corpus remains unchanged when the corpus is augmented with Fineweb documents.

  • May combine two claims.

    Interpretationown data
    § 4.8.4 Effect of Corpus Size

    The original 100K corpus offers both strong positive coverage and sufficient challenge for robust evaluation.

    “supporting our claim that the original 100K corpus offers both strong positive coverage and sufficient challenge for robust evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.8.2 Impact of Reasoning Effort

    Increasing reasoning effort can significantly boost answer quality, but at the cost of retrieval overhead.

    “These findings indicate that increasing reasoning effort can significantly boost answer quality, but at the cost of retrieval overhead”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 3.2.2 Evidence Document Verification

    Human annotators verified that each clue is sufficiently justified by supporting documents and that the combination of clues and evidence enables answering the question correctly.

    “Annotators are asked to: 1. Confirm that each clue is sufficiently justified by the supporting documents. Instead of simply confirming the match, annotators must label the text spans in the documents that justify each clue, as this explicit step encourages high-quality verification. 2. Determine whether the combination of clues and supporting evidence enables a human to answer the entirety of the”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.8.3 Effect of Document Reading Strategy

    Context engineering is valuable in optimizing how retrieval results are presented to the LLM agent.

    “This also highlights the value of context engineering in optimizing how retrieval results are presented to the LLM agent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 4.7 Effect of Retrieval Quality

    Better retrieval improves both effectiveness (accuracy) and efficiency (fewer tool calls).

    “This shows that better retrieval not only improves effectiveness (accuracy) but also efficiency (fewer tool calls).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.7 Effect of Retrieval Quality

    Documents returned by BM25 are less useful in the iterative deep research process, whereas Qwen3-Embedding-8B provides more relevant and informative documents.

    “This indicates that documents returned by BM25 are less useful in the iterative deep research process, whereas Qwen3-Embedding-8B provides more relevant and informative documents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 4.8.3 Effect of Document Reading Strategy

    Model quality is important in effective tool integration.

    “highlighting once again the importance of model quality in effective tool integration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 4.8.4 Effect of Corpus Size

    Lower retrieval scores for embedding models on Fineweb do not necessarily indicate worse final answers; some unjudged, top-ranked Fineweb documents may be 'false negatives' that still provide useful evidence.

    “It is important to note that lower retrieval scores for embedding models on Fineweb do not necessarily indicate worse final answers, some unjudged, top-ranked Fineweb documents may be 'false negatives' that still provide useful evidence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Abstract

    BrowseComp-Plus allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.

    “This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.6 End-to-End Deep-Research Performance

    More exhaustive search correlates with better outcomes, reflecting a test-time scaling effect.

    “more exhaustive search correlates with better outcomes”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.6 End-to-End Deep-Research Performance

    Open-source LLMs have current limitations in interleaved reasoning and tool-use capabilities, despite comparable performance when directly given relevant documents.

    “These results illustrate current limitations in the interleaved reasoning and tool-use capabilities of open-source LLMs, despite their comparable performance when directly given relevant documents (as shown in Section 4.8.1).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.8.4 Effect of Corpus Size

    The improvement in BM25 retrieval effectiveness is likely due to better inverse document frequency (IDF) estimation in the larger corpus, which strengthens BM25's lexical scoring.

    “likely due to better inverse document frequency (IDF) estimation in the larger corpus, which strengthens BM25's lexical scoring.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6 Conclusion

    Retrieval quality substantially impacts both the effectiveness and efficiency of deep research systems.

    “Through extensive experiments pairing diverse retrievers with both open- and closed-source agents, we demonstrate that retrieval quality substantially impacts both the effectiveness and efficiency of deep research systems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning
    § 6 Conclusion

    BrowseComp-Plus is a new benchmark designed to address the reproducibility, fairness, and transparency challenges in evaluating Deep-Research Agents.

    “Weintroduced BrowseComp-Plus , a new benchmark designed to address the reproducibility, fairness, and transparency challenges in evaluating Deep-Research Agents.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 5 Future Work and Discussion

    An oracle retriever capable of surfacing gold or highly relevant documents can greatly improve accuracy.

    “Finally, as shown in this work, an oracle retriever capable of surfacing gold or highly relevant documents can greatly improve accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 6 Conclusion

    BrowseComp-Plus provides a robust platform for probing these dynamics.

    “BrowseComp-Plus provides a robust platform for probing these dynamics”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § C Problematic Cases

    Scraping static snapshots of Google Maps pages to include in the corpus is not a valid substitute for answering such questions as intended.

    “Moreover, scraping static snapshots of Google Maps pages to include in the corpus is not a valid substitute; answering such questions as intended should require agents to be augmented with access to the Google Maps API, rather retrieving from a corpus.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § 5 Future Work and Discussion

    Such retrievers may also reduce the number of search iterations required, improving the overall efficiency of the research process.

    “Such retrievers may also reduce the number of search iterations required, improving the overall efficiency of the research process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 5 Future Work and Discussion

    BrowseComp-Plus serves as an ideal testbed for pursuing these directions, enabling systematic and fine-grained analyses of agent-retriever interactions within the Deep-Research paradigm.

    “Overall, BrowseComp-Plus serves as an ideal testbed for pursuing these directions, enabling systematic and fine-grained analyses of agent-retriever interactions within the Deep-Research paradigm.”
    Quote in the paperNot a substantial claim
61 of 61

  • The paper relies on this without support.

    The LLM-as-judge (gpt-4.1) used for accuracy evaluation reliably matches human judgment of answer correctness.

    “Accuracy follows BrowseComp: an LLM-as-judge (gpt-4.1) compares the model's final answer against the ground truth using the evaluation prompt listed in Appendix F.”

  • The paper relies on this without support.

    Truncating retrieved documents to the first 512 tokens does not materially change the relative performance of deep-research agents.

    “The retriever tool is set to retrieve the top k = 5 search results, where each result is truncated to the first 512 token of the corresponding document.”

  • The paper relies on this without support.

    The mined hard negative documents are sufficiently challenging to test retrieval and agent systems.

    “we mine hard negative documents via web search to form the corpus.”

  • The paper relies on this without support.

    The relative ranking of documents from the original small corpus remains unchanged when the corpus is augmented with Fineweb documents.

    “the relative ranking of documents from the original small corpus remains unchanged, but the newly added Fineweb documents can now appear in the top ranks.”

  • The paper relies on this without support.

    High-quality documents discussing specific Google Maps distances between arbitrary locations are difficult to obtain.

    “These are removed because high-quality documents discussing specific Google Maps distances between arbitrary locations are difficult to obtain.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The task data are drawn from GAIA, OfficeQA Pro, BrowseComp-Plus, and Terminal-Bench 2.0.

    “Task data come from GAIA [16], OfficeQA Pro [18], BrowseComp-Plus [6], and Terminal-Bench 2.0 [15].”

    From HarnessOpt-Bench: Evaluating LLMs at Harness Optimization (2026) · cites this paper as [6]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Expanded corpus is about 98 times, not 10 times, the original

    Minor

    The reported expanded corpus has 9,771,311 documents versus 100,195 in the original corpus. That is about 97.5 times as many documents, not roughly 10 times.

    Reported 10 · recomputed 97.52294

  • BM25 original-corpus Recall@1000 differs between tables

    Minor

    Table 2 reports BM25 evidence-document Recall@1000 as 13.7, while Table 6 reports 13.6% for BM25 on the original corpus. Both values are printed to one decimal place.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 7

  1. [2][2] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. -
  2. [3][3] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=hSyW5go0v8 . -
  3. [7][7] Gemini 2.5 Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/ 2507.06261 . -
  4. [17][17] Stephen E. Robertson. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference (TREC-3) . NIST, 1994. -
  5. [31][31] Adrien Barbaresi. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations , pages 122-131. Association for Computational Linguistics, 2021. URL https://aclanthology.org/2021.acl-demo.15 . -
  6. [35][35] Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-2. In Donna K. Harman, editor, Proceedings of The Second Text REtrieval Conference, TREC 1993, Gaithersburg, Maryland, USA, August 31 - September 2, 1993 , volume 500-215 of NIST Special Publication , pages 21-34. National Institute of Standards and Technology (NIST), 1993. URL http://trec.nist.gov/pubs/trec2/papers/ps/ city.ps . -
  7. [37][37] Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT: A realistic and challenging benchmark for reasoningintensive retrieval. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=ykuc5q381b . -

Cited by

Sorted by citations

Authors
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Liu, A., Green, J., Patel, K., Meng, R., Su, M., Sharifymoghaddam, S., Li, Y., Hong, H., Shi, X., Liu, X., Thakur, N., Zhang, C., Gao, L., Chen, W., & Lin, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2508.06600
arXiv
2508.06600
OpenAlex
W4416176141
Semantic Scholar
a1b1590452727b80805cf9c90d4a181856267794
Cited by
3
References
0
Citations by year
2026: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
deep research agents, benchmark evaluation, large language models, web search agents, reproducibility, search planning, citation accuracy, context engineering, GPT-5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review