BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents

May combine two claims.

BrowseComp-Plus extends the original BrowseComp benchmark with a fixed, human-verified document corpus, where each query is paired with explicitly identified supporting documents and hard negatives.

“we introduce BrowseComp-Plus , a new benchmark that extends the original BrowseComp benchmark (Wei et al., 2025) with a fixed, human-verified document corpus, where each query is paired with explicitly identified supporting documents and hard negatives.”

• cites Wei et al., 2025 sourceSupported

May combine two claims.

Search-time data contamination can leak answers; e.g., Humanity's Last Exam leaked 3% of their answers on HuggingFace alone.

“This problem is exacerbated by search-time data contamination, where challenging benchmarks such as Humanity's Last Exam (Phan et al., 2025) leaked 3% of their answers on HuggingFace alone (Han et al., 2025).”

• cites Han et al., 2025 sourceSupported
• cites Phan et al., 2025 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BrowseComp stands out as a benchmark explicitly designed for this purpose, offering complex queries paired with verifiable answers.

“BrowseComp (Wei et al., 2025) stands out as a benchmark explicitly designed for this purpose, offering complex queries paired with verifiable answers.”

• cites Wei et al., 2025 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Deep search agents conduct tasks through iterative query reasoning, search planning, and reflection on retrieved results, outperforming the traditional single-round retrieval-augmented generation paradigm.

“Deep search agents conduct tasks through iterative query reasoning, search planning, and reflection on retrieved results (Asai et al., 2024), outperforming the traditional single-round retrieval-augmented generation paradigm (Lewis et al., 2020).”

• cites Asai et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lewis et al., 2020 sourcePartly supported

  Section 2 (p. 2) describes Lewis et al.’s RAG as retrieving documents from the input and using them to generate an answer, supporting its identification as the retrieval-augmented generation approach used for comparison. The paper does not evaluate deep search agents or show that iterative reasoning, planning, and reflection outperform RAG; its reported comparisons are with other models and retrieval-based systems (Introduction, p. 2; §4.1, p. 5). (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

The BrowseComp benchmark contains 1,266 challenging fact-seeking questions specifically designed to assess the capability of deep search agents to interactively and creatively navigate the web for complex, hard-to-find information.

“The BrowseComp benchmark contains 1,266 challenging fact-seeking questions specifically designed to assess the capability of deep search agents to interactively and creatively navigate the web for complex, hard-to-find information (Wei et al., 2025).”

• cites Wei et al., 2025 sourceSupported

May combine two claims.

HotpotQA offers multi-hop questions, but they are shallow in depth (2 hops), and the corpus is limited to Wikipedia.

“Although datasets like HotpotQA (Yang et al., 2018) offer multi-hop questions, they are shallow in depth (2 hops), and the corpus is limited to Wikipedia.”

• cites Yang et al., 2018 sourcePartly supported

  The Introduction says HotpotQA contains multi-hop questions, and Data Collection says its corpus is the entire English Wikipedia dump. The Generating Candidate Paragraph Pairs section illustrates two-step reasoning, but Dataset Analysis says some questions require more than two supporting facts; the paper does not establish a strict two-hop limit or characterize the questions as shallow. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Hard negative mining via web search has proven effective in evaluating information retrieval systems using a sub-sampled corpus.

“This has proven to be effective in evaluating information retrieval systems using a sub-sampled corpus (Fröbe et al., 2025; Zhuang and Zuccon, 2022).”

• cites Fröbe et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhuang and Zuccon, 2022 sourcePartly supported

  The Demonstration Use Case (p. 4) evaluates dense-retriever checkpoints on sub-sampled corpora, and reports that their validation trends match those from the full corpus. The paper does not attribute that result to hard-negative mining via web search; its subsets are constructed from BM25 and TCTColBERTv2 retrieval runs. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Traditional benchmarks such as NaturalQuestions and TriviaQA primarily feature single-hop questions, which typically do not require multi-step reasoning or iterative retrieval.

“However, these benchmarks primarily feature single-hop questions, which typically do not require multi-step reasoning or iterative retrieval.”

• cites Kwiatkowski et al., 2019 sourceNot checked — no full text of the source
• cites Joshi et al., 2017 sourcePartly supported

  The Dataset Analysis (§4; page not supplied) finds that 40% of sampled TriviaQA question–evidence pairs required information scattered across multiple sentences, consistent with a majority not requiring that kind of reasoning. But the paper does not classify TriviaQA questions as single-hop or assess iterative retrieval, and it does not address NaturalQuestions. A single-sentence question (§7.2) is not necessarily a single-hop question. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The questions in BrowseComp are deliberately constructed to be difficult for both humans and LLMs, yet they feature verifiable, concise answers, enabling straightforward evaluation through simple answer matching.

“The questions are deliberately constructed to be difficult for both humans and LLMs, yet they feature verifiable, concise answers, enabling straightforward evaluation through simple answer matching.”

• cites Wei et al., 2025 sourceSupported

Classified as probably not a substantial claim. May combine two claims.

BrowseComp-Plus retains the challenging nature of the original BrowseComp while providing a more controlled and transparent experimental setup.

“BrowseComp-Plus retains the challenging nature of the original BrowseComp while providing a more controlled and transparent experimental setup similar to early pivotal evaluation benchmarks like Natural Questions (NQ) (Kwiatkowski et al., 2019) and HotpotQA (Yang et al., 2018).”

• cites Kwiatkowski et al., 2019 sourceNot checked — no full text of the source
• cites Yang et al., 2018 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-5 achieves 59.9% accuracy when evaluated using the Google Search API.

“For reference, GPT-5 achieves 59.9% accuracy when evaluated using the Google Search API.”

The quote only partly states this claim.

Pairing GPT-5 with Qwen3-Embedding8B retriever reduces search calls compared to using BM25.

“while reducing search calls.”

May combine two claims.

BrowseComp-Plus is a benchmark derived from BrowseComp that employs a fixed, human-verified corpus, enabling controlled retrieval for deep search agents.

“We introduce BrowseComp-Plus , a benchmark derived from BrowseComp that employs a fixed, human-verified corpus, enabling controlled retrieval for deep search agents.”

May combine two claims.

Better retrieval systems not only improve the overall accuracy but also reduce the number of search calls.

“whereas better retrieval systems not only improve the overall accuracy but also reduce the number of search calls”

May combine two claims.

After deduplicating positive and negative documents, the corpus contains 100,195 documents and 830 queries.

“After deduplicating the positive and negative documents collected, we arrive at a corpus of 100,195 documents, along with 830 queries.”

May combine two claims.

Stronger retrievers improve both the effectiveness and efficiency of search agents.

“while stronger retrievers improve both the effectiveness and efficiency of search agents.”

May combine two claims.

With the Qwen3-Embedding-8B retriever, Qwen3-32B achieves 10.36% accuracy, GPT-4.1 achieves 35.42%, and GPT-5 achieves 70.12%.

“With the Qwen3-Embedding-8B retriever, Qwen332B achieves only 10.36% accuracy, compared to 35.42% for GPT-4.1 and 70.12% for GPT-5.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The entire labeling process involved 14 university student annotators and required over 400 hours of manual effort.

“The entire labeling process involved 14 university student annotators and required over 400 hours of manual effort.”

May combine two claims.

GPT-5 and o3 issue an average of more than 20 search calls per query, while Qwen3-32B and SearchR1-32B make fewer than 2, despite being prompted to use the tool.

“For instance, OpenAI's GPT-5 and o3 issue an average of more than 20 search calls per query, while Qwen3-32B and SearchR1-32B make fewer than 2, despite being explicitly prompted to use the tool.”

May combine two claims.

The most common failure mode occurs when the documents provided by OpenAI o3 do not satisfy the two verification criteria, and human annotators are unable to gather sufficient additional evidence within a reasonable effort.

“The most common failure mode occurs when the documents provided by OpenAI o3 do not satisfy the two verification criteria, and human annotators are unable to gather sufficient additional evidence within a reasonable effort.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On average, prompting GPT-4o to break each question into sub-queries results in about seven sub-queries per original query.

“On average, this results in about seven sub-queries per original query.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This leaves 1,005 queries for human verification.

“This leaves us with 1,005 queries for the next stage: human verification.”

May combine two claims.

The paper curates evidence documents through a two-stage pipeline and presents a 100k-document corpus that effectively supports the study of deep search agents.

“we curate evidence documents through a two-stage pipeline, involving automated evidence mining followed by human verification, and mine hard-negatives via web search to attach distracting documents to each query. The sections below describe this process in detail, and present a 100k-document corpus that effectively supports the study of deep search agents.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each document averages 5179.2 words and 32296.2 characters.

“Each document averages 5179.2 words and 32296.2 characters.”

May combine two claims.

On average, each query contains 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents.

“On average, each query contains 6.1 evidence documents, 76.28 negatives, and 2.9 gold documents.”

May combine two claims.

gpt-oss-120B in high reasoning mode achieves 42.89% accuracy, surpassing Opus 4 when both are paired with Qwen3-Embedding-8B.

“Notably, the only high-performing open-source model we studied is gpt-oss-120B in its high reasoning mode with 42.89% accuracy, surpassing Opus 4 when both are paired with Qwen3-Embedding-8B.”

May combine two claims.

For gpt-oss-20B with Qwen3Embed-8B, accuracy improves from 13.37% in low mode to 34.58% in high mode, and recall jumps from 17.37% to 49.29%.

“For example, gpt-oss-20B with Qwen3Embed-8B improves from 13.37% accuracy in low mode to 34.58% in high mode, along with a recall jump from 17.37% to 49.29%.”

May combine two claims.

For both GPT-4.1 and gpt-oss-20B (high reasoning effort), integrating reranking improves Accuracy and Recall substantially.

“For both models, Accuracy and Recall improve substantially.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Calibration error tends to decrease with higher reasoning effort.

“Interestingly, calibration error tends to decrease with higher reasoning effort, suggesting that the models give better confidence estimates as they reason more extensively.”

May combine two claims.

Expanding the reranking candidate set to 100 improves all models, with larger gains for the 32B models, widening the effectiveness gap between 8B and 32B models at higher rerank depths.

“Expanding the reranking candidate set to 100 improves all models, with larger gains for the 32B models, widening the effectiveness gap between 8B and 32B models at higher rerank depths.”

Probably not a substantial claim: may not pass S3 (load-bearing).

13 queries were removed due to ambiguity in expected answer format or multiple valid answers.

“We remove 13 such queries due to this kind of ambiguity.”

May combine two claims.

Relative to BM25, Qwen3-Embedding-8B and ReasonIR-8B achieve substantially higher recall and nDCG for both evidence document retrieval and gold document retrieval.

“Relative to BM25, Qwen3-Embedding8B and ReasonIR-8B achieve substantially higher recall and nDCG for both evidence document retrieval and gold document retrieval.”

May combine two claims.

After the labeling process, 10 examples from each labeler were randomly sampled and discussed in the group channel, showing over 80% agreement.

“After the labeling process, 10 examples from each labeler were randomly sampled and discussed in the group channel, showing over 80% agreement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The corpus in BrowseComp-Plus contains approximately 100K documents.

“The corpus in BrowseComp-Plus contains approximately 100K documents.”

Probably not a substantial claim: may not pass S2 (checkable).

Evaluating different retrievers under a fixed agent setup is equally important.

“Equally important is the converse : evaluating different retrievers under a fixed agent setup.”

Probably not a substantial claim: may not pass S4 (based).

The approach of using BrowseComp with web search access forbids disentangled measurement of retrieval effectiveness within deep search.

“While widely employed for end-to-end evaluation of deep search agents with web search access, this approach forbids disentangled measurement of retrieval effectiveness within deep search.”

May combine two claims.

BrowseComp-Plus enables fair evaluation of deep search agents under a fixed retriever and corpus, while disentangling retriever and agent contributions.

“This enables fair evaluation of deep search agents under a fixed retriever and corpus, while disentangling retriever and agent contributions.”

Classified as probably not a substantial claim.

The paper reveals new findings specific to the emerging multi-turn, agentic search.

“and we reveal new findings specific to the emerging multi-turn, agentic search.”

Probably not a substantial claim: may not pass S2 (checkable).

To robustly evaluate deep search agents capable of complex reasoning and iteratively retrieve many turns, benchmarks requiring deep multi-turn query interactions are essential.

“To robustly evaluate deep search agents capable of complex reasoning and iteratively retrieve many turns, benchmarks requiring deep multi-turn query interactions are essential.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Neural retrieval methods have significantly improved retrieval effectiveness compared to traditional lexical-based methods like BM25.

“These methods have significantly improved retrieval effectiveness compared to traditional lexical-based methods like BM25 (Robertson et al., 1994).”

• cites Robertson et al., 1994Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Fair evaluation of deep search agents requires a fixed retriever system to make comparisons meaningful.

“Fair evaluation of such agents, however, requires a fixed retriever system to make comparisons meaningful.”

Probably not a substantial claim: may not pass S3 (load-bearing).

BrowseComp-Plus enhances accessibility by replacing costly web search APIs with inexpensive local retrieval.

“and enhances accessibility by replacing costly web search APIs with inexpensive local retrieval.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Existing benchmarks primarily focus on question-answer evaluations of search agent systems without fixed corpora, complicating comparative assessments of retrieval methodologies.

“Existing benchmarks primarily focus on question-answer evaluations of search agent systems without fixed corpora, complicating comparative assessments of retrieval methodologies.”

Classified as probably not a substantial claim. May combine two claims.

Constructing a corpus for BrowseComp questions is non-trivial, presenting three key challenges: comprehensive coverage, retrieval difficulty, and practical size.

“Constructing a corpus for BrowseComp questions is non-trivial, presenting three key challenges:”

• AssumptionRests on an assumption: Constructing a corpus for BrowseComp questions is non-trivial.
• AssumptionRests on an assumption: The corpus must contain complete evidence to support the entire reasoning chain to answer each question.
• AssumptionRests on an assumption: The corpus should contain enough distracting negative documents so that search agents and retrievers are challenged in locating the correct evidence.
• AssumptionRests on an assumption: The corpus should be large enough to yield reliable research insights, while avoiding overly large computation costs for research purposes.

Classified as probably not a substantial claim.

By releasing a shared benchmark, BrowseComp-Plus places agents and retrievers on the same playing field.

“By releasing a shared benchmark, we place agents and retrievers on the same playing field.”

May combine two claims.

BrowseComp-Plus facilitates disentangled evaluations by providing a human-verified corpus, expanding the classic Cranfield paradigm to modern deep search evaluation.

“BrowseComp-Plus facilitates disentangled evaluations by providing a human-verified corpus, expanding the classic Cranfield paradigm (Voorhees, 2002) to modern deep search evaluation.”

• cites Voorhees, 2002 sourceNot checked — no full text of the source

May combine two claims.

Better retrieval improves the same agent's accuracy and reduces token costs.

“better retrieval not only improves the same agent's accuracy and reduces token costs”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

High-quality citations improve verifiability and trustworthiness for human users beyond accuracy alone.

“high-quality citations improve verifiability and trustworthiness for human users beyond accuracy alone.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The initial evidence-gathering step uses an OpenAI model (o3) to propose candidate URLs, which may introduce bias toward distributions that are more easily surfaced by that model.

“Second, the initial evidence-gathering step uses an OpenAI model (o3) to propose candidate URLs, which may introduce bias toward distributions that are more easily surfaced by that model”

May combine two claims.

Better retrieval improves both effectiveness and efficiency.

“That is, better retrieval not only improves effectiveness, but also efficiency.”

May combine two claims.

The corpus is constructed through careful human verification, but the absence of false negatives cannot be guaranteed.

“although the corpus is constructed through careful human verification, we cannot guarantee the absence of false negatives, where documents contain relevant information but are not labeled as evidence.”

• cites Fröbe et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

BrowseComp-Plus is a benchmark to address the fairness and disentanglement challenges in evaluating deep search agents.

“We introduce BrowseComp-Plus , a benchmark to address the fairness and disentanglement challenges in evaluating deep search agents.”

• AssumptionRests on an assumption: Prior works that rely on opaque web APIs obscure the role of search in search agents.

Probably not a substantial claim: may not pass S3 (load-bearing).

The parametric-only performance of o3 and GPT-5 may suggest that these models were trained on BrowseComp.

“this may suggest that these models were trained on BrowseComp.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In this work, evaluation is based on short, conclusive answers and cited documents within long-form responses.

“Finally, in this work we focus on evaluation based on short, conclusive answers and cited documents within long-form responses.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

BrowseComp-Plus serves as an ideal testbed for pursuing these directions, enabling systematic and fine-grained analyses of agent-retriever interactions for deep search.

“Overall, BrowseComp-Plus serves as an ideal testbed for pursuing these directions, enabling systematic and fine-grained analyses of agent-retriever interactions for deep search.”

May combine two claims.

Retrievers substantially benefit both the effectiveness and efficiency of deep search agents.

“demonstrate that they substantially benefit both the effectiveness and efficiency of deep search agents.”

May combine two claims.

The oracle result validates the quality of the BrowseComp-Plus corpus: GPT-4.1, a non-reasoning model, correctly answers 93.49% of questions using only the evidence documents in the corpus.

“Second, it validates the quality of the BrowseComp-Plus corpus itself: GPT-4.1, a non-reasoning model, is able to correctly answer 93.49% of questions using only the evidence documents in the corpus.”

May combine two claims.

BrowseComp-Plus improves evaluation rigor and reproducibility for agents.

“For agents, we improve evaluation rigor and reproducibility.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BrowseComp-Plus primarily evaluates textual evidence and does not fully capture the diversity of real-world web content, such as interactive pages, dynamic layouts, multimedia, or unparsed PDFs.

“Third, BrowseComp-Plus primarily evaluates textual evidence and does not fully capture the diversity of real-world web content, such as interactive pages, dynamic layouts, multimedia, or unparsed PDFs.”

Classified as probably not a substantial claim.

Better retrieval opens the option of pairing a cheaper agent with better retrieval to match orders-of-magnitude larger models.

“but also opens the option of pairing a cheaper agent with better retrieval to match orders-of-magnitude larger models”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each annotator underwent approximately 1 hour of training on the labeling task on an internal development set prior to the real annotation on BrowseComp-Plus.

“Each annotator underwent approximately 1 hour of training on the labeling task on an internal development set prior to the real annotation on BrowseComp-Plus .”

Classified as probably not a substantial claim.

Pairwise McNemar's tests at p ≤ 0.05 were conducted on answer accuracy.

“Pairwise McNemar's tests at p ≤ 0 . 05 were conducted, where a green cell at Row (i), Column (j) indicates that the method in Row (i) performs significantly better than the method in Column (j).”

May combine two claims.

Reranking improves the precision and recall of retrieved evidence at higher ranks, helping the agent surface more relevant information.

“This further indicates that reranking improves the precision and recall of retrieved evidence at higher ranks, helping the agent surface more relevant information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The improvement in BM25 retrieval effectiveness is likely due to better IDF estimation in the larger corpus, which strengthens BM25's lexical scoring.

“For BM25, retrieval effectiveness improves across all metrics, likely due to better inverse document frequency (IDF) estimation in the larger corpus, which strengthens BM25's lexical scoring.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Lower retrieval scores for embedding models on FineWeb do not necessarily indicate worse final answers; some unjudged, top-ranked FineWeb documents may be 'false negatives' that still provide useful evidence.

“It is important to note that lower retrieval scores for embedding models on FineWeb do not necessarily indicate worse final answers; some unjudged, top-ranked FineWeb documents may be 'false negatives' that still provide useful evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For cases where a labeler made a mistake, the labeler was instructed to relabel all of their prior examples.

“For the cases where a labeler made a mistake, the labeler was instructed to relabel all of their prior examples, avoiding similar mistakes.”

May combine two claims.

The drop in neural retriever effectiveness on FineWeb is theoretically expected because the relative ranking of documents from the original small corpus remains unchanged, but newly added FineWeb documents can appear in top ranks and are treated as non-relevant under standard TREC-style evaluation.

“This drop is theoretically expected: the relative ranking of documents from the original small corpus remains unchanged, but the newly added FineWeb documents can now appear in the top ranks. Since these additional documents are unjudged, they are treated as non-relevant under standard TREC-style evaluation, inevitably lowering measured retrieval effectiveness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A 40-minute demonstration video covering many edge cases was created for annotators to refer to.

“Additionally, besides detailed text instructions, we created a 40minute long demonstration video, covering many edge cases, which the annotators could constantly refer to.”

May combine two claims.

The original 100K corpus offers both strong positive coverage and sufficient challenge for robust evaluation.

“Overall, substantially expanding the corpus size does not lead to different conclusions about the ranking or effectiveness level among the retrievers and agents, supporting our claim that the original 100K corpus offers both strong positive coverage and sufficient challenge for robust evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Unsure cases were consolidated in a group channel, possibly relabeling prior cases for consistency.

“Further unsure cases were consolidated in a group channel, possibly relabeling prior cases for consistency.”