DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents

May combine two claims.

DeepSearchQA explicitly tests three capabilities: systematic collation of fragmented information from disparate sources, de-duplication and entity resolution to ensure precision, and reasoning about stopping criteria within an open-ended search space.

“This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space.”

May combine two claims.

Gemini Deep Research agent demonstrates strong performance scaling with accuracy increasing monotonically as more test-time compute (samples) is applied.

“When evaluated on this diverse set, the Gemini Deep Research agent demonstrates strong performance scaling (Right), with accuracy increasing monotonically as more test-time compute (samples) is applied.”

May combine two claims.

The benchmark features a balanced distribution of prompts across diverse topics, preventing domain overfitting.

“The benchmark features a balanced distribution of prompts across diverse topics (Left), preventing domain overfitting.”

May combine two claims.

The results highlight the scaling behavior of agentic capability and the persistent difficulty of the Comprehensiveness Gap.

“The results highlight the scaling behavior of agentic capability and the persistent difficulty of the Comprehensiveness Gap .”

May combine two claims.

Failure to resolve entities leads to inflated lists and degraded precision, a common failure mode observed in experiments.

“In deep research context, failure to resolve entities leads to inflated lists and degraded precision, a common failure mode we observe in our experiments.”

May combine two claims.

Agents exhibit distinct failure modes including premature stopping (under-retrieval) and hedging behaviors where agents cast an overly wide net of low-confidence answers to artificially boost recall.

“We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall.”

Classified as probably not a substantial claim.

Each DeepSearchQA task is structured as a causal chain where discovering information for one step depends on successful completion of the previous one.

“Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention.”

May combine two claims.

Gemini Deep Research Agent and GPT-5 Pro High Reasoning establish the state-of-the-art on DeepSearchQA.

“Gemini Deep Research Agent and GPT-5 Pro High Reasoning (OpenAI, 2025b) establish the state-of-the-art on this benchmark”

• cites OpenAI, 2025bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Gemini 2.5 Flash drops to an F1 of 42.99%, roughly half that of the leader.

“Gemini 2.5 Flash , drops to an 𝐹 1 of 42.99%, roughly half that of the leader.”

May combine two claims.

Gemini Deep Research Agent achieves an F1-Score of 81.90% but a strict success rate of 66.09%, a roughly 15-point gap.

“For instance, while the Gemini Deep Research Agent achieves an F1-Score of 81.90%, its strict success rate is 66.09%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A constraint violation occurs when the model fails to filter data points according to the prompt's requirements, indicating a breakdown in the agent's stopping criteria.

“Finally, the third example demonstrates a constraint violation, where the model fails to filter data points according to the prompt's requirements, indicating a breakdown in the agent's stopping criteria.”

May combine two claims.

Gemini Deep Research Agent achieves the lowest Fully Incorrect rate (9.95%) among evaluated models, lower than GPT-5 Pro High Reasoning (14.13%).

“the Gemini Deep Research Agent minimizes catastrophic failures more effectively, achieving the lowest Fully Incorrect rate of 9.95% vs 14.13% for GPT-5 Pro high reasoning.”

May combine two claims.

Allocating more test-time compute and sampling n times increases the Fully Correct rate from 67.18% (n=1) to 85.71% (n=8), with 74.51% at n=2 and 81.72% at n=4.

“On the other side of the spectrum, we can observe that by allocating more test-time compute and sampling 𝑛 times, we can increase the Fully Correct rate from 67.18% ( 𝑛 = 1) to 85.71% ( 𝑛 = 8). Even sampling only twice yields 74.51%, and with 𝑛 = 4, we can already achieve 81.72%.”

May combine two claims.

Gemini 2.5 Flash has a Fully Incorrect rate of 45.27%, nearly five times the failure rate of the SOTA agent.

“More critically, the Fully Incorrect rate spikes to 45.27%-nearly five times the failure rate of the SOTA agent.”

May combine two claims.

o3 Deep Research has a Fully Correct rate of 44.24% and o4 Mini Deep Research has a Fully Correct rate of 40.36%.

“The mid-tier consists of o3 Deep Research (Fully Correct: 44.24%) and o4 Mini Deep Research (Fully Correct: 40.36%).”

May combine two claims.

o3 Deep Research has a Fully Incorrect rate of 20.09% and o4 Mini Deep Research has a Fully Incorrect rate of 24.19%.

“Their Fully Incorrect rates rise to 20.09% and 24.19% respectively”

Classified as probably not a substantial claim. May combine two claims.

The evaluation uses an automated LLM-as-a-Judge pipeline with Gemini 2.5 Flash in zero-shot setting to determine semantic equivalence.

“We employ an automated LLM-as-a-Judge pipeline to determine the semantic equivalence of extracted answers.”

• Also stated in: § 3.2. Automated Evaluation Pipeline · p. 7 · plain

  “For evaluation purposes, we use Gemini 2.5 Flash (Gemini Team, Google, 2025) in a zero-shot setting using the prompt provided in Section A.”

• cites Gemini Team, Google, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: The LLM-as-a-Judge pipeline accurately determines semantic equivalence between submitted and ground-truth answers.

Probably not a substantial claim: may not pass S4 (based).

All DeepSearchQA tasks are grounded in the open web with objectively verifiable answer sets.

“All tasks are grounded in the open web with objectively verifiable answer sets.”

Classified as probably not a substantial claim.

The questions were rigorously filtered to focus on objective, information-seeking tasks where the ground truth is definitive.

“These were rigorously filtered to focus on objective, information-seeking tasks where the ground truth is definitive.”

May combine two claims.

The shift from simple retrieval to agentic information retrieval necessitates three higher-order capabilities: systematic collation, entity resolution, and reasoning about stopping criteria.

“This shift from simple retrieval to agentic information retrieval (Zhang et al., 2024) necessitates a fundamentally different cognitive profile. It requires three higher-order capabilities that existing benchmarks fail to isolate:”

• cites Zhang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Systematic collation, entity resolution, and reasoning about stopping criteria are the critical higher-order capabilities required for agentic information retrieval.

Classified as probably not a substantial claim.

The broad domain distribution ensures agents are tested on their ability to generalize across varying web structures and content types.

“ensuring that agents are tested on their ability to generalize across varying web structures and content types.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All ambiguous prompts were filtered during the verification protocol.

“All ambiguous prompts were filtered at this stage.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Gemini Deep Research Agent likely achieves state-of-the-art more efficiently, judging by significantly lower cost.

“although Gemini Deep Research Agent likely does so more efficiently, judging by the significantly lower cost 1 .”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

F1-score serves as the primary ranking metric, providing a balanced measure of performance by calculating the harmonic mean of precision and recall.

“F1-Score ( 𝐹 1) serves as the primary ranking metric, providing a balanced measure of performance by calculating the harmonic mean of Precision ( 𝑃 ) and Recall ( 𝑅 ).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For single-answer tasks, the F1 score is equivalent to exact match accuracy (binary 1.0 or 0.0).

“For single-answer tasks: The F1 score is equivalent to exact match accuracy (binary 1.0 or 0.0).”

Classified as probably not a substantial claim.

To ensure ground truth accuracy, the authors implemented a standardized three-phase verification protocol.

“To ensure ground truth accuracy, we implemented a standardized three-phase verification protocol.”

Probably not a substantial claim: may not pass S4 (based).

The task complexity taxonomy helps diagnose specific weaknesses in agent architectures.

“This taxonomy helps diagnose specific weaknesses in agent architectures.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

For set-answer tasks, the F1 score ensures performance reflects both completeness and accuracy, and strictly penalizes agents that maximize recall by outputting many guesses.

“For set-answer tasks: The F1 score ensures performance reflects both completeness and accuracy. This metric strictly penalizes agents that attempt to maximize recall by simply outputting a large number of guesses (hallucinations or 'drift').”

Probably not a substantial claim: may not pass S4 (based).

To achieve a high F1 score, an agent is forced to master the trade-off between exploration and exploitation.

“To achieve a high F1 score, an agent is effectively forced to master the trade-off between exploration (casting a wide net) and exploitation (verifying candidates), navigating the open web without a predefined map.”

May combine two claims.

The iterative loop of an agent is required to close the Comprehensiveness Gap, while reasoning models provide raw intelligence.

“This confirms that while reasoning models provide the raw intelligence, the iterative loop of an agent is required to close the Comprehensiveness Gap.”

May combine two claims.

DeepSearchQA tasks are intractable via simple semantic search; they require structured retrieval and multi-step reasoning.

“This confirms that DeepSearchQA tasks are intractable via simple semantic search; they require structured retrieval and multi-step reasoning.”

May combine two claims.

Recent work on unified retrieval evaluation begins to address agentic information retrieval but lacks the exhaustive scope required here.

“Recent work on unified retrieval evaluation, such as Fact-Fetch-Reason (Krishna et al., 2025) and MoNaCo (Wolfson et al., 2025) begin to address the agentic nature of information retrieval, but lacks the exhaustive scope required here.”

• cites Krishna et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wolfson et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A task's ground truth may become outdated if source websites are removed or their content is significantly altered.

“A task's ground truth may become outdated if source websites are removed or their content is significantly altered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The main results on DeepSearchQA were independently evaluated by Kaggle.

“Table 4 | Main results on DeepSearchQA, independently evaluated by Kaggle.”

• cites KaggleNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Existing benchmarks like Humanity's Last Exam and BrowseComp have been instrumental in pushing the frontiers of agentic reasoning and information discovery, but the Comprehensiveness Gap remains a critical blind spot.

“While existing benchmarks like Humanity's Last Exam and BrowseComp have been instrumental in pushing the frontiers of agentic reasoning and information discovery, the Comprehensiveness Gap remains a critical blind spot.”

• cites Humanity's Last ExamNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites BrowseCompNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This is a prevalent challenge for all benchmarks operating on the live web, necessitating periodic manual reviews and updates to the dataset.

“This is a prevalent challenge for all benchmarks operating on the live web, necessitating periodic manual reviews and updates to the dataset.”

May combine two claims.

The F1-Score vs strict success gap is driven by two opposing failure modes: under-retrieval, where the agent finds the majority of items but misses the 'long tail' of obscure entities; and over-retrieval, where the agent achieves perfect recall (R=1.0) but fails to recognize when the search is complete, hallucinating extra items or drifting into adjacent topics.

“This is driven by two opposing failure modes: under-retrieval , where the agent finds the majority of items but misses the 'long tail' of obscure entities; and over-retrieval , where the agent achieves perfect recall ( 𝑅 = 1 . 0) but fails to recognize when the search is complete, hallucinating extra items or drifting into adjacent topics.”

Probably not a substantial claim: may not pass S4 (based).

DeepSearchQA directly measures the skills of systematic exploration, multi-source synthesis, and search completion-competencies that are vital for many real-world research and analysis tasks.

“DeepSearchQA directly measures the skills of systematic exploration, multi-source synthesis, and search completion-competencies that are vital for many real-world research and analysis tasks.”

Probably not a substantial claim: may not pass S4 (based).

The benchmark's outcome-based evaluation protocol, grounded in Precision, Recall, and F1-score, provides a robust and flexible framework for assessing performance without constraining architectural innovation.

“The benchmark's outcome-based evaluation protocol, grounded in the established information retrieval metrics of Precision, Recall, and F1-score, provides a robust and flexible framework for assessing performance without constraining architectural innovation.”