WideSearch: Benchmarking Agentic Broad Info-Seeking

May combine two claims.

The average human completion time per task is 2.33 hours (English 2.29, Chinese 2.37).

“As illustrated in Figure 6a, we report the distribution of human completion times. It directly reflects the significant cognitive and temporal investment demanded by each task, with an overall average completion time of 2.33 hours. This is remarkably consistent across languages, with English tasks averaging 2.29 hours and Chinese tasks averaging 2.37 hours.”

May combine two claims.

The overall success rate is exceptionally low, with the top-performing multi-agent framework achieving a mere 5.1%, and individual humans struggling at 20%.

“The overall success rate is exceptionally low, with even the top-performing multi-agent framework achieving a mere 5.1%, and individual humans also struggling at 20%.”

May combine two claims.

Most systems achieve overall success rates near 0%, with the best performer reaching just 5%.

“Most systems achieve overall success rates near 0%, with the best performer reaching just 5%.”

May combine two claims.

Detailed error analysis traces failure to incomplete planning, lack of reflection to iterate on failed searches, and inability to correctly use retrieved evidence.

“Our detailed error analysis traces this failure to fundamental deficiencies in advanced agentic capabilities, such as incomplete planning, a lack of reflection to iterate on failed searches, and the inability to correctly use retrieved evidence.”

Probably not a substantial claim: may not pass S4 (based).

The performance of existing agents on broad information-seeking tasks hasn't been thoroughly evaluated.

“However, the performance of these agents on broad information-seeking tasks hasn't been thoroughly evaluated.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The average answer data volume is 2001.2 for Chinese questions and 938.6 for English questions.

“| Average Volume | 2001.2 | 938.6 |”

May combine two claims.

Current advanced large language models exhibit poor planning by struggling to break down complex questions into comprehensive sub-queries, which leads to incomplete information seeking.

“They exhibit poor planning by struggling to break down complex questions into comprehensive sub-queries, which leads to incomplete information seeking.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Given sufficient time, cross-validation by multiple human testers can achieve a near 100% success rate.

“However, given sufficient time, cross-validation by multiple human testers can achieve a near 100% success rate.”

May combine two claims.

WideSearch comprises 200 manually curated questions, 100 in English and 100 in Chinese, from over 15 diverse domains, grounded in real user queries.

“The benchmark features 200 manually curated questions (100 in English, 100 in Chinese) from over 15 diverse domains, grounded in real user queries.”

May combine two claims.

The final WideSearch benchmark comprises 200 high-quality tasks, distributed equally between English and Chinese (100 tasks per language).

“The rigorous curation pipeline culminates in the final WideSearch benchmark, which comprises 200 high-quality tasks. For robust cross-lingual evaluation, these tasks are distributed equally between English and Chinese (100 tasks per language).”

• AssumptionRests on an assumption: The 200 tasks are representative of real-world wide-scale information seeking.

May combine two claims.

Current advanced large language models lack reflection and fail to dynamically adjust their search strategy when initial attempts are unsuccessful, often giving up or answering with insufficient data instead of trying new methods.

“Furthermore, they lack reflection and fail to dynamically adjust their search strategy when initial attempts are unsuccessful, often giving up or answering with insufficient data instead of trying new methods.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The agent initiated the search with a very broad query: 'World bank africa GDP annual growth rate 2022 2023 2024'.

“The agent initiated the search with a very broad query:”

May combine two claims.

Models exhibit refusal behaviors for a subset of queries, with two primary patterns: perceiving ambiguity and requesting clarification, or deeming required information too extensive for a single output and directly refusing.

“Response Refusal. The model exhibits refusal behaviors for a subset of queries. We have identified two primary patterns of refusal: 1. The model perceives ambiguity in the user's question and consequently requests further clarification to narrow the scope of the inquiry. 2. The model deems the required information too extensive to be presented in a single output, leading to a direct refusal to”

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent failed to generate the necessary subsequent queries to gather the additional required data points for each university.

“However, it failed to generate the necessary subsequent queries to gather the additional required data points for each university.”

May combine two claims.

Agents fail to correctly ground final answers in retrieved evidence, either by misinterpreting or disregarding relevant source content, or by failing to validate a source's context and relevance and thus misapplying factually correct but inappropriate information.

“This failure occurs when the agent does not correctly ground its final answer in the evidence it retrieves, revealing a critical gap between information retrieval and generation. This deficiency typically appears in two ways: the agent either misinterprets or disregards the content of a relevant source, or it fails to validate a source's context and relevance, thereby misapplying factually correct”

Probably not a substantial claim: may not pass S3 (load-bearing).

The source of the retrieved GPA information was cive.uh.edu, the website for the University of Houston, not Harvard.

“However, the source of this information was cive.uh.edu, the website for the University of Houston, not Harvard.”

May combine two claims.

The agent failed to validate the context and relevance of the retrieved evidence, incorrectly attributing the GPA requirement from the University of Houston to Harvard University in its final table.

“The agent failed to validate the context and relevance of the retrieved evidence, incorrectly attributing the GPA requirement from the University of Houston to Harvard University in its final table.”

May combine two claims.

The Item-level F1 score shows significant improvement as the number of attempts increases, reaching close to 80 points with 128 attempts.

“As shown in Figure 7, the Item-level F1 score shows a significant improvement as the number of attempts increases. With the compute volume of 128 attempts, it even reaches a level close to 80 points.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent successfully decomposed the initial part of the task, generating specific queries to identify the top universities in each subject area.

“The agent successfully decomposed the initial part of the task, generating specific queries to identify the top universities in each subject area”

May combine two claims.

When the search engine fails to return relevant information, LLMs sometimes use their internal knowledge, frequently leading to hallucinations where the model fabricates non-existent facts or provides incorrect information conflicting with established knowledge.

“When the search engine fails to return any relevant information, LLMs sometimes just use their internal knowledge. This frequently leads to”

Probably not a substantial claim: may not pass S3 (load-bearing).

A subset of LLMs fails to strictly adhere to output format requirements, such as failing to generate a Markdown table or producing a malformed one.

“Output Formatting Error. A subset of LLMs fails to strictly adhere to the output format requirements specified in the instructions, such as failing to generate a Markdown table or producing a malformed one.”

May combine two claims.

Even humans cannot achieve a high success rate in single-player mode; the success rate for a single individual completing the task independently is 20% when given ample time and access to any tools.

“Experimental results show that even when given ample time and access to any tools, the success rate for a single individual completing the task independently is merely 20%.”

• Also stated in: § 4.2 Main Results · p. 12 · plain

  “Even humans cannot achieve a high success rate in single-player mode.”

May combine two claims.

Even the most advanced systems have extremely low success rates on table-level tasks, with the top performer achieving only 5%, while most systems score near 0%.

“The results show that even the most advanced systems have extremely low success rates on table-level tasks, with the top performer achieving only 5%, while most systems score near 0%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The search results provided only aggregated data for the entire region, not the country-specific data required.

“The search results provided only aggregated data for the entire region, not the country-specific data required.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The agent's search for the future-dated entrance fee information correctly returned no results.

“The agent's search for this future-dated information correctly returned no results.”

May combine two claims.

Even advanced large reasoning models lack a dynamic adjustment mechanism; they tend to abandon the search after an initial failed attempt and proceed to answer based on incomplete information or internal knowledge.

“However, we find that even advanced large reasoning models lack this dynamic adjustment mechanism. They tend to abandon the search after an initial failed attempt and proceed to answer based on incomplete information or their internal knowledge, reflecting a deficiency in critical thinking and adaptive planning.”

May combine two claims.

Some models show a clear advantage in domains like 'Healthcare,' while others excel in topics like 'Automotive' or 'Law.'

“some models show a clear advantage in domains like”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tasks are prematurely terminated because the model generated overly verbose intermediate steps or became trapped in ineffective loops, causing total input to exceed the model's maximum context window.

“Context Length Exceedance. The task is prematurely terminated because the model generated overly verbose intermediate steps or became trapped in ineffective loops, causing the total input to exceed the model's maximum context window.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

The agent issued a correct query and retrieved a snippet of text stating a GPA requirement of '3.0/4.0'.

“It issued a correct query and retrieved a snippet of text stating a GPA requirement of”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Tool invocation errors are one of the most common failures, typically caused by the model generating incorrect parameter formats, omitting necessary arguments, or attempting to call a non-existent tool, leading to an API call failure.

“Tool Invocation Error. This is one of the most common failures, typically caused by the model generating incorrect parameter formats, omitting necessary arguments, or attempting to call a non-existent tool, leading to an API call failure.”

May combine two claims.

Current advanced large language models show fundamental weaknesses when performing large-scale information-seeking tasks, with failures stemming from fundamental cognitive deficits beyond simple search inaccuracies.

“Current advanced large language models show fundamental weaknesses when performing large-scale information-seeking tasks, with failures stemming from fundamental cognitive deficits beyond simple search inaccuracies.”

May combine two claims.

The automated evaluation for each task is calibrated to and predictive of human judgment.

“Most critically, it also ensures that the automated evaluation for each task is calibrated to and predictive of human judgment.”

Classified as probably not a substantial claim.

A multi-stage data curation and validation pipeline ensures every question adheres to the design principles.

“To ensure every question in the benchmark rigorously adheres to the design principles, we implement a multi-stage data curation and validation pipeline, as illustrated in Figure 3.”

May combine two claims.

Unlike existing benchmarks that test deep reasoning on a single query, WideSearch evaluates an agent's ability to gather broad information across multiple parallel entities by requiring it to populate a structured table.

“Unlike existing benchmarks that test deep reasoning on a single query, WideSearch evaluates an agent's ability to gather broad information across multiple parallel entities by requiring it to populate a structured table.”

May combine two claims.

A human annotation study with 30 participants was conducted on a representative subset of 100 tasks (50 Chinese, 50 English).

“To quantitatively substantiate the complexity inherent in our benchmark, we conduct a detailed human annotation study with 30 participants. This evaluation is performed on a representative subset of 100 tasks, drawn equally from the Chinese and English pools (50 tasks each).”

Classified as probably not a substantial claim. May combine two claims.

An agent's response is assigned a score of zero if it is not a valid Markdown table or if its column headers do not match the ground truth in number and name.

“An agent's response is immediately assigned a score of zero if it is not a valid Markdown table that can be correctly parsed, or if its column headers do not match the ground truth in number and name.”

Probably not a substantial claim: may not pass S4 (based).

A rigorous five-stage quality control pipeline ensures the difficulty, completeness, and verifiability of the dataset.

“A rigorous five-stage quality control pipeline ensures the difficulty, completeness, and verifiability of the dataset.”

Probably not a substantial claim: may not pass S4 (based).

WideSearch is the first benchmark specifically designed for wide-context information gathering.

“we introduce WideSearch, the first benchmark specifically designed for this purpose, supported by a sophisticated multi-stage data collection and verification framework, as well as a hybrid automated evaluation system that ensures objectivity.”

Classified as probably not a substantial claim. May combine two claims.

Tasks that a human annotator completes in less than 10 minutes or by consulting fewer than 10 unique web pages are discarded.

“Based on our current heuristics, this includes any task that a human annotator completes in less than 10 minutes or by consulting fewer than 10 unique web pages.”

Classified as probably not a substantial claim.

WideSearch tasks require extensive informational breadth, necessitating a high volume of search interactions and a prolonged, multi-step procedural trajectory.

“High Search Volume and Breadth : Tasks are defined by their extensive informational breadth, requiring the agent to collate numerous distinct data points across multiple entities. This inherent breadth necessitates a high volume of search interactions and a prolonged, multi-step procedural trajectory for completion, distinguishing them from tasks that require only a singular, deep line of inquiry.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A task is only accepted into the final benchmark after passing all filtering stages and the final iterative validation loop.

“A task is only accepted into the final benchmark after successfully passing through all filtering stages and the final iterative validation loop.”

May combine two claims.

WideSearch diverges from DeepSearch and DeepResearch; its primary challenge is operational scale and fidelity, not cognitive difficulty.

“Instead, WideSearch tackles tasks whose primary challenge is not cognitive difficulty but operational scale and fidelity-the”

May combine two claims.

WideSearch is a robust and reliable testbed for advanced search agents.

“This final validation loop provides a strong guarantee that WideSearch is a robust and reliable testbed for advanced search agents.”

• AssumptionRests on an assumption: The automated evaluation pipeline with GPT-4.1 as judge is reliable and objective.

May combine two claims.

The five-stage pipeline ensures that every task is grounded in a real-world need, demonstrably complex, verifiable, and resistant to simple memorization.

“This rigorous, five-stage pipeline ensures that every task in the final WideSearch benchmark is grounded in a real-world need, demonstrably complex, verifiable, and resistant to simple memorization.”

May combine two claims.

Each task was independently completed by two annotators, and only the data from the annotator with higher accuracy was used.

“To mitigate the impact of such potential errors and establish a robust performance ceiling, we implement a dual-annotation protocol. Each task is independently completed by two annotators, and we exclusively utilize the data from the annotator who achieved higher accuracy.”

Probably not a substantial claim: may not pass S4 (based).

Parametric knowledge filtering ensures that all tasks in WideSearch genuinely evaluate an agent's search and synthesis capabilities.

“This critical filtering step ensures that all tasks in WideSearch genuinely evaluate an agent's search and synthesis capabilities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4.1-2025-04-14 is used as the default judge LLM.

“We use the GPT-4.1-2025-04-14 as the default judge LLM.”

Probably not a substantial claim: may not pass S4 (based).

Each WideSearch task is associated with a deterministically verifiable set of facts, allowing objective, consistent, and reproducible scoring.

“Objective Verifiability : Each task is associated with a deterministically verifiable set of facts. This allows for objective, consistent, and reproducible scoring against a predefined gold standard.”

May combine two claims.

The bottleneck is ensuring absolute completeness and accuracy of each atomic unit; any single omission, error, or extra data results in total failure of the task execution.

“Rather, the bottleneck is that we must ensure the absolute completeness and accuracy of each atomic unit of information within a large-scale search. Any single data omission or error, or the integration of extra data into the final result, results in total failure of the task execution.”

Classified as probably not a substantial claim.

Each WideSearch task requires agents to collect large-scale atomic information that can be verified one by one objectively and arranged into a well-organized output.

“Each task requires agents to collect large-scale atomic information, which could be verified one by one objectively, and arrange it into a well-organized output.”

Classified as probably not a substantial claim. May combine two claims.

WideSearch tasks are designed to exceed the bounds of an LLM's parametric knowledge, so successful completion requires active web search.

“Reliance on External Tools : Tasks are explicitly designed to exceed the bounds of an LLM's parametric knowledge. Successful completion is therefore contingent upon the agent's ability to engage in active, iterative, and effective web search, rather than relying on memorized information.”

Classified as probably not a substantial claim.

WideSearch is a new benchmark engineered to evaluate agent reliability on large-scale collection tasks.

“we introduce WideSearch, a new benchmark engineered to evaluate agent reliability on these large-scale collection tasks.”

May combine two claims.

The fabricated fee response was not based on any retrieved evidence but was instead generated from the model's internal parameters, resulting in a factual inconsistency.

“This response was not based on any retrieved evidence but was instead generated from the model's internal parameters, resulting in a factual inconsistency.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This error highlights a fundamental gap between information retrieval and faithful generation, where the agent does not properly ground its answer in correctly attributed evidence.

“This error highlights a fundamental gap between information retrieval and faithful generation, where the agent does not properly ground its answer in correctly attributed evidence.”

May combine two claims.

Large-scale information retrieval on a fixed topic is an extremely difficult task, requiring comprehensive and accurate search results.

“This strongly suggests that large-scale information retrieval on a fixed topic is an extremely difficult task. It requires not only comprehensive but also accurate search results.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because the distribution of questions per domain is uneven, the heatmap serves primarily to identify general patterns rather than to provide a precise statistical comparison.

“As the distribution of questions per domain is uneven, the heatmap serves primarily to identify general patterns rather than to provide a precise statistical comparison.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The greater challenge of domains such as 'Academics' and 'Transportation' for all models is likely due to the need for highly specialized and nuanced information.

“likely due to the need for highly specialized and nuanced information.”

May combine two claims.

The core deficiencies in current agent systems lie in a lack of advanced agentic capabilities, such as the inability to decompose complex problems into comprehensive sub-queries, a lack of reflection and iteration after initial search failures, and the failure to correctly utilize retrieved evidence.

“The core deficiencies in current agent systems lie in a lack of advanced agentic capabilities, such as the inability to decompose complex problems into comprehensive sub-queries, a lack of reflection and iteration after initial search failures, and the failure to correctly utilize retrieved evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This case underscores the challenge of ensuring that LLMs remain strictly grounded in external sources, especially when those sources do not contain the requested information.

“This case underscores the challenge of ensuring that LLMs remain strictly grounded in external sources, especially when those sources do not contain the requested information.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The omission of subsequent queries demonstrates a failure in comprehensive task planning, as the agent did not create a complete set of subtasks needed to fulfill all aspects of the user's request, resulting in a final output with significant information gaps.

“This omission demonstrates a failure in comprehensive task planning, as the agent did not create a complete set of subtasks needed to fulfill all aspects of the user's request, resulting in a final output with significant information gaps.”