The DOI matches its DataCite record; 38 of the 38 references it cites were found in a registry or an index; 6 of the 13 sources its claims cite were checked for support.
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Open accessAbstract
As large language models (LLMs) evolve into tool-using agents, the ability to browse the web in real-time has become a critical yardstick for measuring their reasoning and retrieval competence. Existing benchmarks such as BrowseComp concentrate on English and overlook the linguistic, infrastructural, and censorship-related complexities of other major information ecosystems -- most notably Chinese. To address this gap, we introduce BrowseComp-ZH, a high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning 11 diverse domains. Each question is reverse-engineered from a short, objective, and easily verifiable answer (e.g., a date, number, or proper noun). A two-stage quality control protocol is applied to strive for high question difficulty and answer uniqueness. We benchmark over 20 state-of-the-art language models and agentic search systems on our proposed BrowseComp-ZH. Despite their strong conversational and retrieval capabilities, most models struggle severely: a large number achieve accuracy rates below 10%, and only a handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch, reaches just 42.9%. These results demonstrate the considerable difficulty of BrowseComp-ZH, where success demands not only effective retrieval strategies, but also sophisticated reasoning and information reconciliation -- capabilities that current models still struggle to master. Our dataset, construction guidelines, and benchmark results have been publicly released at https://github.com/PALIN2018/BrowseComp-ZH.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
BrowseComp-ZH is a new 289-question Chinese-web benchmark built by reverse-designing multi-hop queries from short, verifiable answers across 11 domains. It evaluates over 20 LLMs and AI search systems, finding most models below 10% accuracy and the best system, OpenAI DeepResearch, at 42.9%.
Key findings
- Most standalone LLMs perform poorly: Qwen2.5-72B-Instruct 6.6%, GPT-4o 6.2%, Claude-3.5-Sonnet 5.5%, LlaMa4 4.8%; many systems remain below 10% accuracy.
- Reasoning-capable models improve substantially: DeepSeek-R1 reaches 23.2% vs. DeepSeek-V3’s 8.7%, and Claude-3.7-Sonnet reaches 17.7% vs. Claude-3.5-Sonnet’s 5.5%.
- Agentic search systems with multi-round retrieval perform best: OpenAI DeepResearch 42.9%, Doubao (Deep Search) 26.0%, Perplexity (Research) 22.6%.
- Enabling web search can hurt: DeepSeek-R1 drops from 23.2% in direct-answer mode to 7.6% with web search enabled; calibration error also rises from 59% to 65%.
Methods
- Dataset construction used a reverse strategy: 10 expert contributors selected factual answers and crafted multi-constraint Chinese queries, tested on Baidu, Bing, and Google so answers did not appear on first-page results.
- Two-stage quality control: Stage 1 filtered 76 simple samples from 480 preliminary samples, leaving 404 high-difficulty candidates; Stage 2 used AI agents plus manual verification to eliminate 115 ambiguous samples, yielding 289 final questions across 11 domains.
- Evaluation covered over 20 open-source models, closed-source APIs, and AI search products; accuracy and calibration error were reported, with regex extraction plus GPT-4o grading for models and human extraction for AI search products.
Limitations
- Stated: the dataset is relatively small, and increasing sample size and question-type diversity would improve representativeness.
- Stated: answer uniqueness cannot be fully guaranteed, and the dynamic web means factual answers may evolve or become inconsistent over time, challenging stability and reproducibility.
- Evident from the text: evaluation depends on GPT-4o grading for model outputs and human annotators for AI search products, which may introduce grader or annotator variability; topic coverage is also uneven, e.g., Film & TV 15.6% versus Academic Papers 2.4%.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Own result · 20 Own interpretation · 18 Own method · 16 No basis given · 6 Sources not found · 3 Supported · 2 Couldn’t confirm · 1 Partly supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“Inspired by Wei et al. (2025), we adopt a reverse construction strategy: each task begins with a factual answer, from which an elaborate, multi-constraint query is crafted to make direct retrieval non-trivial.”
What the sources say
- Supportedcites Wei et al. (2025) source
In “Easy to verify, but hard to solve” (page not specified), BrowseComp says trainers were encouraged to start with a fact and construct an inverted question whose answer would be hard to find. It describes building the question from characteristics of a seed, supporting the cited inspiration for the reverse-construction strategy. The claim that BrowseComp-ZH adopts that strategy is the citing authors’ own description, not a claim about the source. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“enabling search functionality for DeepSeek-R1 leads to a substantial decline in performance”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Doubao (Standard) (26.0% vs. 18.7%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yuanbao (12.2%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the calibration error for DeepSeek-R1 increases from 59% in the direct-answer setting to 65% when search is enabled”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“OpenAI's O1 (29.1%)”
- Also stated in: § 4.4 Performance · p. 7 · plain
“O1 (29.1%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.4 Performance · p. 7 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-3.5-Sonnet (5.5%)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Doubao (Deep Search) achieves a nearly 8% absolute improvement over Doubao (Standard)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DeepSeek-V3's 8.7%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Doubao (Deep Search) achieves a nearly 8% absolute improvement over Doubao (Standard) (26.0% vs. 18.7%)”
- Also stated in: § 4.6 Analysis of AI Search Products · p. 8 · plain
“Doubao (Deep Search) achieving 26.0%”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.6 Analysis of AI Search Products · p. 8 · plain
- It may state two separate claims as one. 34
- Probably not a substantial claim. 27
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Nonetheless, Wei et al. (2025) primarily operates within the English-language web, missing the linguistic, structural, and cultural complexities inherent to other language environments.”
What the sources say
- Could not verifycites Wei et al. (2025) sourceExperimental — accuracy not yet measured
The Introduction and Related work and discussion describe BrowseComp as a benchmark for finding hard-to-find information on the web, but do not specify that it primarily uses the English-language web. Searches for English, non-English, multilingual, and cultural or linguistic limitations found no passage assessing its coverage of other language environments. (LLM, conf=0.73)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Liu et al. (2023) and Li et al. (2023b) focus on domain-specific medical QA, but do not evaluate open-ended retrieval or browsing strategies.”
What the sources say
- Supportedcites Liu et al. (2023) source
The Experimental Setting evaluates answers to medical multiple-choice questions and open-ended solution explanations, not retrieval or browsing strategies. The Conclusion and Discussions identifies CMExam as a medical question-answering benchmark. This verdict concerns Liu et al. only; it does not assess the separate claim about Li et al. (LLM, conf=0.88)
- Not matchedcites Li et al. (2023b)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“While it offers finer-grained evaluation for web browsing agents, it remains confined to the English web and lacks generalizability to non-English environments with fragmented platforms and diverse linguistic structures.”
What the sources say
- Could not verifycites Wei et al. (2025) sourceExperimental — accuracy not yet measured
The Introduction and Section 5 describe BrowseComp as a benchmark for agents searching the web, but do not state that it is confined to the English web. Searches for English-only coverage, multilingual evaluation, and non-English generalizability found no passage addressing fragmented platforms or diverse linguistic structures. (LLM, conf=0.74)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“To address this gap, Wei et al. (2025) introduced a benchmark of reverse-designed, evidence-grounded queries that challenge English-language agents to search and reason over difficult-to-access information.”
What the sources say
- Supportedcites Wei et al. (2025) source
The Introduction (page unavailable) introduces BrowseComp as a benchmark requiring agents to browse many websites. “Easy to verify, but hard to solve” describes constructing inverted questions from known facts, while “Data collection and verification” says the questions were supported by evidence. The Related work section describes its focus as hard-to-find information on the web. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The grading prompts are adapted from those used in the original BrowseComp benchmark Wei et al. (2025).”
What the sources say
- Supportedcites Wei et al. (2025) source
Section 2.3 Grading says the full BrowseComp grading prompt appears in Appendix B. The Additional information section reproduces that prompt, identifying the original prompt to which the citing sentence refers. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While well-designed retrieval pipelines can significantly improve performance, not all systems benefit from retrieval integration. For instance, enabling web search for DeepSeekR1 results in a substantial decline in performance, with accuracy dropping from 23.2% in the direct-answer (no web access) setting to 7.6% when web search is enabled.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BrowseComp-ZH consists of 289 multi-hop questions spanning 11 diverse domains.”
- Also stated in: § 1 Introduction · p. 3 · plain
“The final BrowseComp-ZH dataset consists of 289 complex questions, each with multiple constraints and unique answers, spanning 11 diverse domains, including film, technology, law, and medicine.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This process eliminates 115 ambiguous samples, resulting in a final benchmark of 289 validated questions that maintains high levels of difficulty and answer verifiability.”
plaindirectn = 115- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated, the most represented categories include Film & TV (15.6%), Art (13.8%), and Geography (12.8%), reflecting the diverse interests of annotators and a broad coverage of Chinese web content. The dataset also features Music (11.1%), History (10.0%), and Medicine (9.0%). Conversely, topics like Policy & Law (3.5%) and Academic Papers (2.4%) have fewer samples”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models endowed with inherent reasoning capabilities exhibit consistent improvements in accuracy. For example, DeepSeek-R1 surpasses DeepSeek-V3 by 14.5%, while Claude-3.7Sonnet outperforms Claude-3.5 by 12.2%.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite their strong conversational and retrieval capabilities, most models struggle severely: a large number achieve accuracy rates below 10%, and only a handful exceed 20%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this stage, annotators identified 76 simple samples, and after filtering them out, we are left with 404 high-difficulty candidates.”
plainn = 76- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This process yields 480 preliminary samples, which are subsequently reviewed via the quality control procedure detailed in Section 3.2.”
plaindirectn = 480- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Agentic systems augmented with external retrieval mechanisms tend to achieve comparatively higher scores. Notably, OpenAI's DeepResearch and Doubao (Deep Search) demonstrate that well-orchestrated retrieval and reasoning pipelines can substantially enhance performance, achieving accuracies of 42.9% and 26.0%, respectively.”
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Fig. 3 (b), the answer length distribution shows that answers are generally short, typically falling within the range of 5 to 10 characters.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we construct challenging question-answer pairs that require multi-hop retrieval, information filtering, and logical reasoning to derive concise, factual answers”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the current dataset is relatively small”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We construct 289 high-quality samples across 11 diverse topics”
plainn = 289- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“canonical information pathways in English (e.g., Wikipedia, IMDb) lack equivalents or exhibit vastly different structures in the Chinese web”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the Chinese web presents its own set of retrieval obstacles-fragmented and platform-specific content, inconsistent naming conventions, and linguistic traits such as ellipsis, cultural references, and implicit reasoning that undermine linear search strategies.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These length distributions reflect the high information density of the dataset, emphasizing both complexity in the questions and simplicity in the answers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings suggest that BrowseComp-ZH not only challenges models' browsing and reasoning capabilities but also exposes their limitations in processing, evaluating, and aligning retrieved information with internal representations.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each question combines temporal, spatial, categorical, or descriptive conditions to ensure answer uniqueness; not all conditions are required, but at least two dimensions are typically combined”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Lyu et al. (2025) evaluates RAG systems under the CRUD (Create, Read, Update, Delete) paradigm, yet primarily emphasizes generation quality over retrieval path validation.”
What the sources say
- Not matchedcites Lyu et al. (2025)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To fill this gap, we introduce BrowseComp-ZH , the first benchmark specifically designed to evaluate web-enabled LLMs in the Chinese information environment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results demonstrate the considerable difficulty of BrowseComp-ZH, where success demands not only effective retrieval strategies, but also sophisticated reasoning and information reconciliation-capabilities that current models still struggle to master.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure high-quality annotation, we recruited 10 expert contributors (with Master's or PhD degrees) who have extensive experience in both LLM usage and web search.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The selected facts must be objective statements that can be independently verified through reliable sources, without the need for interpretation or inference”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Every candidate query is verified on three widely-used search engines: Baidu 1 , Bing 2 , and Google 3 , and is accepted only if the answer is not surfaced in the first-page results of any engine.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Annotators test each query on Baidu, Bing, and Google, using three distinct keyword combinations per search engine. If the correct answer appears on the first page of any search engine, the query will be revised or further constrained”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Stage 1 screens for keyword retrievability as above.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Hu et al. (2024) evaluates Chinese web search agents but lacks rigorous control over task difficulty and answer accessibility.”
What the sources say
- Not matchedcites Hu et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These questions require multi-step reasoning and are difficult to answer directly through search engines, making them ideal for evaluating the ability of Chinese LLM agents to conduct multihop retrieval, perform factual reasoning, and integrate online information.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All queries, evidence chains, and browsing steps are authored in Chinese and fully localized.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It establishes a high-difficulty benchmark for Chinese web retrieval, supporting systematic evaluation of agents' ability to navigate fragmented, unstructured, and linguistically diverse Chinese information sources.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This trend aligns with the nature of tasks in BrowseComp-ZH, which often involve multi-faceted queries, making it challenging to obtain accurate answers through a single retrieval operation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Xu et al. (2024) focuses on dynamic, time-sensitive QA tasks but does not emphasize multi-hop retrieval.”
What the sources say
- Not matchedcites Xu et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We employ a human-in-the-loop approach, where multiple top-performing AI agents first perform reasoning and provide answers based on the designed questions. These results are then manually verified by annotators to check if any additional answers satisfy the constraints of the question.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To further discourage shortcut-based resolution, all questions are tested using GPT-4o and DeepSeek, both operating in web-enabled search mode. If both models consistently retrieve the correct answer with minimal effort, the query will be revised to increase its complexity by methods including introducing implicit constraints or obfuscating key lexical signals.”
plaindirectn = 480- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“they must be concrete and specific enough to exclude overly generic or widely known common-sense facts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A two-stage quality control protocol is applied to strive for high question difficulty and answer uniqueness.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To address these limitations in Chinese retrieval benchmarking, BrowseComp-ZH introduces three key innovations: (1) reverse-designed tasks in native Chinese to avoid translation artifacts; (2) rigorous multi-step validation to ensure high retrieval difficulty and answer verifiability; and (3) broad model coverage for evaluating both open-source and proprietary agents.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each sample includes at least one authoritative source URL that validates the logical connection between the query constraints and the target answer.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, since many answers can be retrieved via simple keyword searches, these benchmarks often fail to evaluate an agent's ability to plan complex search trajectories and synthesize information across documents.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, existing English-centric benchmarks struggle to generalize, leaving a critical gap in assessing real-world browsing capabilities in non-English environments.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Stage 2 focuses on ensuring the uniqueness of the answers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This study introduces BrowseComp-ZH, the first benchmark specifically designed to evaluate the web browsing and reasoning capabilities of large language models (LLMs) in the Chinese information environment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“This trend is consistent with the observations reported in BrowseComp.”
What the sources say
- Not matchedcites BrowseComp
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“In the evaluation, we followed the BrowseComp instructions for model predictions.”
What the sources say
- Not matchedcites BrowseComp
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results reflect the inherent challenges of the Chinese web environment, where fragmented information and inconsistent indexing complicate single-shot search strategies.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we implement a rigorous two-stage quality control pipeline that includes a three-engine keyword validation process and human-in-the-loop verification”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“increasing both the sample size and the diversity of question types would improve its representativeness”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“ensuring that answers are both difficult to retrieve and unambiguous”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We adopt the same grading prompt as used in BrowseComp and employ GPT-4o for the grading process.”
What the sources say
- Not matchedcites BrowseComp
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper relies on this without support.
All versions of DeepSeek's search system perform only a single round of retrieval.
“all versions of DeepSeek's search system perform only a single round of retrieval”
The paper relies on this without support.
The dynamic nature of the web means that factual answers may evolve or become inconsistent over time.
“the dynamic nature of the web means that factual answers may evolve or become inconsistent over time”
The paper relies on this without support.
DeepSeek R1, O1, and Gemini-2.5-Pro are models without browsing capabilities and rely solely on internal world knowledge.
“even models without browsing capabilities, such as DeepSeek R1, O1, and Gemini-2.5-Pro, which rely solely on their internal world knowledge”
The paper relies on this without support.
The tasks in BrowseComp-ZH require both extensive world knowledge and strong reasoning abilities.
“the tasks in our proposed BrowseComp-ZH benchmark require both extensive world knowledge and strong reasoning abilities”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These benchmarks typically evaluate agents using blackbox web search APIs to retrieve supporting documents in real time (Zhou et al., 2025; Chen et al., 2025).”
From BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026) · cites this paper as Zhou et al., 2025
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Current benchmarks typically focus on assessing isolated capabilities-such as web browsing and information retrieval [28, 38, 39], or generative abilities disconnected from real-time information acquisition [23, 1, 31].”
From DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Performance · p. 8
“| Grok3 (Research) | AI Search Product | - | Y | 12.9% | 39 | xAI |”
§ 3.2 Quality Control · p. 5
“resulting in a final benchmark of 289 validated questions”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Performance · p. 8
“| Yuanbao (Hunyuan Model) | AI Search Product | - | Y | 12.2% | 56 | Tencent |”
§ 3.2 Quality Control · p. 5
“resulting in a final benchmark of 289 validated questions”
Citations
Cites
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“In parallel, retrieval-augmented generation (RAG) frameworks Guu et al. (2020); Lewis et al. (2020) have been widely adopted in QA Wiratunga et al. (2024), summarization Edge et al. (2024), and fact verification Martin et al. (2024), serving as a mechanism for injecting external knowledge into LLMs.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 1×
“To assess retrieval capabilities, a variety of widely used English benchmarks have been proposed, including TriviaQA Joshi et al. (2017), HotpotQA Yang et al. (2018), FEVER Thorne et al. (2018), KILT Petroni et al. (2020), and GAIA Mialon et al. (2023).”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 1×
“To assess retrieval capabilities, a variety of widely used English benchmarks have been proposed, including TriviaQA Joshi et al. (2017), HotpotQA Yang et al. (2018), FEVER Thorne et al. (2018), KILT Petroni et al. (2020), and GAIA Mialon et al. (2023).”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 1×
“To assess retrieval capabilities, a variety of widely used English benchmarks have been proposed, including TriviaQA Joshi et al. (2017), HotpotQA Yang et al. (2018), FEVER Thorne et al. (2018), KILT Petroni et al. (2020), and GAIA Mialon et al. (2023).”
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · cited 1×
“Representative works such as WebGPT Nakano et al. (2021), Toolformer Schick et al. (2023), and ReAct Yao et al. (2023) explore how LLMs leverage search engines, tool usage, and reasoning strategies to tackle complex question-answering tasks.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 1×
“As large language models (LLMs) evolve from static knowledge repositories to dynamic agents capable of using external tools, tasks that involve web browsing have emerged as a critical lens through which to evaluate their real-world reasoning and information-seeking capabilities Xi et al. (2025).”
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., Cao, N. D., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2009.02252 · Cited by 940
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1×
“Representative works such as WebGPT Nakano et al. (2021), Toolformer Schick et al. (2023), and ReAct Yao et al. (2023) explore how LLMs leverage search engines, tool usage, and reasoning strategies to tackle complex question-answering tasks.”
Cited by
Sorted by citations
Items 101–144 of 144
CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic PlanningPreprint
Ye, Y., Lan, T., Jiang, F., Ye, M., Zhu, B., Jia, Q., Wang, L., Xu, Z., Luo, W., & Shi, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02074 · Cited by 1
“We evaluate CHIME on four long-horizon agent benchmarks: (1) τ 2 -bench , multi-turn customer-service interactions (Barres et al., 2025); (2) VitaBench , multi-turn life-service tasks with changing user intent and large tool spaces (He et al., 2025); (3) BrowseComp-ZH , long-horizon InfoSeeking (Zhou et al., 2025); an…”
From Inertia to Objectivity: Improving Deep Research Agents with Noise Isolation
Zhang, X.-X., Zhang, Z.-W., Fu, Z.-H., Lin, B., & Wang, W. (2026) · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2608.23045 · Cited by 1
Liu, Q., Chen, Y.-Q., Chen, Z., et al. (2026) · doi:10.48550/arxiv.2608.02097 · Cited by 1
Yao, X., Liu, Y., Wang, C., Yu, Y., Tan, H., Zhang, Y., Ren, R., Peng, M., & Liu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.23524 · Cited by 1
QuarkMedSearch: A Long-Horizon Deep Search Agent for Exploring Medical IntelligencePreprint
Lin, Z., Liang, Z., Liu, G., Xu, M., Xiang, B., Zhao, S., Wu, Y., Xu, J., & Jiang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.12867 · Cited by 1
LightThinker++: From Reasoning Compression to Memory ManagementPreprint
Zhu, Y., Zhang, J., Wan, Z., Luo, Y., Qiao, S., Gui, Z., Zheng, D., Liang, L., Chen, H., & Zhang, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.03679 · Cited by 3
“We benchmark our models across three representative agentic datasets: xbench-DeepSearch-2510 [50], BrowseComp-EN [51], and BrowseComp-ZH [52], which we refer to as xbench, BC-EN, and BC-ZH for brevity.”
InfluentialLiu, C., Kuang, C., Zhuang, T., Cheng, Y., Zhou, H., Li, X., & Shang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.08117 · Cited by 2
“Our experiment results reveal that even the top information-seeking agents struggle in UIS-QA, where the strongest baseline yielding only 25.45% accuracy, much lower than their GAIA and BrowseComp-zh (Zhou et al., 2025b) performance (over 70% and 45%, respectively).”
InfluentialSelf-citationOn the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample EfficiencyPreprint
Wang, Y., Zhang, Z., & Wang, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.21619 · Cited by 1
“…D . we select seven datasets: MATH500 [19], AMC [32], AIME24 [33], AIME25 [34] from the mathematical domain with progressively increasing difficulty; and GPQA [37], MMLU-Pro [54], BrowseComp (en [55] + zh [69]) from diverse domains, including general QA, multi-subject knowledge, and agent tasks.”
Yunque DeepResearch Technical ReportPreprint
Cai, Y., Lai, X., Yuan, P., Liu, W., Li, H., Li, M., Wang, X., Zheng, S., Hao, Y., Yin, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.19578 · Cited by 5
“• BrowseComp-ZH[32] -”
Qiu, B., Liu, Z., Liu, X., Wang, B., Zhang, F., Qin, Y., Li, C., Yang, H., & Gao, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.24151 · Cited by 2
“Building on this idea, BrowseComp-ZH [7] extends the framework to the Chinese web domain, aiming to benchmark multilingual web-browsing capabilities of large language models.”
WideSearch: Benchmarking Agentic Broad Info-Seeking
Wong, R., Wang, J., Zhao, J., Chen, L., Gao, Y., Zhang, L., Zhou, X., Wang, Z., Xiang, K., Zhang, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07999
“Benchmarks like BrowseComp-en/zh [27, 32] further elevate the difficulty by designing tasks with intricately coupled entities and deliberate information obfuscation, demanding sophisticated, non-linear exploration to reduce a high degree of initial uncertainty.”
GISA: A Benchmark for General Information-Seeking Assistant
Zhu, Y., Zhang, X., Zhang, M., Jin, J., Zhang, L., Song, X., Zhao, K., Zeng, W., Tang, R., Li, H., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.08543
Poudel, B., Chapagain, N., Kumar, A., et al. (2026) · doi:10.5281/zenodo.23524293
Benchmarking and Error Analysis of Deep Search and Deep Research LLM AgentsPreprint
Yu, Z., Tang, Z., Chu, X., & Han, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.29346
EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents
Luo, S.-J., Gao, M.-J., He, Y.-X., et al. (2026) · doi:10.48550/arxiv.2610.25269
Chen, Y.-J., Wang, Y.-H., Wang, J.-Z., et al. (2026) · doi:10.48550/arxiv.2610.21512
“On WebSailor, +RL achieves the best performance among trained refiners, improving over the baseline by 6.79, 6.23, 5.99, and 3.11 points on BrowseC-omp, BrowseComp-ZH, GAIA, and HLE, respectively.”
InfluentialSelf-citationKnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Chen, S.-Q., Lu, Z., Yan, Y., et al. (2026) · doi:10.48550/arxiv.2610.22331
TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training
Fu, Y.-Q., Jiang, K.-W., Song, Z.-J., et al. (2026) · doi:10.48550/arxiv.2610.17726
Neos: Learning to Interleave Search and Visual ReasoningPreprint
He, E., Hong, J., Yan, S., Zhang, X., Zhang, Y., Li, W., Sun, H., Jiang, K., Fu, J., Zhang, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.17043
“We further evaluate Neos-9B on BrowseComp [Wei et al., 2025], BrowseComp-ZH [Zhou et al., 2025], HLE-Text [Phan et al., 2025], and DeepSearchQA [Gupta et al., 2026] to examine its applicability beyond multimodal tasks.”
Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents
Pang, C., Su, J.-Q., Wu, C.-R., et al. (2026) · doi:10.48550/arxiv.2610.16998
SEEDS: Preview-Guided Evidence Admission for Search AgentsPreprint
Liu, J., Xu, C., Fan, D., Sun, Y., & Tan, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12932
TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-TrainingPreprint
Peng, M., Zhang, Q., Chen, N., Li, Y., Yan, G., Gu, X., Wu, H., Wang, H., Huang, L., Zhang, W., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.04973 · Cited by 1
“BrowseComp-ZH (Zhou et al., 2025a) targets multi-hop information seeking on the Chinese web.”
Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web Search
Ma, J.-Y., Aster, L., Zhang, H., et al. (2026) · doi:10.48550/arxiv.2609.37082
“We evaluate our model on five search benchmarks: BrowseComp for deep information seeking, BrowseComp-ZH (BC-ZH) (Zhou et al., 2025) for multilingual search, WideSearch (Wong et al., 2025) for breadth-oriented retrieval, and xbench-DeepSearch (xbench) (Chen et al., 2025a) and DeepSearchQA (DSQA)…”
InfluentialFinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and TraceabilityPreprint
Wang, W., Xiang, H., Zhao, X., Yin, M., Zhong, Y., Huan, Z., Zhou, Q., Zhu, J., Zhang, X., Chang, S., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.25192 · Cited by 1
“BrowseComp (Wei et al., 2025) and BrowseComp-ZH (Zhou et al., 2025) stress multi-step web search for short, time-invariant answers, while BrowseComp-Plus (Chen et al., 2025) adds a fixed human-verified evidence corpus, controlled retrievers, and citation-retrieval metrics.”
PrimeSeeker: Capability-Oriented Supervision for Deep Search Agents
Peng, L.-Z., Chen, H.-T., Chang, H., et al. (2026) · doi:10.48550/arxiv.2609.35816
“We evaluate on BrowseComp-EN and BrowseComp-ZH (Wei et al., 2025; Zhou et al., 2025), DeepSearch-2505 and DeepSearch-2510 from xbench-DeepSearch (Chen et al., 2025), and GAIA (Mialon et al., 2024).”
G-ReAct: Graph-Guided Deep Search via Structure-State Co-Evolution
Yang, S., Zhang, M.-Y., Lin, S.-J., et al. (2026) · doi:10.48550/arxiv.2608.01324
“To evaluate this capability, challenging and web-intensive benchmarks such as BrowseComp (Wei et al. 2025), BrowseComp-ZH (Zhou et al. 2025), GAIA (Mialon et al. 2024), and XBench (xbench-Team 2025) have been proposed.”
InfluentialCai, W., Chen, Z., Zhu, H., Wang, W., Wang, J., & Xu, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.25554
“These paradigms equip models with Deep Search capability, defined as autonomous multi-hop retrieval over extensive information to locate static facts (Wei et al., 2025; Zhou et al., 2025).”
C 3 Flow: SIMD-Style Concurrent Claude Code Workflow for Scaling Deep Research
Gao, Y., Huang, X., Zhong, Y., Fan, Z., Gui, Z., Liang, L., Xiong, Y., & Wang, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808374
“Recent advances in LLM reasoning and planning enable agent systems to handle complex long-horizon workflows [8, 13, 19, 27], especially retrieval-intensive research tasks (e.g., DeepResearch) [7, 16, 29].”
TreeSeeker: Tree-Structured Trial, Error, and Return in Deep Search
Shi, Z., Ma, M.-Z., Wang, L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.11662
“Table 1: Performance comparison on XBench-DS, BrowseComp, and BrowseComp-ZH.”
InfluentialStruct-Searcher: Agentic Structural Thinking Advances Multimodal Deep Information SeekingPreprint
Zhang, F., Zhang, V., Qian, S., Li, H., Lian, Z., Wu, H., Gao, Y., Geng, X., (91924), X. W., & Heng, P.-A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.07689 · Cited by 1
Mind DeepResearch Technical ReportPreprint
MindDR Team, & Li Auto Inc (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.14518 · Cited by 8
“We comprehensively evaluate the systems across two distinct categories of benchmarks: • DeepSearch (DS) Benchmarks: We measure multi-step information retrieval and reasoning capabilities using BrowseComp-ZH [54], BrowseComp [38], xbench-DS [2], GAIA-DS [21], and WideSearch [39].”
MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Almeida, T., Bonás, G. K., Pires, R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.14448
“BrowseComp-ZH [32] found meaningful degradation in web-browsing performance when moving from English to Chinese, demonstrating that search benchmarks designed for English do not transfer straightforwardly to other information ecosystems.”
From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research AgentsPreprint
Liu, S., Tan, Z., Yi, K., Wu, H., Li, Y., Yan, J., Chen, L., Chen, K., & Yang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.25342
“Unlike early systems confined to static knowledge recall (Wei et al., 2025; Zhou et al., 2025; Chen et al., 2025), true deep research is inherently dynamic (Nie et al., 2024).”
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMsPreprint
Zhu, L., Hua, H., Miao, L., Wang, W., & Zhao, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.11674 · Cited by 1
Exploring Information Seeking Agent Consolidation
Yan, G.-C., Wu, J.-L., Tao, Z.-W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.00585
“For web agent evaluation, we use GAIA (Mi-alon et al., 2023), BrowseComp ( BC ) (Wei et al., 2025a), and BrowseComp-zh ( BC-zh ) (Zhou et al., 2025), and report performance using accuracy.”
LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System DynamicsPreprint
Liu, J., Wu, J., Wu, C., Liu, J., Wang, Z., Zhou, H., Huang, W., & Namkoong, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.21010 · Cited by 1
“…Engineering SWE-Bench [12], Multi-SWE-Bench [31], Terminal Bench 2 Agent 10 Search Agent GAIA [15], HLE [17], BrowseComp [27], BrowseComp-zh [34] 11 Tool Use τ 2 -bench [2] Factuality 12 Factual Hallucinations FActScore [16], LongFact (object) [28], Long-Fact (concept) [28] instruction…”
Laser: Governing Long-Horizon Agentic Search via Structured Protocol and Context RegisterPreprint
Wang, S., Xia, Q., Wang, V., Herberttli, Bobsimons, & Dou, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.20458 · Cited by 1
“5-7B (Yang et al., 2024) and Qwen3-8B/32B (Yang et al., 2025), as well as on challenging multi-hop question answering (QA) benchmarks such as BrowseComp-ZH (Zhou et al., 2025) and WebDancer (Wu et al., 2025).”
InfluentialGLM-5: from Vibe Coding to Agentic Engineering
GLM-5-Team, :, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.15763 · Cited by 6
“For agentic benchmarks, we evaluate GLM-5 and frontier models on BrowseComp [50], BrowseComp-ZH [63], τ 2 -Bench [7], MCP-Atlas [6], Tool-Decathlon [22], Vending-Bench 2 [3], and GDPval-AA [33].”
Self-citationDeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL
Lu, R., Hou, Z., Wang, Z., Zhang, H., Liu, X., Li, Y., Feng, S., Tang, J., & Dong, Y. (2025) · arXiv · doi:10.48550/arxiv.2509.10446 · Cited by 1
Budget-Aware Tool-Use Enables Effective Agent Scaling
Liu, T., Wang, Z., Miao, J., Hsu, I.-H., Yan, J., Chen, J., Han, R., Xu, F., Chen, Y., Jiang, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.17006 · Cited by 1
Self-citationBeyond Turn Limits: Training Deep Search Agents with Dynamic Context Window
Tang, Q., Xiang, H., Yu, L., Yu, B., Lu, Y., Han, X., Sun, L., Zhang, W., Wang, P., Liu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.08276
IterResearch: Rethinking Long-Horizon Agents with Interaction Scaling
Chen, G., Qiao, Z., Chen, X., Yu, D., Xu, H., Zhao, W. X., Song, R., Yin, W., Yin, H., Zhang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.07327
“(Phan et al., 2025) 64 BrowseComp-zh (Zhou et al., 2025a) 64 BrowseComp (Wei et al., 2025a) 256 Maximum Round Settings.”
InfluentialEvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale
Zhu, X., Cai, Y., Liu, Z., Wang, C., Li, F., Jin, W., Liu, W., Bing, Z., Zheng, B., Chai, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.17406
“Scientific reasoning and information search is evaluated by Frontier Science [38], BrowseComp [41], BrowseComp-ZH [46], and HLE [30].”
Needle in the Web: A Benchmark for Retrieving Targeted Web Pages in the Wild
Wang, Y., Fan, T., Xu, L., & Huang, C. (2025) · arXiv
“…benchmarks overlook these aspects as they follow the Complex Reasoning Search paradigm, persistently deepening multi-hop reasoning and focusing exclusively on queries with specific factoid answers (Xbench-Team, 2025; Zhou et al., 2025b; Chen et al., 2025; Zhou et al., 2025a; Tao et al., 2025).”
Metadata
- Authors
- Zhou, P., Leon, B., Ying, X., Zhang, C., Shao, Y., Ye, Q., Chong, D., Jin, Z., Xie, C., Cao, M., Gu, Y., Hong, S., Ren, J., Chen, J., Liu, C., & Hua, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.19314
- OpenAlex
- W6929371760
- Semantic Scholar
- 34c20505098fd48ff19e4b5c4901febdd5d27181
- Cited by
- 0
- References
- 34
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, web browsing, benchmarking, multi-hop question answering, LLM agents, agentic search, information retrieval, information reconciliation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
