Deep Research Bench: Evaluating AI Web Research Agents
Open accessAbstract
Amongst the most common use cases of modern AI is LLM chat with web search enabled. However, no direct evaluations of the quality of web research agents exist that control for the continually-changing web. We introduce Deep Research Bench, consisting of 89 multi-step web research task instances of varying difficulty across 8 diverse task categories, with the answers carefully worked out by skilled humans. We provide a "RetroSearch" environment with a large frozen set of scraped web pages, and demonstrate that offline "RetroSearch" agents perform comparably to "live web" agents, enabling reliable evaluations of models over time. We provide robust agent tooling and scaffolding to benchmark major LLMs as they are released, including "thinking" models like o3 and Gemini 2.5 Pro. We include automated evaluations of the lengthy agent traces to report progress over time in hallucinations, tool use, and forgetting. Finally, we evaluate the major web research products branded as "Deep Research", "Deep Search", "Search", or "Research." Results are available on a public leaderboard at https://drb.futuresearch.ai/.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Web agent evaluation frameworks including WebArena and Mind2Web provide comprehensive assessment across thousands of tasks spanning 137 websites, revealing significant performance gaps in current LLM capabilities for complex web interactions.
“Web agent evaluation frameworks including WebArena and Mind2Web provide comprehensive assessment across thousands of tasks spanning 137 websites, revealing significant performance gaps in current LLM capabilities for complex web interactions”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [87]
Deep Research Bench evaluation reveals significant challenges in current systems' ability to conduct complex research tasks requiring synthesis across multiple information sources and reasoning over technical content.
“Deep Research Bench evaluation reveals significant challenges in current systems' ability to conduct complex research tasks requiring synthesis across multiple information sources and reasoning over technical content.”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [87]
BrowseComp, XBench, Deep Research Bench, and WideSearch focus on tasks that have fixed answers, such as specific entities or numbers.
“BrowseComp [Wei et al., 2025], XBench [Chen et al., 2025], Deep Research Bench [FutureSearch et al., 2025], WideSearch [Wong et al., 2025] focus on tasks that have fixed answers, such as specific entities or numbers.”
From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as FutureSearch et al., 2025
Fixed-answer setups capture whether an agent can locate and extract relevant information, but only partially reflect real-world user needs: they rarely test whether the system can decide what to look for under open-ended goals, or how to integrate findings into a coherent narrative.
“These setups capture whether an agent can locate and extract relevant information, but they only partially reflect real-world user needs: they rarely test whether the system can decide what to look for under open-ended goals, or how to integrate findings into a coherent narrative.”
From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as FutureSearch et al., 2025
These fixed-answer benchmarks primarily assess whether the agent can locate and retrieve the correct information, testing planning, reasoning, and discerning information sources.
“These benchmarks primarily assess whether the agent can locate and retrieve the correct information, testing the agent's abilities in planning, reasoning, and discerning information sources.”
From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as FutureSearch et al., 2025
Cited by
Sorted by citations
A Survey of Context Engineering for Large Language Models
Mei, L., Yao, J., Ge, Y., Wang, Y., Bi, B., Cai, Y., Liu, J., Li, M., Li, Z.-Z., Zhang, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.13334 · Cited by 13 · see citing works
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Li, M., Zeng, Y., Cheng, Z., Ma, C., & Jia, K. (2025) · arXiv · doi:10.48550/arxiv.2508.15804 · Cited by 1 · see citing works
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., & Subbiah, V. K. (2026) · arXiv · doi:10.48550/arxiv.2601.06007
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Li, R., Du, M., Xu, B., Zhu, C., Wang, X., & Mao, Z. (2026) · arXiv · doi:10.48550/arxiv.2601.08536
“BrowseComp [Wei et al., 2025], XBench [Chen et al., 2025], Deep Research Bench [FutureSearch et al., 2025], WideSearch [Wong et al., 2025] focus on tasks that have fixed answers, such as specific entities or numbers.”
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Gao, Y., Zhao, R., Deng, Y., & Zhang, W. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1249
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
- Authors
- FutureSearch, :, Bosse, N. I., Evans, J., Gambee, R. G., Hnyk, D., Mühlbacher, P., Phillips, L., Schwarz, D., & Wildman, J.
- ORCID
- Bosse, Nikos I.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2506.06287
- OpenAlex
- W4417114727
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- benchmarking, hallucinations, tool use, forgetting, multi-step reasoning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
