CoChat
Sign Up
PreprintCited by 10Information Retrieval and Search BehaviorWeb Data Mining and AnalysisTopic Modeling

Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge

Open access
Gou, Boyu, Huang, Zanming, Ning, Yuting, Gu, Yu, Lin, Michael, Qi, Weijian, Kopanev, Andrei, Yu, Botao +18 more·2025·arXiv
DOI 10.48550/arxiv.2506.21506arXiv 2506.21506
Open this paper in CoChat — free accountOpen PDF

Abstract

Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.

AI snapshot

Written from the paper’s full text.

TL;DR

Mind2Web 2 introduces 130 long-horizon, time-varying agentic search tasks and an Agent-as-a-Judge framework using tree-structured rubrics to evaluate answer correctness and source attribution. It evaluates ten frontier systems plus humans, finding OpenAI Deep Research best at 0.54 partial completion and 0.28 success rate versus human 0.79 and 0.54.

Key findings

  • The benchmark contains 130 tasks built with at least 1,000 hours of human labor; rubric trees average 50 total nodes (max 603), and on Subset30 humans averaged 18 minutes, 8 websites, and 110 webpages per task (max 44 minutes, 31 websites, 375 webpages).
  • Human evaluation of judge agents on 15 tasks found 35 discrepancies out of 720 leaf-node verifications; after excluding human errors and one source-inconsistency case, only 7 of 720 reflected actual Verifier errors, for 99.03% correctness.
  • OpenAI Deep Research was best among evaluated systems: partial completion 0.54 ± 0.04, success rate 0.28 ± 0.04, Pass@3 0.40, time 8.40 ± 0.71 minutes, answer length 559 ± 19 words; humans scored 0.79 ± 0.01, 0.54 ± 0.07, Pass@3 0.83, 18.40 ± 1.61 minutes, 186 ± 27 words.
  • On 57 explicitly time-varying tasks, most systems performed worse than on other tasks, while OpenAI Operator and humans achieved on-par or superior performance; hallucination rate was 23% for OpenAI Deep Research and at least 50% for other systems, defined as tasks with Invalid Attribution or Unsupported Answer.

Methods

  • Tasks were collected through proposal, refinement, and validation stages, covering 6 broad domains and 24 sub-domains; the benchmark is split into a public development set of 10 tasks and a private test set of 120 tasks, with a human performance study on a 30-task subset where each task was completed by three of seven participants.
  • Evaluation uses task-specific judge agents implementing tree-structured rubrics with critical, non-critical, and sequential nodes; leaf nodes yield binary scores aggregated upward via a gate-then-average strategy, producing Partial Completion and Success Rate metrics, with Extractor and Verifier tools using OpenAI o4-mini.
  • Judge-agent scripts were generated with Claude-3.7-Sonnet plus self-debug and self-reflection, then validated in two human stages; ten frontier agentic search systems were run three times per task, with Pass@3, completion time, and answer length reported, and answers collected between April and June 2025.

Limitations

  • Stated in the paper: task coverage excludes vague or highly subjective queries; URL-based attribution assumes cited URLs are truthful and credible; evaluation assumes critical information can be attributed to individual webpages; LLM-based extraction and verification may introduce errors; and focus on mostly closed-source systems limits interpretation of performance differences and inference costs.
  • Stated in the paper: website changes may affect tasks over time, though the authors commit to periodic review and maintenance; they also keep the private test set and script-generation pipeline hidden to reduce contamination and reward-model abuse.
  • Evident from the text: the framework excludes tasks requiring simultaneous multi-page verification, video understanding, non-English websites, complex reasoning or external tools, and rapidly changing answers; human performance and some timing measurements are limited to Subset30, and the human judge-agent evaluation used only 15 tasks and one evaluator.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.21506

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

28 of 31 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 2

    Many existing benchmarks for autonomous web agents focus on tasks of a moderate horizon (e.g., up to 10 actions) that can be completed on a single website.

    “Many benchmarks have been proposed for autonomous web agents [8, 21, 39, 40, 46] but they primarily focus on tasks of a moderate horizon (e.g., up to 10 actions) that can be completed on a single website.”
    Quote in the paperSubstantial claim4 of 5 sources checked
    • cites [8] [8] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-28114, 2023. - source
    • cites [21] [21] Xing Han Lu, Zdenˇ ek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pages 33007-33056. PMLR, 2024. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [39] [39] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. - source

      Section 2.1 (p. 3) says existing sandbox benchmarks have limited website diversity, and describes WebVoyager tasks as generally simple. It does not establish that existing benchmarks primarily contain tasks of up to 10 actions or that their tasks are confined to a single website. The 10-step threshold in §2.2 (p. 4) classifies tasks in this paper’s new benchmark, not existing benchmarks. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [40] [40] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022. - source

      The Related Work section says MiniWoB lacks long-range decision-making across multiple pages, supporting the comparison with shorter web-agent tasks. It does not establish that most benchmarks are limited to a single website or give a horizon of up to 10 actions. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [46] [46] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. ICLR, 2024. - source

      The Introduction describes WebArena’s tasks as long-horizon, and Related Work contrasts its benchmark with existing web-agent benchmarks. Neither passage establishes that existing benchmarks primarily require up to 10 actions or that their tasks can be completed on a single website. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 1 Introduction · p. 2

    To facilitate automated evaluation, existing cross-website search benchmarks focus on tasks with predefined, time-invariant answers, oftentimes just a single answer string.

    “However, to facilitate automated evaluation, a common compromise was made: they focus on tasks with predefined, time-invariant answers , oftentimes just a single answer string.”
    Quote in the paperSubstantial claim4 of 4 sources checked
    • cites [23] [23] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. - source

      The Introduction says GAIA uses unique factual answers to allow simple automatic evaluation, and §3.1 says its questions have a short, single correct answer. Those passages support the link between answer format and evaluation, but they describe GAIA—not a common compromise among existing cross-website search benchmarks. The paper also discusses changing web evidence; it does not establish that those benchmarks require time-invariant answers. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [34] [34] Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Iyyer. Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025. - source

      The Introduction (p. 1) says each BEARCUBS question has a unique, short answer, making evaluation trivial. It does not say that existing cross-website search benchmarks generally use predefined, time-invariant answers, or that they adopted this format to enable automated evaluation. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
    • cites [41] [41] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711 . - source

      Section 3.1 (page unavailable) says AssistantBench requires closed-form answers that can be automatically verified and are unlikely to change quickly. Section 3.4 (page unavailable) says it uses reference answers, but allows strings, numbers, dictionaries, and up to five answers. These passages support the evaluation trade-off for AssistantBench, not the broader claim that existing cross-website benchmarks commonly make it or often require a single answer string. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [36] [36] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. - source

      Section 2.3 (Grading; page not specified) says BrowseComp uses a single short reference-answer string for grading, and the benchmark description says this makes answers easy to verify. The paper does not establish that existing cross-website search benchmarks commonly make this compromise or that their answers are time-invariant. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Interpretationreasoning
    § 1 Introduction · p. 2

    The complexity of agentic search answers is far beyond what conventional LLM-as-a-Judge methods are used for.

    “The complexity is far beyond what conventional LLM-as-a-Judge [45] methods are used for.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [45] [45] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023. - source
  • Premisecitation
    § 2 Related Work · p. 3

    Several recent benchmarks focus on search over the open web but limit the benchmark to tasks with predefined, time-invariant answers, oftentimes just a single answer string.

    “Several recent benchmarks have a stronger focus on search over the open web [23, 34, 36, 37, 41]. However, for the feasibility of automated evaluation, these benchmarks have made a common compromise: they limit the benchmark to tasks with predefined, time-invariant answers , oftentimes just a single answer string.”
    Quote in the paperSubstantial claim5 of 5 sources checked
    • cites [23] [23] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. - source

      The Introduction and §3.1 describe GAIA questions that often require searching the open web and have a unique factual answer for automatic evaluation. The passage on challenges of relying on the web says evidence can change over time; the authors try to use durable evidence or specify its date. That supports an aim of stable answers, but not a strict limit to time-invariant tasks. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [34] [34] Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Iyyer. Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025. - source

      The Introduction (p. 1) says BEARCUBS tests information-seeking on the live web and gives each question a unique short answer to make evaluation easy. Section 3 (p. 3) likewise describes short, easy-to-evaluate answers. The copy does not establish that those answers are time-invariant; it says the dataset will be updated and questions with invalid trajectories replaced. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [36] [36] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. - source

      The Abstract describes BrowseComp as a web-browsing benchmark, and §2.3 (Grading) says every reference answer is a single short string used to grade predicted answers. Our copy does not establish that all BrowseComp answers are time-invariant, so that qualifier is not verified for this benchmark. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [37] [37] Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572, 2025. -

      Section 3 (p. 3) describes WebWalkerQA as a web-traversal benchmark built from predefined question–answer pairs, and §3.3 (p. 5) says it compares responses with ground-truth answers for evaluation. It does not establish that the answers are time-invariant: §6.1 (p. 9) says the benchmark draws on official websites with dynamically updated information. The claimed common compromise across several benchmarks is therefore not fully supported by this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [41] [41] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711 . - source

      The Introduction describes AssistantBench as an open-web benchmark, and §3.4 says it uses gold answers for automatic evaluation. But §6.4 says only 43.5% of its tasks are static; the rest have answers that may change. Section 3.4 also allows numbers, lists, dictionaries, and up to five answers, so the paper does not support treating its tasks as uniformly time-invariant or single-string answers. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 2 Related Work · p. 3

    Most existing benchmarks for web agents focus on evaluating whether an agent can autonomously perform certain processes on a single website.

    “Most existing benchmarks for web agents focus on evaluating whether an agent can autonomously perform certain processes on a single website [8, 13, 17, 21, 39, 40, 46].”
    Quote in the paperSubstantial claim5 of 7 sources checked
    • cites [8] [8] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-28114, 2023. - source

      The Introduction says prior work often operates on limited, pre-specified websites and supports specific tasks. But Table 1 lists multiple websites or environments for most datasets it compares; the paper does not establish that most benchmarks evaluate processes on a single website. No page numbers are available in our copy. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [13] [13] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864-6890, 2024. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [17] [17] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024. - source

      The Language-Guided Web Agent Benchmarks section describes existing benchmarks that evaluate agents acting on web pages or in simulated web environments. It does not say that most benchmarks confine those tasks to a single website; the paper’s discussion of multi-site tasks concerns VisualWebArena itself. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [21] [21] Xing Han Lu, Zdenˇ ek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pages 33007-33056. PMLR, 2024. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [39] [39] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. - source

      The Introduction (p. 2) says most existing benchmarks use cached website snapshots or a limited number of simulated websites, supporting the claim that their website coverage is narrow. Section 2.1 (p. 3) discusses evaluating agents on web tasks, but it does not establish that most benchmarks test autonomous completion of processes on a single website; it says some offline benchmarks restrict agents to an annotated trajectory. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [40] [40] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022. - source
    • cites [46] [46] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. ICLR, 2024. - source

      The Introduction and Table 4 (§5.1; page unavailable) describe existing benchmarks for evaluating agents on web-based tasks. They do not establish that most such benchmarks confine tasks to a single website; the paper describes cross-website tasks in WebArena, but that does not substantiate the claim about most prior benchmarks. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 59
  • May combine two claims. 33

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 2

    Several benchmarks cover cross-website search tasks, including BrowseComp from OpenAI.

    “Several benchmarks cover cross-website search tasks [23, 34, 41], including most recently BrowseComp [36] from OpenAI.”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [23] [23] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. - source
    • cites [34] [34] Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Iyyer. Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.07919, 2025. - source
    • cites [41] [41] Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711 . - source
    • cites [36] [36] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. - source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2 Related Work · p. 3

    BrowseComp targets tasks that are hard to solve but easy to verify, often with a unique unambiguous string answer.

    “It specifically targets tasks that are hard to solve but easy to verify (e.g., the answer is often a unique, unambiguous string but may require combing through hundreds of webpages to find it).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [36] [36] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 Related Work · p. 4

    PaperBench adopts a tree-structured rubric manually written by human experts to evaluate the replication of AI research.

    “PaperBench [35] (a concurrent work) is most related to ours in that it also adopts a tree-structured rubric, though it is manually written by human experts and used to evaluate the replication of AI research.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [35] [35] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. - source

  • May combine two claims.

    Interpretationcitation
    § 2 Related Work · p. 4

    The complexity of agentic search is far beyond what a few LLM calls can evaluate, necessitating an Agent-as-a-Judge approach.

    “However, the complexity of agentic search is far beyond what a few LLM calls can evaluate, necessitating an Agent-as-a-Judge approach [35, 47].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [35] [35] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. - source

      Appendix A.2 (p. 13) describes evaluating complex outputs against rubrics with hundreds of nodes, and Related Work (p. 9) reports that agent-based judges have been more accurate than non-agent LLM judges on certain tasks. But the paper evaluates AI research replication, not agentic search, and Section 4 (p. 6) develops a simple LLM-based judge rather than saying an Agent-as-a-Judge approach is necessary. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [47] [47] Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agentas-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024. source

      The Introduction argues that evaluating agentic systems requires feedback across their step-by-step thought and action trajectories, and proposes Agent-as-a-Judge; the Abstract reports that it outperforms LLM-as-a-Judge on code-generation tasks. The paper does not establish that agentic search specifically is beyond evaluation by a few LLM calls, so the citing sentence extends its conclusion to a different task and a particular number of calls. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 1 Introduction · p. 2

    Approximately at least 1,000 hours of human labor were spent to construct Mind2Web 2, including tasks and evaluation scripts.

    “Approximately, at least 1 , 000 hours of human labor are spent to construct the benchmark, including the tasks and their evaluation scripts.”
    • Also stated in: § 3.5 Benchmark Statistics · p. 6 · plain
      “In total, the construction of this benchmark (including both task collection and judge-agent development) involves at least 1 , 000 hours of human labor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    Mind2Web 2 is a benchmark of 130 realistic, high-quality, long-horizon tasks requiring real-time web browsing and extensive information synthesis, constructed with over 1,000 hours of human labor.

    “we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1 , 000 hours of human labor.”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “It consists of 130 high-quality tasks across diverse practical domains.”
    • Also stated in: § 3.5 Benchmark Statistics · p. 6 · plain
      “we collect a total of 130 carefully curated tasks, each accompanied by a carefully developed judge-agent script.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.2 Main Results · p. 8

    ChatGPT Search and Perplexity Pro Search are the weakest systems, primarily limited by their restricted search horizon and relatively shallow information synthesis abilities inherent to LLMs.

    “Unsurprisingly, ChatGPT Search and Perplexity Pro Search emerge as the weakest systems, primarily limited by their restricted search horizon and relatively shallow information synthesis abilities inherent to LLMs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    The rubric trees in Mind2Web 2 are highly complex, with an average of 50 nodes and a maximum of 603 nodes.

    “Due to the complexity of our tasks, the rubric trees are also highly complex, with an average of 50 nodes and a max of 603 nodes (Table 2 (a)).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Main Results · p. 9

    Most agent systems accurately interpret the task and articles and achieve better performance than humans in that task.

    “In contrast, most agent systems accurately interpret the task and articles and achieve better performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Main Results · p. 7

    Most tasks in Mind2Web 2 are conceptually straightforward but tedious, posing substantial challenges for both agent systems and human participants, resulting in low success rates (up to 28% for agents and 54% for humans).

    “As shown in Table 3, while most tasks in Mind2Web 2 are conceptually straightforward, their tedious nature poses substantial challenges not only for the agent systems but also for human participants, resulting in low success rates (up to 28 %for agents and 54 %for humans).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 3.5 Benchmark Statistics · p. 6

    Mind2Web 2 is the only agentic search benchmark to date focusing on long-horizon, time-varying tasks.

    “As discussed in §2, Mind2Web 2 is the only agentic search benchmark to date focusing on long-horizon, time-varying tasks, and is made possible due to our advanced Agent-as-a-Judge evaluation methodology.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 4.2 Main Results · p. 8

    The benefit of increased inference time is especially evident when comparing systems within the same family (e.g., Grok and Perplexity), given that they presumably share the same underlying models.

    “The benefit is especially evident when comparing systems within the same family (e.g., Grok and Perplexity), given that they presumably share the same underlying models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Grok and Perplexity systems presumably share the same underlying models.

  • May combine two claims.

    Findingreasoning
    § 4.3 Error Analysis · p. 9

    ChatGPT Search executes only limited search steps and applies simple information synthesis by LLMs without using advanced tools, making it challenging to find and integrate all necessary information.

    “For example, ChatGPT Search executes only limited search steps and applies simple information synthesis by LLMs without using advanced tools (e.g., Python interpreters in many Deep Research systems), making it challenging for it to find and integrate all necessary information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 4.2 Main Results · p. 8

    OpenAI Operator and human participants achieve on-par or superior performance on time-varying tasks.

    “Interestingly, OpenAI Operator and human participants, both excelling at interacting with live websites, achieve relatively on-par or superior performance on time-varying tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Main Results · p. 9

    OpenAI Deep Research achieves 50-70% of human performance while spending less than half the time.

    “The best-performing system, OpenAI Deep Research, already achieves 50 -70 % of human performance while spending less than half the time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.4 Human Evaluation of Judge Agents · p. 10

    The evaluator fully agrees with all 15 rubrics.

    “The evaluator fully agrees with all the 15 rubrics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § D.5 Human Evaluation of Judge Agents · p. 23

    o4-mini is capable of serving as the Extractor and Verifier.

    “Empirically we have found o4-mini capable of serving as the Extractor and Verifier.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Human Evaluation of Judge Agents · p. 10

    27 of the 35 discrepancies arise from human evaluator errors; the original judgments are correct.

    “Upon further validation, we discover that 27 of the discrepancies arise from human evaluator errors; the original judgments are correct.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § D.3 Rubric and Judge Agent Generation · p. 22

    Iterative debugging and self-reflection stages are indispensable and highly useful for generating judge-agent scripts.

    “Empirically, we observe these iterative debugging and self-reflection stages to be indispensable and highly useful”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 Human Evaluation of Judge Agents · p. 10

    For two rubrics, the evaluator offers minor suggestions regarding strictness of partial scoring.

    “However, for two rubrics, the evaluator offers minor suggestions regarding the strictness of partial scoring.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5 Conclusions · p. 11

    The empirical analysis reveals both potential and current limitations of agentic search systems.

    “Our comprehensive empirical analysis, spanning AI-based search engines, deep research systems, and web agents, reveals both their potential and current limitations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingreasoning
    § 4.3 Error Analysis · p. 9

    Deep Research systems (e.g., OpenAI Deep Research) have surpassed humans in avoiding criteria violations.

    “Interestingly, Deep Research systems (e.g., OpenAI Deep Research) have already surpassed humans in this regard”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.3 Error Analysis · p. 10

    Humans sometimes commit synthesis errors due to carelessness when overwhelmed by large volumes of information.

    “Humans also sometimes commit synthesis errors due to carelessness when overwhelmed by large volumes of information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Human Evaluation of Judge Agents · p. 11

    Excluding human mistakes and source inconsistency, only 7 out of 720 nodes reflect actual Verifier errors, achieving a correctness rate of 99.03%.

    “Excluding human mistakes and the source inconsistency case, only 7 out of 720 nodes reflect actual Verifier errors, achieving an exceptional correctness rate of 99 . 03 %.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.3 Error Analysis · p. 10

    Operator struggles to follow instructions to provide attribution, in contrast to AI search systems.

    “Therefore, in contrast to AI search systems, Operator struggles to follow our instructions to provide attribution.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Web agents are usually designed or trained on web navigation or citation-free information-seeking tasks.

  • May combine two claims.

    Findingown data
    § 4.4 Human Evaluation of Judge Agents · p. 11

    Of the remaining eight discrepancies, three result from Verifier mistakes, four from hidden critical information in collapsed content, and one from inconsistent information across sources.

    “Of the remaining eight discrepancies, we find: - Three cases result from mistakes by the Verifier, due to overly strict or lenient judgments. - Four cases occur because critical information for attribution evaluation is hidden within collapsed content sections of webpages, making it inaccessible during automated retrieval-a known limitation that we have sought to avoid during task validation. -”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § D.4 Two-Stage Validation of Judge Agents · p. 22

    Necessary adjustments to the prompts of the Extractor and the Verifier, as well as changes to allow reasonable edge cases, were often found empirically.

    “Empirically, we often find necessary adjustments to the prompts of the Extractor and the Verifier, as well as making necessary changes to allow reasonable edge cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § A Limitations · p. 16

    The constraint that critical information can be attributed to individual webpages has not prevented developing a large, diverse, and meaningful benchmark.

    “However, this constraint has not prevented us from developing a large, diverse, and meaningful benchmark.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Note: · p. 46

    The URL-based LLM-as-a-Judge cannot obtain information from webpages that load content dynamically with additional clicks or interactions.

    “Please be aware that our current URL-based LLM-as-a-Judge cannot obtain information from webpages that load content dynamically with additional clicks or interactions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § Note: · p. 46

    Tasks involving video understanding or non-English websites are currently not supported.

    “Tasks involving video understanding or non-English websites are currently not supported.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § F.2 Additional Case Studies · p. 31

    Hugging Face Open Deep Research is the only open-source solution included in the experiments.

    “Notably, Hugging Face Open Deep Research is the only open-source solution included in our experiments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § F.2 Additional Case Studies · p. 31

    Humans have no issue with completeness: all human answers fully fulfill task requirements without omission, and without hallucinations of webpage URLs.

    “Intuitively and empirically, we find that humans indeed have no issue with completeness: all human answers fully fulfill task requirements without omission, and without hallucinations of webpage URLs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § 3.5 Benchmark Statistics · p. 6

    Even though there are only 130 tasks, each task contains dozens to hundreds of fine-grained evaluation nodes, thus still providing sufficient differentiation power.

    “It is worth noting that even though there are only 130 tasks, each task contains dozens to hundreds of fine-grained evaluation nodes, thus still providing sufficient differentiation power.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2 Related Work · p. 3

    BrowseComp's strategy sidesteps the challenge of automatically evaluating complex, time-varying answers but at the cost of systematically deviating from the true user query distribution.

    “This strategy is adopted to sidestep the challenge of automatically evaluating complex, time-varying answers, but at the cost of systematically deviating from the true user query distribution.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 4.2 Main Results · p. 8

    Web agents navigate more complex and noisier environments, manage complex action spaces, and handle substantially longer and more intricate context than agents primarily leveraging search APIs.

    “Compared to agents primarily leveraging search APIs, web agents navigate more complex and noisier environments, manage complex action spaces, and handle substantially longer and more intricate context.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 2 Related Work · p. 4

    This work has more sophisticated score aggregation methods beyond simple weighted averaging due to the diversity of tasks.

    “We also have more sophisticated score aggregation methods beyond simple weighted averaging due to the diversity of our tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 1 Introduction · p. 2

    A rubric evaluates two main aspects of an answer: correctness (whether the answer satisfies all requirements of the task) and attribution (whether every statement in the answer can be attributed to the cited sources).

    “At a high level, a rubric evaluates two main aspects of an answer: correctness (i.e., whether the answer satisfies all the requirements of the task) and attribution (i.e., whether every statement in the answer can be attributed to the cited sources).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Error Analysis · p. 9

    Systems not optimized for 'Deep Research' inherently lack sufficient capabilities to complete long-horizon tasks.

    “On the other hand, they inherently lack sufficient capabilities to complete these long-horizon tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 2 Related Work · p. 3

    A gap exists in simultaneously evaluating the broad spectrum of agentic search systems, which this work aims to bridge.

    “However, there is yet a benchmark designed to simultaneously evaluate this broad spectrum of agentic search systems, a gap that our work aims to bridge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.2 Main Results · p. 8

    Integrating web browsing into agentic search systems is important and likely contributes substantially to OpenAI Deep Research's superior performance over other Deep Research systems.

    “These collectively highlight the importance of integrating web browsing into agentic search systems, likely contributing substantially to OpenAI Deep Research's superior performance over the other Deep Research systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.5 Benchmark Statistics · p. 6

    The benchmark is split into a public development set of 10 tasks (with task descriptions and evaluation scripts) and a private test set of 120 tasks (with only task descriptions) to reduce risk of data contamination and judge agents being abused as reward models.

    “To reduce the risk of data contamination and our judge agents being abused as reward models for reinforcement learning, we split our benchmark into a public development set ( 10 tasks), which includes both the task descriptions and evaluation scripts, and a private test set ( 120 tasks), that only includes the task descriptions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction · p. 2

    Agentic search systems typically produce lengthy, time-varying answers ranging from hundreds to thousands of words on these tasks.

    “Agentic search systems typically produce lengthy, time-varying answers (e.g., the product catalog of a shopping website constantly changes) ranging from hundreds to thousands of words on these tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 4.2 Main Results · p. 9

    Humans are not perfect at many complex tasks because they are subject to cognitive fatigue and limited working memory.

    “Humans are not perfect at many of such complex tasks because we are subject to cognitive fatigue and limited working memory.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Humans are subject to cognitive fatigue and limited working memory.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 4.3 Error Analysis · p. 9

    Systems not optimized for 'Deep Research' often exhibit early termination behaviors, showing a degree of laziness.

    “On the one hand, systems not optimized for 'Deep Research' often exhibit early termination behaviors, showing a degree of laziness .”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Task Collection · p. 4

    Only tasks independently validated by at least two validation experts are included in Mind2Web 2.

    “Only tasks independently validated by at least two validation experts are included in Mind2Web 2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given
    § 2 Related Work · p. 3

    Existing web agent benchmark tasks tend to be short (e.g., less than 10 actions) and transactional (e.g., purchasing a flight ticket).

    “The tasks tend to be short (e.g., less than 10 actions) and transactional (e.g., purchasing a flight ticket).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § E.3 Human Performance on Subset30 · p. 25

    Each task was assigned to three completers without prior knowledge of the task (excluding creators or reviewers).

    “Each task is assigned to three completers without prior knowledge of the task (excluding creators or reviewers).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § C.4 Future Maintenance of the Benchmark · p. 19

    The Agent-as-a-Judge evaluation can reliably evaluate time-varying answers.

    “our Agent-as-a-Judge evaluation can reliably evaluate time-varying answers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § D.4 Two-Stage Validation of Judge Agents · p. 22

    In the second stage, scripts underwent practical validation against real answers collected from various agent systems.

    “In the second stage, scripts undergo practical validation against real answers collected from various agent systems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § E.3 Human Performance on Subset30 · p. 25

    Completers first undertook two simplified trial tasks from Mind2Web 2 to ensure quality of human performance.

    “To ensure the quality of human performance, completers first undertake two simplified trial tasks from Mind2Web 2.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Error Analysis · p. 10

    The incorrect URL reporting by Operator may be partially due to the challenge of generating answers grounded in a long context.

    “which may be partially due to the challenge of generating answers grounded in a long context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § C.3 Task Collection Pipeline · p. 18

    Tasks for Mind2Web 2 were collected and refined under a three-stage pipeline (Proposal, Refinement, Validation) to ensure adherence to task principles and evaluation practicability.

    “We collect and refine tasks for Mind2Web 2 under a three-stage pipeline: Proposal , Refinement , and Validation , ensuring adherence to the task principles as well as evaluation practicability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § A Limitations · p. 16

    The paper conducts human evaluations of judge-agent outputs, confirming overall reliability of LLM-based judgments.

    “We further conduct human evaluations of judge-agent outputs, systematically assessing and confirming the overall reliability of LLM-based judgments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § C.4 Future Maintenance of the Benchmark · p. 19

    Mind2Web 2 primarily involves broad information-seeking goals, allowing flexibility for agents in selecting sources.

    “our benchmark primarily involves broad information-seeking goals, allowing flexibility for agents in selecting sources.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § C.3 Task Collection Pipeline · p. 18

    Expert annotators reviewed and iteratively refined each proposed task together with the initial proposers.

    “Expert annotators further review and iteratively refine each proposed task together with the initial proposers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Error Analysis · p. 9

    The system errors of HF Open Deep Research may also apply to other open-source agent systems that leverage off-the-shelf models without post-training.

    “which may also apply to other open-source agent systems that simply leverage off-the-shelf models to build deep research systems without post-training the underlying models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 4.3 Error Analysis · p. 10

    Retrieval errors can result from failing to retrieve relevant information.

    “Retrieval errors can result from failing to retrieve relevant information.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § C.3 Task Collection Pipeline · p. 19

    Only tasks successfully passing validation from at least two validators were included in the final benchmark.

    “Only tasks successfully passing validation from at least two validators are included in the final benchmark.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § F.1 Error Analysis · p. 26

    Attribution errors are often related to hallucinations in LLM-based agent systems.

    “Attribution errors are often related to hallucinations in LLM-based agent systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § D.4 Two-Stage Validation of Judge Agents · p. 22

    The remaining answers were held out as an additional set to further verify the generalization of the finalized evaluation scripts.

    “The remaining answers are held out as an additional set to further verify the generalization of the finalized evaluation scripts”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § B Broader Impacts and Ethical Considerations · p. 16

    By emphasizing rigorous evaluation through structured rubrics and explicit verification of source attribution, Mind2Web 2 facilitates the development of transparent and accountable agentic search systems.

    “By emphasizing rigorous evaluation through structured rubrics and explicit verification of source attribution, Mind2Web 2 facilitates the development of transparent and accountable agentic search systems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § D.5 Human Evaluation of Judge Agents · p. 23

    One human evaluator, familiar with the tasks but not having reviewed the judge agents, evaluated judge agents on 15 sampled tasks.

    “We involve one human evaluator, who is familiar with our tasks but has never reviewed the judge agents, to conduct a human evaluation of judge agents with 15 sampled tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § E.2 Webpage Pre-caching for Evaluation · p. 24

    Because webpage contents may evolve, caching is essential for establishing a stable reference for evaluation, reflecting the exact state of online sources at the time answers are generated.

    “Given that webpage contents may evolve, especially for time-varying tasks (e.g., fluctuating product prices), this caching step is essential for establishing a stable reference for evaluation, reflecting the exact state of online sources at the time answers are generated.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Webpage contents may evolve over time, especially for time-varying tasks.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning
    § C.3 Task Collection Pipeline · p. 18

    Refinement ensures that tasks remain realistic and challenging yet clearly defined and objectively verifiable.

    “Refinement ensures that tasks remain realistic and challenging yet clearly defined and objectively verifiable.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 4.4 Human Evaluation of Judge Agents · p. 10

    The human evaluator had no prior experience in judge-agent development but deep understanding of task criteria, ensuring unbiased and accurate assessments.

    “We involve a human evaluator who has no prior experience in judge-agent development but possesses a deep understanding of the task criteria gained from participating in error analysis, thus ensuring unbiased and accurate assessments.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § E.3 Human Performance on Subset30 · p. 25

    The data collection provides critical benchmark statistics regarding task complexity and human effort.

    “This data collection provides critical benchmark statistics regarding task complexity and human effort.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Error Analysis · p. 9

    Humans struggle to remain patient and careful due to the tedious and demanding nature of the tasks.

    “primarily due to the tedious and demanding nature of the tasks, where humans appear to struggle to remain patient and careful.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D.5 Human Evaluation of Judge Agents · p. 23

    Each judge agent underwent a two-stage careful validation and refinement.

    “for each judge agent, we have done a two-stage careful validation and refinement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § 4.3 Error Analysis · p. 10

    A potential reason for invalid attribution is that agents generate URLs directly without actually accessing the webpages.

    “One potential reason could be that agents generate URLs directly without actually accessing the webpages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § C.3 Task Collection Pipeline · p. 19

    Each task underwent validation by two more independent annotators.

    “Finally, each task undergoes validation by two more independent annotators.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § D.5 Human Evaluation of Judge Agents · p. 23

    A human evaluation study was conducted to further validate the reliability of the judge agents.

    “to further validate the reliability of our judge agents, we conduct this human evaluation study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § D.4 Two-Stage Validation of Judge Agents · p. 22

    A two-stage manual validation process was conducted to ensure the quality and robustness of the generated judge-agent scripts.

    “We conduct a two-stage manual validation process to ensure the quality and robustness of the generated judge-agent scripts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § C.2 Design Principles of Tasks · p. 18

    Each task is carefully validated and iteratively refined by initial proposers, refiner experts and validation experts to ensure full compliance before final inclusion.

    “Each task is carefully validated and iteratively refined by initial proposers, refiner experts and validation experts to ensure full compliance before final inclusion into Mind2Web 2.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § E.2 Webpage Pre-caching for Evaluation · p. 24

    Caching provides quick, consistent and reliable access to webpage screenshots and text for verification.

    “This caching provides quick, consistent and reliable access to webpage screenshots and text for verification.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § E.1 System Selection and Settings · p. 23

    Each agent system was independently run and evaluated three times per task to test variability in outputs.

    “To test the variability in outputs, we independently run and evaluate each agent system three times per task.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § E.2 Webpage Pre-caching for Evaluation · p. 24

    Webpage contents referenced in agent-generated answers were pre-fetched and cached to ensure stability and efficiency.

    “To ensure stability and efficiency, we pre-fetch and cache webpage contents referenced in agent-generated answers.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Loading webpages on-the-fly during evaluation can introduce significant overhead.

  • May combine two claims.

    Interpretationreasoning
    § 4.4 Human Evaluation of Judge Agents · p. 11

    The success of the judge agents is attributed to the tree-structured rubric design, the agentic code generation pipeline, and the rigorous human refinement process.

    “We attribute this success to our tree-structured rubric design that cleanly decomposes the complex evaluation, the agentic code generation pipeline for generating judge agents, as well as the rigorous human refinement process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § E.3 Human Performance on Subset30 · p. 25

    Only completers who successfully followed instructions and met quality expectations in the trials could participate in the formal human study.

    “Only completers who have successfully followed instructions and met quality expectations in these trials can participate in the formal human study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § D.4 Two-Stage Validation of Judge Agents · p. 22

    In the first stage, trained annotators independently inspected each generated judge-agent script.

    “In the first stage, trained annotators independently inspect each generated judge-agent script.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § F.2 Additional Case Studies · p. 31

    Several challenges likely contributed to Operator's poor performance, including insufficient long-term reasoning, planning, and potential grounding failures.

    “Several challenges likely contributed to this issue, including insufficient long-term reasoning, planning, and potential grounding failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § F.2 Additional Case Studies · p. 31

    This likely contributes substantially to the frequent occurrence of these system errors, which may also apply to other open-source agents that utilize current off-the-shelf models.

    “This likely contributes substantially to the frequent occurrence of these system errors, which may also apply to other open-source agents that utilize current off-the-shelf models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Notes: · p. 50

    A task may allow for multiple reasonable rubric trees.

    “A task may allow for multiple reasonable rubric trees.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § F.2 Additional Case Studies · p. 31

    Some human mistakes are unlikely to occur in answers from capable agents.

    “Notably, some of these human mistakes are unlikely to occur in answers from capable agents.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given
    § F.2 Additional Case Studies · p. 31

    Tasks in Mind2Web 2 are intentionally designed to be tedious while ensuring complete feasibility for human participants.

    “Tasks in Mind2Web 2 are intentionally designed to be tedious while ensuring complete feasibility for human participants.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § F.2 Additional Case Studies · p. 31

    Operator lacks optimized long-term memory mechanisms for managing retrieved information and associated sources.

    “In particular, it lacks optimized long-term memory mechanisms for managing retrieved information and associated sources.”
    Quote in the paperNot a substantial claim
85 of 85

  • The paper relies on this without support.

    Current web search is inherently user-driven and places significant cognitive load on users.

    “Current web search is inherently user-driven : it retrieves pieces of information but relies on users to interpret and assemble those pieces. That places a significant cognitive load on users, especially as the complexity of the digital world grows.”

  • The paper relies on this without support.

    Many websites dynamically render information not indexed by search engines based on user interaction.

    “For example, many websites dynamically render information not indexed by search engines based on user interaction.”

  • The paper relies on this without support.

    Humans are subject to cognitive fatigue and a limited working memory.

    “After all, humans are subject to cognitive fatigue and a limited working memory.”

  • The paper relies on this without support.

    Grok and Perplexity systems presumably share the same underlying models.

    “given that they presumably share the same underlying models.”

  • The authors state this as an assumption.

    The evaluation framework relies on URL-based attribution, presupposing that cited URLs provide truthful and credible information.

    “Our evaluation framework relies on URL-based attribution, presupposing that cited URLs provide truthful and credible information, despite potential misinformation on the web.”

  • The authors state this as an assumption.

    The evaluation and task design assume critical information can be attributed to individual webpages.

    “Additionally, our evaluation and task design assume critical information can be attributed to individual webpages, which may not always hold true for all possible tasks.”

  • The authors state this as an assumption.

    Each verification of attribution can be conducted on a single webpage.

    “We currently assume each verification of attribution can be conducted on a single webpage.”

  • The paper relies on this without support.

    Web agents are usually designed or trained on web navigation or citation-free information-seeking tasks.

    “Web agents are usually designed or trained on web navigation or citation-free information-seeking tasks.”

  • The paper relies on this without support.

    LLMs with massive parametric memory sometimes tend to directly produce or hallucinate information without conducting actual searches.

    “Moreover, LLMs with massive parametric memory sometimes tend to directly produce or hallucinate information without conducting actual searches”

  • The paper relies on this without support.

    Humans are subject to cognitive fatigue and limited working memory.

    “Humans are not perfect at many of such complex tasks because we are subject to cognitive fatigue and limited working memory.”

  • The authors state this as an assumption.

    Webpage contents may evolve over time, especially for time-varying tasks.

    “Given that webpage contents may evolve, especially for time-varying tasks (e.g., fluctuating product prices)”

  • The paper relies on this without support.

    A small number of websites block automated visits, preventing automatic content retrieval.

    “A small number of websites block automated visits, preventing automatic content retrieval.”

  • The paper relies on this without support.

    Loading webpages on-the-fly during evaluation can introduce significant overhead.

    “loading webpages on-the-fly during evaluation can introduce significant overhead.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing benchmarks face two primary limitations: reliance on static datasets or answer keys, and use of non-expert or automated evaluation including coarse metrics or auto-generated rubrics.

    “To summarize, existing benchmarks thus face two primary limitations: a reliance on static datasets or answer keys [15, 16], and the use of non-expert or automated evaluation, including coarse metrics [20] or auto-generated rubrics [8, 11, 24].”

    From ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents (2025) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • WebWalkerQA is assigned the AssistantBench reference

    Minor

    Table 1 cites reference [41] for WebWalkerQA, but [41] is the AssistantBench paper. The reference list identifies [37] as the WebWalker paper.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. [13][13] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864-6890, 2024. -
  2. [14][14] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. -
  3. [18][18] Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. Attributionbench: How hard is automatic attribution evaluation? In Findings of the Association for Computational Linguistics ACL 2024, pages 14919-14935, 2024. -
  4. [21][21] Xing Han Lu, Zdenˇ ek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pages 33007-33056. PMLR, 2024. -

Cited by

Sorted by citations

Authors
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., Song, C. H., Wu, J., Chen, S., Moussa, H. N., Zhang, T., Xie, J., Li, Y., Xue, T., Liao, Z., Zhang, K., Zheng, B., Cai, Z., Rozgic, V., Ziyadi, M., Sun, H., & Su, Y.
Affiliations
  • The Ohio State University (US)
  • Adobe Systems (United States) (US)
  • Amazon (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
192393-192451
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2506.21506
arXiv
2506.21506
OpenAlex
W4415183952
Semantic Scholar
87e0d4f6e4fb8c16d69c4444de7319d7698b1711
Cited by
3
References
0
Citations by year
2026: 3
FWCI
3.53
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
agentic search, benchmark, information synthesis, source attribution, deep research, error analysis, human performance comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review