CoChat
Sign Up
Cited by 18Web Data Mining and AnalysisInformation Retrieval and Search BehaviorTopic Modeling

WebWalker: Benchmarking LLMs in Web Traversal

Open access
Wu, Jialong, Yin, Wenbiao, Jiang, Yong, Wang, Zhenglin, Xi, Zekun, Fang, Runnan, Zhang, Linhai, He, Yulan +3 more·2025·Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2025.acl-long.508
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieval-augmented generation (RAG) demonstrates remarkable performance across tasks in open-domain question-answering. However, traditional search engines may retrieve shallow content, limiting the ability of LLMs to handle complex, multi-layered information. To address it, we introduce WebWalkerQA, a benchmark designed to assess the ability of LLMs to perform web traversal. It evaluates the capacity of LLMs to traverse a website's subpages to extract high-quality data systematically. We propose WebWalker, which is a multi-agent framework that mimics human-like web navigation through an explore-critic paradigm. Extensive experimental results show that WebWalkerQA is challenging and demonstrates the effectiveness of RAG combined with WebWalker, through the horizontal and vertical integration in real-world scenarios.

Verified 2026-10-22 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

WebWalkerQA is a 680-query bilingual benchmark for evaluating LLMs on web traversal, where agents must click through subpages to answer single-source and multi-source questions. The authors also propose WebWalker, a multi-agent explorer-critic framework that outperforms ReAct and Reflexion but still remains below 40% overall accuracy with GPT-4o.

Key findings

  • WebWalkerQA is challenging: the best WebWalker configuration, using GPT-4o, reaches 37.50 overall accuracy and 4.67 action count, and no method surpasses 40% overall accuracy.
  • WebWalker outperforms Reflexion, which outperforms ReAct; closed-source models outperform open-source models, and accuracy declines as depth or number of sources increases.
  • RAG systems struggle on WebWalkerQA: the best commercial system, Tongyi, reaches 40.73 overall accuracy, while close-book Gemini-1.5-Pro and o1-preview reach only 8.08 and 9.85 overall accuracy, respectively.
  • Combining WebWalker with a naive RAG system improves performance across all difficulty levels, especially multi-source QA, and scaling action count K ∈ {5, 10, 15, 20, 25} leads to better performance.

Methods

  • Benchmark construction: 680 QA pairs over 1373 webpages from conference, organization, education, and game domains; bilingual Chinese/English; generated with GPT-4o annotation followed by human verification; includes single-source and multi-source QAs with difficulty defined by page depth.
  • WebWalker framework: a multi-agent explorer-critic system; the explorer follows a ReAct thought-action-observation paradigm over markdown page content and clickable HTML buttons, while the critic maintains memory and decides when to answer; all methods run zero-shot with a maximum of 15 actions.
  • Evaluation: accuracy is judged by GPT-4 with CoT prompting, and action count is measured for successful executions; baselines are ReAct and Reflexion, with backbones including GPT-4o, Qwen-Plus, and Qwen2.5-{7,14,32,72}B-Instruct; commercial and open-source RAG systems are also compared.

Limitations

  • Stated: WebWalkerQA currently has 680 high-quality QA pairs, with about 14k silver QA pairs not carefully human-verified; the work uses only HTML-DOM and not visual modalities such as screenshots; WebWalker is prompting-only without agent tuning; better integration with RAG systems is left for future work.
  • Evident from the text: evaluation relies on GPT-4 as an evaluator rather than exact match; actions are constrained to “click”; the benchmark covers four domains and two languages, so it does not represent all web environments; in the RAG integration, the root URL is provided for WebWalker to execute.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.acl-long.508

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 8 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 1

    Previous HTML-based instruction-action benchmarks fail to capture the complexities of real-world scenarios where relevant information is buried deep within web pages and requires multiple layers of interaction.

    “Additionally, they fail to capture the complexities of real-world scenarios where relevant information is buried deep within web pages and requires multiple layers of interaction.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Deng et al., 2023 source

      Section 2.3, “Comparison with Existing Work and Research Challenges” (page unavailable), says earlier studies used simpler environments and step-by-step directives, whereas real-world pages have complex DOM structures and require greater planning. This supports the comparison with previous benchmarks, but the copy does not specifically say that relevant information is buried deep within pages or requires multiple layers of interaction. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Zhou et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § 1 Introduction · p. 2

    WebWalkerQA contains 680 queries.

    “We construct a challenging benchmark, WebWalkerQA , which is composed of 680 queries from four real-world scenarios across over 1373 webpages.”
    • Also stated in: § 3.2 Data Statistics · p. 4 · plain
      “Through such data construction method with LLM and human participation, we obtain 680 questionanswer pairs for WebWalkerQA.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction · p. 2

    For information-seeking tasks, vertical exploration within the page proves to be beneficial.

    “Extensive experiments show that the WebWalkerQA is challenging, and for information-seeking tasks, vertical exploration within the page proves to be beneficial.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction · p. 2

    WebWalkerQA spans over 1373 webpages.

    “We construct a challenging benchmark, WebWalkerQA , which is composed of 680 queries from four real-world scenarios across over 1373 webpages.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 5.2 Main Results · p. 7

    For open-source models, performance and efficiency improve as the model size increases.

    “For open-source models, performance and efficiency improves as the model size increases.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 18

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    Previous HTML-based instruction-action benchmarks face challenges such as excessively noisy information and overly long inputs, which can significantly hinder performance due to limitations in long-context understanding.

    “Previous works related to web pages focus on addressing action-based requests, such as Mind2Web (Deng et al., 2023) and WebArena (Zhou et al., 2024a); these HTML-based instruction-action benchmarks face challenges such as excessively noisy information and overly long inputs, which can significantly hinder performance due to limitations in long-context understanding.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Deng et al., 2023 source

      Section 2.3 describes Mind2Web as an action-based web benchmark and says its real-world pages contain long, complex DOM structures that challenge modeling. Sections 4.2 and 3 describe cleaning HTML and say feeding raw pages directly to LLMs can be infeasible or costly. The paper does not establish that noisy or long inputs significantly hinder performance specifically because of limitations in long-context understanding. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Zhou et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3.2 Data Statistics · p. 5

    In terms of language distribution, Chinese and English account for 60.5%, 39.5%, respectively.

    “In terms of language distribution, Chinese and English account for 60.5%, 39.5%, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 5.4 Error Assessment · p. 8

    For incorrect execution, errors can be categorized into three types: refusal to answer or locating wrongly, reasoning error, and exceeding the maximum number of steps K.

    “For incorrect execution, errors can also be categorized into three types: refusal to answer or locating wrongly, reasoning error, and exceeding the maximum number of steps K .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Main Results · p. 7

    The closed-source models outperform the open-source models in both performance and efficiency.

    “The closed-source models outperform the opensource models in both performance and efficiency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 3.2 Data Statistics · p. 5

    The proportions of the conference, organization, education, and game domains are 24.0%, 7.9%, 46.3%, and 24.0%, respectively.

    “The proportions of the conference, organization, education, and game domains are 24.0%, 7.9%, 46.3%, and 24.0%, respec-”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Main Results · p. 7

    Even the best-performing WebWalker using GPT-4o as its backbone does not surpass 40% accuracy, highlighting the challenge posed by WebWalkerQA.

    “Even the best-performing WebWalker using GPT-4o as its backbone does not surpass 40%, highlighting the challenge posed by WebWalkerQA.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1 RAG Performance on WebWalkerQA · p. 9

    Both commercial and open-sourced RAG systems exhibit relatively poor performance on WebWalkerQA, with the best result coming from Tongyi, which only reaches 40%.

    “Both commercial and open-sourced RAG systems exhibit relatively poor performance on WebWalkerQA, with the best result coming from Tongyi, which only reaches 40%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Main Results · p. 7

    WebWalker outperforms Reflexion, which in turn outperforms ReAct.

    “Our proposed WebWalker framework outperforms Reflexion, which in turn outperforms React.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    Even with the most powerful LLMs as the backbone, WebWalker's performance on WebWalkerQA remains suboptimal, validating the challenge posed by WebWalkerQA.

    “However, even with the most powerful LLMs as the backbone, its performance on WebWalkerQA remains suboptimal, thereby validating the challenge posed by WebWalkerQA.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3.2 Data Statistics · p. 4

    WebWalkerQA contains two types of data: multi-source and single-source QAs.

    “WebWalkerQA contains two types of data: multi-source and single-source QAs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § G.2 Time Cut-off · p. 20

    The cutoff date for o1's temporal data is October 2023, rendering it unable to provide answers regarding web information published beyond this point.

    “the cutoff date for o1's temporal data is October 2023, rendering it unable to provide answers regarding web information published beyond this point.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.2 WebWalker Combined with RAG System · p. 9

    After integrating WebWalker (built on Qwen-2.5-Plus) into the naive RAG system, performance improved across all difficulty levels, especially in the multi-source category.

    “It is observed that, after the integration, performance has improved across all difficulty levels, especially in the multi-source category.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Scaling Up on Action Count K · p. 10

    Larger values of K (from 5 to 25) lead to better performance when tracing source information, validating the feasibility of vertical scaling within a certain range.

    “larger values of K lead to better performance, validating the feasibility of vertical scaling within a certain range.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A Limitations and Discussion · p. 16

    The authors possess a collection of approximately 14k silver QA pairs that are not yet carefully human-verified.

    “we possess a collection of approximately 14k silver QA pairs, which, although not yet carefully humanverified, can serve as supplementary training data”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 5.1 Experimental Setting · p. 7

    WebWalker and the two baselines all operate in a zero-shot setting.

    “Considering the context limitation of models, our proposed WebWalker, along with two baselines, all operate in a zero-shot setting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    Integrating external search engines via retrieval-augmented generation (RAG) allows LLMs to retrieve up-to-date information from the web, enhancing their utility in dynamic, knowledge-intensive scenarios.

    “While their knowledge base remains static post-training, integrating external search engines via retrieval-augmented generation (RAG) allows LLMs to retrieve up-to-date information from the web, enhancing their utility in dynamic, knowledgeintensive scenarios (Lewis et al., 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lewis et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 1 Introduction · p. 2

    Vertical exploration of pages offers a promising direction for scaling inference time in RAG systems.

    “(iii) Vertical exploration of pages offers a promising direction for scaling inference time in RAG systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5.2 Main Results · p. 7

    As the model size increases, the action count grows, indicating that larger LLMs have enhanced long-range information-seeking ability.

    “We only counted the action count ( A.C. ) from correct executions, and as the model size increases, the A.C. grows, indicating that larger LLMs have enhanced long-range information-seeking ability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Data Collection · p. 4

    Multi-source QA tasks cannot be easily exploited by search engine shortcuts.

    “Notably, the multi-source QAtasks can not be easily exploited by search engine shortcuts (Mavi et al., 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mavi et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § 3.1 Data Collection · p. 4

    GPT-4o performs initial annotations, followed by crowd-sourced human annotators conducting quality control and filtering.

    “In the first stage, GPT-4o (OpenAI, 2022b), performs initial annotations, followed by a second stage, where crowd-sourced human annotators conduct quality control and filtering to refine the final results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites OpenAI, 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 5.3 Results across Domains and Languages · p. 8

    The framework performs similarly in both Chinese and English, as the models employed are both pre-trained and supervised-fine-tuned in a bilingual setting.

    “The framework performs similarly in both Chinese and English, as the models we employed are both pre-trained and supervised-fine-tuned in a bilingual setting.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLMs employed are both pre-trained and supervised-fine-tuned in a bilingual setting.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 3.1 Data Collection · p. 4

    The data collection uses a two-stage funnel annotation strategy combining LLM-based and human annotation.

    “To make the annotation process cost-efficient and accurate, we employ a two-stage funnel annotation strategy, combining LLM-based and human annotation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.4 Error Assessment · p. 8

    The small ReAct model tends to 'give up' and exhibits characteristics of impatience.

    “It tends to ' give up ' and exhibits characteristics of impatience .”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.3 Web Traversal Task and Evaluation · p. 5

    WebWalkerQA can be evaluated from both performance and efficiency perspectives, using question-answering accuracy as the performance metric and action count of successful agentic executions answering correctly as the efficiency metric.

    “WebWalkerQA can be evaluated from both performance and efficiency perspectives. using questionanswering accuracy ( acc. ) as the performance metric and the action count ( A.C. ) of successful agentic executions answering correctly as the efficiency metric.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.1 Data Collection · p. 4

    Human annotators rewrite and calibrate questions and answers to ensure QA pairs are correct and consistent.

    “After the synthetic queries are generated by LLM, human annotators can rewrite and calibrate the questions and answers to ensure the QA pairs are correct and consistent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.3 Results across Domains and Languages · p. 8

    In the conference domain, the framework demonstrates relatively superior performance, likely due to the more explicit and directive nature of the button information, which facilitates more straightforward inferences.

    “In the domain of conference , the framework demonstrates relatively superior performance, likely due to the more explicit and directive nature of the button information, which facilitates more straightforward inferences.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5.2 Main Results · p. 7

    Increasing the model size or introducing reflection on the process of each action can address certain problems requiring multi-step solutions, thereby enabling long-distance task-solving capabilities in web traversal tasks.

    “We observe that increasing the model size or introducing reflection on the process of each action can address certain problems requiring multi-step solutions, thereby enabling long-distance task-solving capabilities in web traversal tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 3.3 Web Traversal Task and Evaluation · p. 5

    GPT-4 is used as the evaluator, determining correctness of responses by comparing the predicted answer with the ground truth using CoT prompting.

    “We use GPT-4 as the evaluator, which determines the correctness of responses by comparing the predicted answer with the ground truth using CoT prompting strategy (Wei et al., 2022) 4 .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wei et al., 2022 source

  • May combine two claims.

    Interpretationreasoning
    § A Limitations and Discussion · p. 16

    WebWalker can function independently as a web information retrieval assistant for a given webpage, or seamlessly integrate with RAG systems to expand their scope.

    “WebWalker can function independently as a web information retrieval assistant for a given webpage or seamlessly integrate with RAG systems to expand their scope.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data
    § 6.3 Scaling Up on Action Count K · p. 10

    Scaling the process of digging through links could represent a potential direction for vertical exploration in RAG systems.

    “Findings (iii) : Scaling the process of digging through links could represent a potential direction for vertical exploration in RAG systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6.2 WebWalker Combined with RAG System · p. 9

    WebWalker can seamlessly integrate into standard RAG systems to acquire deep information and enhance problem-solving capabilities.

    “WebWalker can seamlessly integrate into standard RAG systems to acquire deep information and enhance problemsolving capabilities.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § C.1 Commercial Systems · p. 16

    The five commercial RAG systems (Doubao, ERNIE-4.0-8K, Tongyi, Kimi, and Gemini-Search) were all accessed through their business-oriented API interfaces to ensure reproducibility.

    “are all accessed through their businessoriented API interfaces to ensure reproducibility.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § A Limitations and Discussion · p. 16

    Under the agentic RAG paradigm, the click action proves to be highly effective.

    “Under the agentic RAG paradigm, the click action proves to be highly effective.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given
    § 6.1 RAG Performance on WebWalkerQA · p. 9

    WebWalkerQA is built on official websites with dynamically updated information, whereas pre-trained models rely on static knowledge limited by a cutoff date and lack dynamic updates.

    “WebWalkerQA is built on official websites with dynamically updated information, while pre-trained models rely on static knowledge limited by a cutoff date and lack dynamic updates”
    Quote in the paperSubstantial claim
34 of 34

  • The paper relies on this without support.

    Traditional online search engines perform horizontal searches of queries and may not effectively trace deeper content embedded within websites.

    “However, traditional online search engines, e.g. , Google or Bing, perform horizontal searches of queries and may not effectively trace the deeper content embedded within websites.”

  • The paper relies on this without support.

    The selected domains (conference, organization, education, and game) provide authoritative information and rich clickable content with substantial depth for exploration.

    “These domains are selected because they provide authoritative information relevant to their respective fields, and their pages contain rich clickable content, offering substantial depth for exploration.”

  • The paper relies on this without support.

    The LLMs employed are both pre-trained and supervised-fine-tuned in a bilingual setting.

    “The framework performs similarly in both Chinese and English, as the models we employed are both pre-trained and supervised-fine-tuned in a bilingual setting.”

  • The paper relies on this without support.

    The LLM-based CoT-QA evaluator grades student answers correctly, so the reported accuracy figures reflect true correctness.

    “CoT-QA Evaluator You are a teacher grading a quiz. You are given a question, the context the question is about, and the student's answer. You are asked to score the student's answer as either CORRECT or INCORRECT, based on the context.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Domain percentages total 102.2%

    Minor

    The four listed domains are presented as WebWalkerQA's domain distribution, but their percentages sum to 102.2%, not 100%. This exceeds what rounding to one decimal place could explain.

    Reported 100 · recomputed 102.2

  • Example golden path does not lead to its listed source page

    Minor

    Figure 8 lists the industry-track page as a source, but its golden path goes through `student_research_workshop` rather than `industry_track`. The path and source URL therefore identify different pages for that part of the example.

  • Model count is stated as both nine and six

    Minor

    The experimental setting says nine models were validated, while the main-results section says the results cover six LLMs. The listed backbones also comprise six models: two closed-source and four open-source.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.
  2. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=WE\_vluYUL-X .

Cited by

Sorted by citations

Related

Authors
Wu, J., Yin, W., Jiang, Y., Wang, Z., Xi, Z., Fang, R., Zhang, L., He, Y., Zhou, D., Xie, P., & Huang, F.
ORCID
Venue
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
10290-10305
DOI
10.18653/v1/2025.acl-long.508
OpenAlex
W4412945435
Cited by
18
References
0
Citations by year
2026: 16 · 2025: 2
FWCI
20.94
Type
conference paper
Language
en
Open access
Locations
  • Annual Meeting of the Association for Computational Linguistics (ACL) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, benchmarking, web search, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review