CoChat
Sign Up
PreprintCited by 3Information Retrieval and Search BehaviorTopic ModelingNatural Language Processing Techniques

Search Arena: Analyzing Search-Augmented LLMs

Open access
Miroyan, Mihran, Wu, Tsung-Han, King, Logan, Li, Tianle, Pan, Jiayi, Hu, Xinyan, Chiang, Wei-Lin, Angelopoulos, Anastasios N. +3 more·2025·arXiv
DOI 10.48550/arxiv.2506.05334arXiv 2506.05334
Open this paper in CoChat — free accountOpen PDF

Abstract

Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static encyclopedic sources are not always appropriate and reliable. To assess performance across different settings, we conduct cross-arena analyses by testing search-augmented LLMs in a general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research. Our dataset and code are available at: https://github.com/lmarena/search-arena.

AI snapshot

Written from the paper’s full text.

TL;DR

Search Arena is a crowd-sourced, large-scale human-preference dataset of over 24,000 multi-turn interactions with search-augmented LLMs, with 12,652 preference votes across 13 models and ~70 languages. The paper finds that user preference is positively associated with citation count and certain source types, even when citations are irrelevant, and that web search helps in search-intensive settings while not hurting general chat performance.

Key findings

  • The released dataset contains 24,069 conversations and 12,652 human preference votes from 11,650 users across 136 countries; 22.4% of conversations are multi-turn, and prompts span ~70 languages with 11% multilingual prompts.
  • User preference is positively associated with response length (β_length = 0.334) and citation count (β_citations = 0.209); supporting citations (β_support = 0.29) and irrelevant citations (β_irrelevant = 0.27) are both positively correlated with preference, while contradicting citations are not significant.
  • Source-type coefficients show positive associations for tech-related platforms (β_tech = 0.073), community platforms (β_community = 0.061), and social media (β_social = 0.057), but a negative association for Wikipedia (β_wiki = -0.071).
  • Cross-arena tests: in Text Arena, search vs. non-search Gemini-2.5 Pro had 544 battles with 245 ties (45%), 143 search-preferred (26%), and 156 non-search-preferred (28%), not statistically significant (p = 0.244); in Search Arena, a non-search model had 315 battles with 99 ties (31%), 126 search-preferred (40%), and 90 non-search-preferred (29%), significantly underperforming (p = 0.009).

Methods

  • Crowd-sourced Search Arena platform deployed for seven weeks (March 18–May 8, 2025) within Chatbot Arena, showing anonymized side-by-side search-augmented LLM responses in multi-turn sessions and collecting user preference votes.
  • Preference analysis uses Bradley-Terry models with normalized pairwise feature differences, including response length, search context size, citation count, citation source categories, and citation attribution labels.
  • Intent analysis uses a nine-category taxonomy annotated by GPT-4.1, validated against human labels with Cohen’s kappa = 0.812 on top-2 intents; citation attribution was automated on ~100 English conversations per intent, collecting over 20,000 web documents across 780 conversations and labeling claim-citation pairs as Support, Irrelevant, or Contradict.

Limitations

  • The paper states that crowd-sourced data may reflect demographic skews, may not represent the broader population, and that not all users vote and human judgments are inherently subjective.
  • The paper states that releasing conversational data with human preferences requires careful consideration of privacy and equitable access; it anonymized responses, randomized left/right placement, and obtained user consent.
  • Evident from the text: citation attribution analysis was limited to roughly 100 English conversations per intent and excluded URLs that could not be scraped or accessed; cross-arena comparisons rely on relatively small battle counts (544 in Text Arena, 315 in Search Arena).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.05334

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 27 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    User preferences are positively associated with the number of citations.

    “user preferences are influenced by the number of citations”
    • Also stated in: § 1 Introduction · plain
      “users prefer responses with a higher number of cited sources (Figure 4).”
    • Also stated in: § 3.2 Citation Features · hedged
      “We find a positive and statistically significant coefficient for the number of citations (βcitations = 0.209), indicating that users favor responses with more references.”
    Quote in the paperSubstantial claim
  • Findingown data

    The quality in search settings is significantly affected if solely relying on the model's parametric knowledge.

    “the quality in search settings is significantly affected if solely relying on the model's parametric knowledge.”
    • Also stated in: § 1 Introduction · plain
      “the internal parametric knowledge of models alone is not sufficient in search-intensive settings.”
    Quote in the paperSubstantial claim
  • Findingown data

    User preferences vary across cited sources.

    “user preferences vary across cited sources”
    • Also stated in: § 3.2 Citation Features · plain
      “Figure 6 (Left) shows source category coefficient estimates with 95% confidence intervals”
    Quote in the paperSubstantial claim
  • Findingown data

    Community-driven platforms are generally preferred as cited sources.

    “community-driven platforms are generally preferred”
    Quote in the paperSubstantial claim
  • Findingown data

    User preferences are positively associated with the number of irrelevant citations (citations that do not directly support the attributed claims).

    “even when the cited content does not directly support the attributed claims”
    • Also stated in: § 1 Introduction · plain
      “we also observe a positive association between user preference and the number of irrelevant citations (β = 0.273).”
    • Also stated in: § Citation Attribution. · plain
      “both supporting (βsupport = 0.29) and irrelevant (βirrelevant = 0.27) claims are positively correlated with user preference.”
    Quote in the paperSubstantial claim
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 32
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Web search does not degrade and may even improve performance in non-search settings.

    “web search does not degrade and may even improve performance in non-search settings”
    • Also stated in: § 1 Introduction · hedged
      “web search augmentation does not hurt and may improve performance across settings”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The Search Arena prompts span 71 languages with English (56.4%), Russian (11.4%), and Chinese (6.8%) as the top three.

    “The prompts span 71 languages with English (56.4%),Russian (11.4%), and Chinese (6.8%) as the top three.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Factual Lookup (p-value = 0.012) and Info Synthesis (p-value = 0.095), the difference is in favor of the search model.

    “for Factual Lookup (p-value = 0.012) and Info Synthesis (p-value = 0.095), the difference is more expressed in favor of the search model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The Search Arena dataset includes 11,650 unique users across 136 countries, with the United States (18.7%), Russia (9.9%), Germany (5.9%), and China (5.5%) as the top four.

    “The dataset includes 11,650 unique users across 136 countries, with the United States (18.7%),Russia (9.9%), Germany (5.9%), and China (5.5%) as the top four.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: IP-derived country codes accurately indicate users' countries.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    22.4% of all Search Arena conversations contain more than one turn.

    “Search Arena conversations are multi-turn, with 22.4% of all conversations containing more than one turn.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Search Arena provides a user intent taxonomy in the context of search-enabled human-AI interactions.

    “We also introduce a user intent taxonomy in the context of search-enabled human-AI interactions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    After filtering, we retain approximately 24,000 conversations, about half of which include human preference votes.

    “we retain approximately 24,000 conversations, about half of which include human preference votes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The platform recorded an average of roughly 800-1,500 conversations per day over the 7-week data collection period.

    “the platform recorded an average of roughly 800-1,500 conversations per day.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There are 3,288 Search Arena conversations with 2 turns, 966 with 3 turns, and 460 with 4 turns.

    “Specifically, there are 3,288 conversations with 2 turns, 966 with 3 turns, and 460 with 4 turns.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    In irrelevant citation cases, models may fabricate connections, cite tangentially related sources, or present inferred claims that subtly deviate from the source content.

    “in irrelevant citation cases, models may fabricate connections, cite tangentially related sources, or present inferred claims that subtly deviate from the source content.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Factual lookup prompts are typically shorter (17.2 words on average), whereas the remaining set is associated with longer, more complex queries (66.7 words on average).

    “we find that factual lookup prompts are typically shorter (17.2 words on average), whereas the remaining set is associated with longer, more complex queries (66.7 words on average).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Search Arena spans around 70 languages, with 11% multilingual prompts.

    “around 70 languages (including 11% multilingual prompts)”
    • Also stated in: § 2.1 Linguistic and Conversational Diversity · plain
      “The prompts span over 70 languages, with 30 represented by at least 10 conversations.”
    • Also stated in: § 2.1 Linguistic and Conversational Diversity · plain
      “More than 11% of the prompts are multilingual.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the Text Arena setting, non-search model responses to Text Processing prompts, compared to search model responses, more often use explicitly structured formatting (numbered or bulleted lists, headings) to organize information (74.4% vs 56.4%, p = 0.1).

    “• Uses explicitly structured formatting (numbered or bulleted lists, headings) to organize information (74.4% vs 56.4%) (p = 0.1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Text Arena, we collected 544 battles between the search and non-search versions of the model, yielding 245 ties (45%), 143 search-preferred (26%), and 156 non-search preferred votes (28%).

    “We collected 544 battles between the search and non-search versions of the model, yielding 245 ties (45%), 143 search-preferred (26%), and 156 non-search preferred votes (28%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Search Arena prompt lengths vary across intent categories, with Text Processing (85.1 words) and Analysis (72.7 words) prompts generally longer, and Explanation (24.7 words) and Factual Lookup (16.3 words) prompts shorter.

    “As shown in Figure A5, Search Arena prompt lengths vary across intent categories.As expected, Text Processing (85.1 words) and Analysis (72.7 words) prompts are generally longer, while Explanation (24.7 words) and Factual Lookup (16.3 words) prompts are shorter.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Search Arena data includes 11,650 unique users across 136 countries.

    “Search Arena data includes 11,650 unique users across 136 countries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Search Arena prompt lengths are more spread out and cover the range of BrowseComp and SimpleQA questions.

    “Search Arena prompt lengths are more spread out and cover the range of BrowseComp[61] and SimpleQA[60] questions.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [60]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Findingown data

    We collected over 20,000 web documents across 780 conversations for citation attribution analysis.

    “In total, we collected over 20,000 web documents across 780 conversations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Reasoning models tend to be more verbose except for sonar-reasoning.

    “Reasoning models tend to be more verbose except for sonar-reasoning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For all intent categories, the coefficient for response length difference in the Bradley-Terry model is statistically significant, but the effect is smallest for Factual Lookup prompts.

    “For all intent categories, the coefficient (effect) is statistically significant; however, as expected, the effect is smallest for Factual Lookup prompts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The distribution of battles across Search Arena models is not even due to different sampling weights and models not being added to the platforms at the same time.

    “The distribution is not even due to (1) different sampling weights and (2) models were not added to the platforms at the same time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    OpenAI's GPT-4o-based models are more biased towards mainstream news outlets.

    “OpenAI's GPT-4o-based models are more biased towards mainstream news outlets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Search Arena, 315 pairwise battles between a non-search model and selected search-augmented models were collected, comprising 99 ties (31%), 126 search-preferred (40%), and 90 non-search-preferred votes (29%).

    “We collected 315 pairwise battles between a non-search model and selected search-augmented models in Search Arena, with 99 ties (31%), 126 search-preferred (40%), and 90 non-search-preferred votes (29%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    English prompts account for 58.3% of Search Arena data.

    “with English prompts accounting for 58.3% of the data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Tie votes are less frequent across all intent categories in Search Arena compared with the Text Arena case study, indicating that differences between model responses are more expressed.

    “Compared with the case study in the Text Arena (Figure 8 (Left)), tie votes are less frequent across all categories, indicating that the differences between model responses are more expressed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Findingown data

    The number of irrelevant claim-citation pairs is positively correlated with user preferences.

    “The number of supporting and irrelevant pairs is positively correlated,”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Search improves performance on SimpleQA, while it degrades the score on ArenaHard-v2 for Gemini models.

    “Search improves performance on SimpleQA, while degrades the score on ArenaHard-v2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    On non-English prompts, the gap between the two top models increases, with sonar-reasoning-pro-high performing better on non-English prompts.

    “The gap between the two top models increases, with sonar-reasoning-pro-high performing better on non-English prompts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Each provider uses a distinct citation style, which may unintentionally reveal model identity.

    “Each provider uses a distinct citation style, which may unintentionally reveal model identity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The rankings in ArenaHard-v2 are different from Search Arena, with reasoning models having higher performance in search settings.

    “but the rankings are different, with reasoning models having higher performance in search settings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Search Arena chats are multi-turn, with 22.4% of conversations containing more than 1 turn.

    “Search Arena chats are multi-turn with 22.4% of conversations containing more than 1 turn.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The distribution of battles across Search Arena models is not even due to different sampling weights and models not being added to the platform at the same time.

    “The distribution is not even due to (1) different sampling weights and (2) models were not added to the platforms at the same time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The most prevalent topics in Search Arena include Technology Comparisons (22.0%), Market Analysis (12.3%), and Entertainment Characters (10.6%).

    “The most prevalent topics include Technology Comparisons (22.0%),Market Analysis (12.3%), and Entertainment Characters (10.6%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Search significantly improves model performance on SimpleQA, with performance saturating at around 90%.

    “Search significantly improves model performance on SimpleQA (models' performance saturates at around 90%).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [60]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The rankings of models differ between Search Arena and ArenaHard-v2; the three reasoning models rank lower in ArenaHard-v2.

    “However, the rankings differ; notably, the three reasoning models rank lower in ArenaHard-v2,”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    Approximately 2% of examples were flagged by the automated de-identification process using Google's DLP API.

    “Approximately 2% of examples were flagged by this process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Search Arena prompts span 71 languages.

    “Search Arena prompts span 71 languages.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Users prefer the search model for Factual Lookup and Info Synthesis queries in both settings, and the non-search model for Text Processing queries in Text Arena.

    “Users prefer the search model for Factual Lookup and Info Synthesis queries in both settings and the non-search model for Text Processing queries in Text Arena.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Reasoning models and models with high search context size tend to be more verbose.

    “Reasoning models and models with high search context size tend to be more verbose.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The effect of citation count on Guidance prompts is not significant, and the effect is largest on Analysis prompts.

    “The effect of citation count on Guidance (e.g., debugging, problem-solving) prompts is not significant.The effect is largest on Analysis prompts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ArenaHard-v2 shows greater performance variance and separability than SimpleQA.

    “In contrast, ArenaHard-v2 shows greater performance variance and separability,”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    On ArenaHard-v2, performance degrades with search, with non-search variants achieving higher scores.

    “However, performance degrades on ArenaHard-v2, with non-search variants achieving higher scores.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Response length distribution is uniform across OpenAI's models.

    “Response length distribution is uniform across OpenAI's models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Search Arena is multilingual, spanning over 70 languages.

    “The dataset is multilingual, spanning over 70 languages,”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After controlling for response length, number of citations, and citation sources, model scores and rankings tend to converge, particularly within model families, with the top three Perplexity models showing significant convergence.

    “We observe that the model scores and rankings tend to converge after the controls are applied.This convergence is particularly evident within model families: the top three models from Perplexity show significant convergence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Access to search enables LLMs to provide up-to-date, domain-specific, and factually verifiable responses.

    “Access to search enables LLMs to provide up-to-date, domain-specific, and factually verifiable responses [24,33].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [24]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [33]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    The annotation of user intent using GPT-4.1 achieved a Cohen's kappa of 0.812, indicating strong agreement between model- and human-annotated labels.

    “The resulting Cohen's kappa of 0.812 indicates strong agreement between model-and human-annotated labels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The most widely used datasets for evaluating search-augmented LLMs (SimpleQA and BrowseComp) primarily consist of single-turn, monolingual, fact-based queries and are relatively small in scale (typically ≤5k queries).

    “The most widely used datasets for evaluating these systems (SimpleQA [60] and BrowseComp [61]) primarily consist of single-turn, monolingual, fact-based queries and are relatively small in scale (typically ≤5k queries).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites SimpleQA [60]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites BrowseComp [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Users do not distinguish between supporting and irrelevant citations and generally prefer more citations.

    “Thus, users do not distinguish between supporting and irrelevant citations and generally prefer more citations, even if the citations do not directly support the claims.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    We obtained user consent at interaction time and enforced a strict privacy policy.

    “We obtained user consent at interaction time and enforced a strict privacy policy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductcitation

    Human judgments are inherently subjective.

    “human judgments are inherently subjective [7,12].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    The collected data may reflect demographic skews and may not be fully representative of the broader population.

    “the collected data may reflect demographic skews and may not be fully representative of the broader population”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Wikipedia articles are often very lengthy and broad, not directly relevant to a user's question.

    “Wikipedia articles are often very lengthy and broad, not directly relevant to a user's question”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    During the data collection period, no early access to data was granted to model providers, nor were any pre-release models deployed on the platform.

    “During the data collection period, no early access to data was granted to model providers, nor were any pre-release models deployed on the platform.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    We analyzed known biases (e.g., response length) and reported user demographics.

    “we also analyzed known biases (e.g., response length; Subsection 3.1) and reported user demographics in Figure A4 and Figure A8.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The non-search model often provides structured responses (e.g., numbered or bulleted lists, headings), which suggests that presentation style may impact user evaluations.

    “the non-search model often provides structured responses (e.g., numbered or bulleted lists, headings), which suggests that presentation style may impact user evaluations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We anonymized model responses and randomized their left/right placement to reduce known human biases.

    “we anonymized model responses and randomized their left/right placement to reduce known human biases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Combining web search with reasoning improves performance in Search Arena but may degrade performance on ArenaHard-v2 prompts.

    “suggesting that, while combining web search with reasoning improves performance in Search Arena, it may degrade performance on ArenaHard-v2 prompts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    All conversation data is released under a Creative Commons Attribution (CC-BY) license.

    “Per our Terms of Service (Figure A2), all conversation data is released under a Creative Commons Attribution (CC-BY) license.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    We took care to ensure ethical data usage, relying only on publicly accessible content and following best practices regarding rate limits and terms of service.

    “Overall, we took care to ensure ethical data usage, relying only on publicly accessible content and following best practices regarding rate limits and terms of service.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Each model was evaluated on the same randomly sampled subset of 500 questions, sampled once and shared across all models.

    “Due to the high cost of running search-augmented LLMs, we evaluated each model on the same randomly sampled subset of 500 questions.The subset was sampled once and shared across all models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The long-tail distribution of topics reflects the breadth of real-world usage of search-augmented LLMs.

    “The long-tail distribution reflects the breadth of real-world usage of search-augmented LLMs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Search-augmentation does not hurt performance in non-search settings and can improve responses to queries related to information retrieval and synthesis.

    “These cross-arena experiments demonstrate that search-augmentation does not hurt performance in non-search settings and can improve responses to queries related to information retrieval and synthesis.”
    Quote in the paperSubstantial claim
63 of 63

  • The paper relies on this without support.

    LLMs are constrained by their reliance on static training data, which prevents them from effectively handling time-sensitive questions, emerging topics, or niche domains.

    “However, the capabilities of these models are constrained by their reliance on static training data, which prevents them from effectively handling time-sensitive questions, emerging topics, or niche domains.”

  • The paper relies on this without support.

    Search-augmented LLMs represent a hybrid interface different from both standalone LLMs and traditional web search engines.

    “However, search-augmented LLMs represent a hybrid interface different from both: they not only retrieve information through web search but also rely on their reasoning and conversational capabilities.”

  • The paper relies on this without support.

    In the Text Arena, users typically assume that models operate in closed-book settings without access to external information, while in the Search Arena, user expectations are explicitly conditioned on the search setting.

    “In the Text Arena, users typically assume that models operate in closed-book settings without access to external information, while in the Search Arena, user expectations are explicitly conditioned on the search setting.”

  • The paper relies on this without support.

    Citations are central to the trustworthiness of web-grounded outputs in search-enabled scenarios.

    “Citations are central to the trustworthiness of web-grounded outputs in search-enabled scenarios and are a unique feature in Search Arena compared to the prior Text Arena [10,70].”

  • The paper relies on this without support.

    Existing benchmarks for search-augmented LLMs lack full human-AI interaction traces, including queries, retrieved documents, responses, and preferences.

    “For search augmented LLMs, existing benchmarks lack full human-AI interaction traces, including queries, retrieved documents, responses, and preferences”

  • The paper relies on this without support.

    Citation formatting can influence user preferences.

    “At the same time, citation formatting can influence user preferences.”

  • The paper relies on this without support.

    IP-derived country codes accurately indicate users' countries.

    “Users' demographic information in Search Arena, based on country codes extracted from IP addresses, is shown in Figure A4.”

  • The paper relies on this without support.

    LLM-based citation attribution judgments (Support, Irrelevant, Contradict) are accurate.

    “Each attribution was labeled as Support, Irrelevant, or Contradict.”

  • The paper relies on this without support.

    The 100 English prompts randomly sampled for taxonomy development are representative of the full dataset.

    “First, the three co-authors of the paper did open-text annotations on 100 English prompts randomly sampled from the collected dataset.”

  • The paper relies on this without support.

    The 150 samples drawn from the top three languages are representative for evaluating GPT-4.1 label quality.

    “we compared GPT-4.1 annotations with human labels on 150 samples drawn from the top three languages in our dataset (English, Russian, and Chinese).”

  • The paper relies on this without support.

    The approximately 100 conversations per intent category are representative for citation attribution analysis.

    “we analyzed citation attribution on approximately 100 conversations per intent category.”
11 of 11

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting factual-lookup prompt lengths

    Minor

    The main text gives an average of 17.2 words for Factual Lookup prompts; Appendix B gives 16.3 words.

  • BrowseComp citation points to a different paper

    Minor

    The text cites reference [61] for BrowseComp, but the bibliography lists BrowseComp as reference 62.

  • Non-search-preferred vote percentage is rounded incorrectly

    Minor

    In the 544 Text Arena battles, 156 non-search-preferred votes constitute about 28.7%, which rounds to 29%, not the reported 28%.

    Reported 28 · recomputed 28.676471

  • SimpleQA citation points to a different paper

    Minor

    The text cites reference [60] for SimpleQA, but reference 60 is a paper on chain-of-thought prompting; the bibliography lists the short-form factuality paper as reference 61.

  • Figure 4's right panel is described as the wrong relationship

    Minor

    The citation-count discussion says Figure 4's right panel shows response length, but the figure caption says that panel shows citation count. The caption assigns response length to the left panel.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie, and Ji-Rong Wen. The web can be your oyster for improving language models. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023. URL https://aclanthology.org/2023.findings-acl.46/.
  2. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  3. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference (TREC-3), 1994.
  4. Mark Steyvers, Heliodoro Tejeda Lemus, Aakriti Kumar, Catarina Belém, Sheer Karny, Xinyue Hu, Lukas Mayer, and Padhraic Smyth. The calibration gap between model and human confidence in large language models, 2024. URL https://api.semanticscholar.org/CorpusID: 267211649.

Cited by

Sorted by citations

Authors
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., & Gonzalez, J. E.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.05334
arXiv
2506.05334
OpenAlex
W4416139385
Semantic Scholar
6bb6f9157fc10bc509d99d15b934704a5e8ff232
Cited by
1
References
68
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • U.S. Department of Defense
  • Cisco Systems
  • Accenture
  • Nvidia
  • Samsung
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
web search, source credibility, citation influence, user preferences, parametric knowledge, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review