LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

May combine two claims.

They repeatedly observe failure modes such as mismatched in-text citations and references, missing or incomplete URLs, inconsistent citation formats, uncited references appearing in the bibliography, broken or incomplete table formatting, and out-of-order reference numbering.

“Across systems, we repeatedly observe failure modes that are trivial for humans but challenging for current agents, such as mismatched in-text citations and references (Figure 14), missing or incomplete URLs (Figure 15), inconsistent citation formats (Figure 16), uncited references appearing in the bibliography (Figure 17), broken or incomplete table formatting (Figure 18), and outof-order”

May combine two claims.

Gemini 2.5 Pro achieves the highest consistency score overall (76.5), with Perplexity Sonar Reasoning and Perplexity Sonar Reasoning Pro close behind (73.0 and 71.9).

“Gemini 2.5 Pro achieves the highest consistency score overall (76.5), with Perplexity Sonar Reasoning and Perplexity Sonar Reasoning Pro close behind (73.0 and 71.9).”

May combine two claims.

Each LiveResearchBench query is multi-faceted, specifying scope, audience, and format whenever appropriate, and many require real-time search at evaluation time.

“Each query is multi-faceted: it specifies scope, audience, and format whenever appropriate, and many require real-time search at evaluation time.”

Classified as probably not a substantial claim. May combine two claims.

DeepEval adopts Checklist-based for Presentation & Organization and Coverage & Comprehensiveness; Pointwise (additive) for Factual & Logical Consistency and Citation Association; Pairwise comparison for Analysis Depth.

“As a result, we adopt Checklist-based for ❶ Presentation & Organization and ❸ Coverage & Comprehensiveness; Pointwise (additive) for ❷ Factual & Logical Consistency and ❺ Citation Association; Pairwise comparison for ❹ Analysis Depth.”

May combine two claims.

Averaged by family, multi-agent deep research systems (69.5) outperform both single-agent web (62.8) and single-agent deep research.

“Averaged by family, multi-agent deep research systems (69.5) outperform both single-agent web (62.8) and single-agent deep research, indicating that collaborative decomposition and aggregation provide reliable gains.”

May combine two claims.

Open Deep Research achieves the highest average (73.6), followed by GPT-5 (72.7) and Deerflow+.

“At the system level, Open Deep Research achieves the highest average (73.6), followed by GPT-5 (72.7) and Deerflow+.”

Probably not a substantial claim: may not pass S4 (based).

DeepEval is a comprehensive evaluation suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis.

“To evaluate citation-grounded long-form reports, we introduce DeepEval , a comprehensive suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis.”

May combine two claims.

Longer reports are common, but not consistently better.

“Obs. ❶ Longer reports are common, but not consistently better.”

Classified as probably not a substantial claim. May combine two claims.

DeepEval's content-level metrics assess coverage and comprehensiveness, analysis depth, citation association, and factual accuracy; report-level metrics evaluate presentation & organization, and logical & factual consistency.

“Content-level metrics assess coverage and comprehensiveness, analysis depth, citation association, and factual accuracy. Report-level metrics evaluate presentation & organization, and logical & factual consistency.”

Classified as probably not a substantial claim.

DeepEval assesses both report- and content-level quality across six complementary dimensions: Presentation & Organization, Factual & Logical Consistency, Coverage & Comprehensiveness, Analysis Depth, Citation Association, and Citation Accuracy.

“DeepEval assesses both report- and content-level quality across six complementary dimensions: ❶ Presentation & Organization, ❷ Factual & Logical Consistency, ❸ Coverage & Comprehensiveness, ❹ Analysis Depth, ❺ Citation Association, and ❻ Citation Accuracy.”

May combine two claims.

Even with SoTA judges (Gemini-2.5 Pro and GPT-5), agreement with human judgments for single-rating LLM judge fell below 60%, and assigned scores exhibited large variability across runs.

“Even with SoTA judges (Gemini-2.5 Pro and GPT-5), agreement with human judgments fell below 60%, and the assigned scores exhibited large variability across runs, for example, differences exceeding 50 points when directly rating analysis depth.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LiveResearchBench was built with over 1,500 hours of human labor.

“Built with over 1,500 hours of human labor, LiveResearchBench provides a rigorous basis for systematic evaluation.”

Classified as probably not a substantial claim.

LiveResearchBench is designed as a user-centric, unambiguous benchmark for deep research tasks that require broad, real-time information gathering.

“LiveResearchBench is designed as a user-centric, unambiguous benchmark for deep research tasks that require broad, real-time information gathering.”

Classified as probably not a substantial claim. May combine two claims.

The example query in Figure 1 requires dynamic, up-to-date information search and specifies breadth, depth, audience, and format.

“For instance, the query in Figure 1 Create a comprehensive report about the evolution of artistic styles ...Cover major historical periods from Ancient civilizations up to the present {{date}}. ...Target the report for undergraduate-level students ...Structure as a formal academic report ...Include analysis of major artists, techniques, materials, and stylistic characteristics for each period and”

Probably not a substantial claim: may not pass S3 (load-bearing).

Citation Traceability checks whether all factual claims are linked to corresponding sources.

“❺ Citation Traceability. This metric checks whether all factual claims are linked to corresponding sources.”

May combine two claims.

Grok-4 Heavy Deep Research, though best on Coverage & Comprehensiveness, barely outperforms Open Deep Research in depth.

“Grok-4 Heavy Deep Research, though best on Coverage &Comprehensiveness, barely outperforms ODR in depth.”

May combine two claims.

Single-agent deep research models perform worst on Citation Association, primarily because many key statements remain uncited and URLs are often mismatched.

“By contrast, single-agent deep research models perform worst on Citation Association, primarily because many key statements remain uncited and URLs are often mismatched.”

Classified as probably not a substantial claim.

The checklists function as unit tests for the coverage dimension, verifying whether generated reports address all essential points in the query.

“These checklists function as unit tests for the coverage dimension, verifying whether generated reports address all essential points in the query.”

May combine two claims.

Four core task design principles for deep research are grounded in user survey and comparative analysis and ensure tasks are realistic, unambiguous, and evaluation-ready.

“We establish four core task design principles for deep research, grounded in user survey and comparative analysis, that ensure tasks are realistic, unambiguous, and evaluation-ready.”

May combine two claims.

Presentation is weakly correlated with association or consistency; visually well-organized reports may still contain inconsistent claims.

“Yet, presentation is weakly correlated with association or consistency-visually well-organized reports may still contain inconsistent claims.”

May combine two claims.

Multi-agent systems achieve the highest presentation scores (77.7 avg.), with Open Deep Research (81.0) and Deerflow+ (78.8) producing the most polished outputs.

“Multi-agent systems achieve the highest presentation scores (77.7 avg.), with Open Deep Research (81.0) and Deerflow+ (78.8) producing the most polished outputs.”

Classified as probably not a substantial claim.

Analysis depth is assessed along five dimensions: granularity of reasoning, multi-layer insights, critical evaluation, analytical use of evidence, and insight density.

“We assess depth along five dimensions: (1) granularity of reasoning (causal chains), (2) multi-layer insights (trade-offs and implications), (3) critical evaluation (critiques and limitations), (4) analytical use of evidence , and (5) insight density .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

DeepEval offers a comprehensive suite including report-level metrics (presentation, consistency) and fine-grained dimensions (depth, checklist-based coverage, citation traceability, rubric-tree accuracy).

“In contrast, DeepEval offers a comprehensive suite : report-level metrics (presentation, consistency) and fine-grained dimensions (depth, checklist-based coverage, citation traceability, rubric-tree accuracy).”

May combine two claims.

Gemini 2.5 Pro aligned most closely with human preferences, followed by GPT-5, while Claude 4 Sonnet exhibited inconsistent and low agreement with human experts.

“Among the candidates-Gemini 2.5 Pro, GPT-5, and Claude 4 Sonnet-results showed that Gemini 2.5 Pro aligned most closely, followed by GPT-5, while Claude 4 Sonnet exhibited inconsistent and low agreement with human experts (Appendix C).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Citation Accuracy evaluates whether citations genuinely support their claims.

“❻ Citation Accuracy. Complementary to association, this metric evaluates whether citations genuinely support their claims.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

DeepEval is reliable and human-aligned, using structured checklists and aggregate results that yield stable grades closely tracking expert judgments.

“Crucially, DeepEval is reliable and human-aligned : instead of holistic scores, we use structured checklists (e.g., factual errors as atomic tests) and aggregate results, yielding stable grades that closely track expert judgments.”

May combine two claims.

LiveResearchBench consists of 100 expert-curated questions, each paired with detailed checklists, created through a six-stage curation pipeline and validated via a five-step quality control process.

“It consists of 100 expert-curated questions, each paired with detailed checklists, created through a six-stage curation pipeline (Figure 3) and validated via a five-step quality control process (Figure 4).”

Probably not a substantial claim: may not pass S4 (based).

Each DeepEval metric is paired with a tailored evaluation protocol selected to ensure stable model assessment and high agreement with human evaluations.

“Each metric is paired with a tailored evaluation protocol ( e.g. , checklist-based, pointwise, pairwise, or rubric-tree), selected to ensure stable model assessment and high agreement with human evaluations.”

May combine two claims.

Once a model commits to an overall positive assessment, it rarely examines the report in detail, leading to missed errors and inflated scores.

“In other words, once a model commits to an overall positive assessment, it rarely examines the report in detail, leading to missed errors and inflated scores.”

Probably not a substantial claim: may not pass S4 (based).

DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments.

“DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments.”

May combine two claims.

Longer reports do not necessarily translate into higher quality; improvements in citation discipline and factual consistency are not monotonically related to length.

“As we show later, longer reports do not necessarily translate into higher quality; improvements in citation discipline and factual consistency are not monotonically related to length.”

May combine two claims.

OpenAI o3 Deep Research often produces long reports but leaves key statements/facts uncited.

“For example, OpenAI o3 Deep Research often produces long reports but leaves key statements/facts uncited, while Grok-4 Heavy Deep Research often cites URLs that fail to support the associated claims.”

May combine two claims.

LiveResearchBench tasks are explicitly multi-faceted and grounded in user needs.

“In contrast, tasks in LiveResearchBench are explicitly multi-faceted and grounded in user needs.”

May combine two claims.

Deerflow+ (61.6) and Open Deep Research (65.3) lag on Coverage & Comprehensiveness despite strong citation association.

“By contrast, Deerflow+ (61.6) and Open Deep Research (65.3) lag despite strong citation association.”

May combine two claims.

LiveResearchBench tasks span seven diverse domains and ten categories such as market analysis, literature review, policy evaluation, and topic exploration.

“The tasks we benchmark span seven diverse domains ( e.g., science, business, healthcare) and ten categories such as market analysis, literature review, policy evaluation, and topic exploration.”

May combine two claims.

On average, single agents like GPT-5 series and the Grok family perform strongly on Coverage & Comprehensiveness, with Grok-4 Heavy Deep Research achieving the highest score (89.3).

“On average, single agents like GPT-5 series and the Grok family perform strongly on Coverage & Comprehensiveness, with Grok-4 Heavy Deep Research achieving the highest score (89.3).”

May combine two claims.

The pilot study observed error patterns including mismatched in-text citations and references, missing links or incomplete URLs, inconsistent citation formats, uncited references appearing in the bibliography, broken or incomplete table formatting, out-of-order reference numbering, embedded citations breaking text flow, and hallucinated information.

“In this section, we provide concrete examples of error patterns observed in our pilot study. This includes mismatched in-text citations and references (Figure 14), missing links or incomplete URLs (Figure 15), inconsistent citation formats (Figure 16), uncited references appearing in the bibliography (Figure 17), broken or incomplete table formatting (Figure 18), out-of-order reference numbering (”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of out-of-order references.

“Figure 19: Error pattern: out-of-order references.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of embedded citations breaking text flow.

“Figure 20: Error pattern: embedded citations breaking text flow.”

May combine two claims.

In wide information search, most citation errors stem from unsupported claims (claims not verifiable from the cited link) rather than invalid or irrelevant URLs, highlighting that hallucinations persist even with web access.

“For example, in wide information search, most errors stem from unsupported claims (claims not verifiable from the cited link) rather than invalid or irrelevant URLs, highlighting that hallucinations persist even with web access.”

May combine two claims.

Single web agent systems favor coherent reasoning; multi-agent systems achieve stronger association and presentation; and specialized designs lead in coverage.

“Single web agent systems favor coherent reasoning; multi-agent systems achieve stronger association and presentation; and specialized designs lead in coverage.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of broken table format.

“Figure 18: Error pattern: broken table format.”

May combine two claims.

The absence of inline-citation control made claims harder to anchor to sources and increased hallucination risk for the same set of retrieved pages.

“We also observed that the absence of inline-citation control made claims harder to anchor to sources and increased hallucination risk for the same set of retrieved pages.”

May combine two claims.

The problem is worse for market analysis, where all systems generate large numbers of unsupported claims (Open Deep Research averages 91.9 errors per report), underscoring citation accuracy as a persistent bottleneck in deep research.

“The problem is worse for market analysis, where all systems generate large numbers of unsupported claims (Open Deep Research averages 91.9 errors per report), underscoring citation accuracy as a persistent bottleneck in deep research.”

May combine two claims.

DEERFLOW+ produced higher-quality reports with better retention of retrieved evidence, improved formatting and factual consistency, and more reliable performance on presentation checks tied to citation management, particularly P4 (Citation Completeness) and P9 (Format Consistency).

“and produced higher-quality reports: better retention of retrieved evidence, improved formatting and factual consistency, and more reliable performance on presentation checks tied to citation management, particularly P4 (Citation Completeness) and P9 (Format Consistency) in Figure 22.”

May combine two claims.

These fixes substantially reduce early termination and bring DEERFLOW on par with other multi-agent systems in the experiments.

“Overall, these fixes substantially reduce early termination and bring DEERFLOW on par with other multi-agent systems in our experiments.”

May combine two claims.

DEERFLOW+ retains more retrieved information, presents stronger evidence structuring and citation discipline, and avoids token-limit failures seen in the vanilla version.

“Our DEERFLOW+ retains more retrieved information, presents stronger evidence structuring and citation discipline, while avoiding token-limit failures seen in the vanilla version.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of inconsistent citation format.

“Figure 16: Error pattern: inconsistent citation format.”

May combine two claims.

For Coverage & Comprehensiveness, across all 72 questions, the final human-aligned judgment corresponds to either Gemini 2.5 Pro or GPT-5 in 100.0% of cases, with no instance uniquely favoring Claude 4 Sonnet.

“Across all 72 questions, the final humanaligned judgment corresponds to either Gemini 2.5 Pro or GPT-5 in 100.0% of cases, with no instance uniquely favoring Claude 4 Sonnet.”

• AssumptionRests on an assumption: Human expert judgments are a reliable ground truth for evaluating deep research reports.

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of references not mentioned in the report.

“Figure 17: Error pattern: references not mentioned in the report.”

Probably not a substantial claim: may not pass S4 (based).

When presentation is not evaluated separately, LLM judges tend to conflate stylistic issues with factual or logical errors.

“We further observed that when presentation is not evaluated separately, LLM judges tend to conflate stylistic issues with factual or logical errors.”

May combine two claims.

DEERFLOW+ exhibits higher information retention, stronger evidence structuring with inline citations, and no token-limit failures.

“DEERFLOW+ exhibits higher information retention, stronger evidence structuring with inline citations, and no token-limit failures .”

May combine two claims.

Independent judging runs using frontier models (GPT-5, Gemini-2.5-Pro, Claude-4-Opus, Claude-4-Sonnet) frequently differ by over 50 points in analysis depth, severely limiting reproducibility.

“Independent judging runs using frontier models (GPT-5, Gemini-2.5-Pro, Claude-4-Opus, Claude-4-Sonnet) frequently differ by over 50 points in analysis depth, severely limiting reproducibility.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of mismatched in-text citations and references.

“Figure 14: Error pattern: mismatched in-text citations and references.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The survey participants were grouped into three categories: enterprise professionals (27%), researchers (24%), and general users (49%).

“We group participants into three categories: enterprise professionals (including sales, marketing, legal, and software engineering; 27% ), researchers (including professors, industrial researchers, postdocs, and Ph.D. students; 24% ), and general users (the rest; 49% ).”

May combine two claims.

LiveResearchBench combines human-verified rubrics, expert-curated tasks, open-ended multidomain queries, time-varying scenarios, and an Agent-ensemble-as-a-judge to ensure consistency and reliability.

“In contrast, LiveResearchBench combines human-verified rubrics, expert-curated tasks, open-ended multidomain queries, time-varying scenarios, and an Agent-ensemble-as-a-judge to ensure consistency and reliability.”

• AssumptionRests on an assumption: The six criteria in Table 6 are the relevant dimensions for comparing deep research benchmarks.

The quote only partly states this claim.

For citation traceability, the agreement rate reaches 85.9% over 128 issues identified by GPT-5 and Gemini-2.5 Pro.

“For citation traceability, the agreement rate reaches 85.9% over 128 issues identified by these judges.”

• AssumptionRests on an assumption: Human expert judgments are a reliable ground truth for evaluating deep research reports.

May combine two claims.

These methods substantially reduce variance and provide finer-grained diagnostics across dimensions.

“These methods substantially reduce variance and provide finer-grained diagnostics across dimensions.”

May combine two claims.

Our evaluation of 17 leading systems shows that while agents can gather and organize information, they struggle with citation reliability and analytical depth.

“Our evaluation of 17 leading systems shows that while agents can gather and organize information, they struggle with citation reliability and analytical depth.”

May combine two claims.

Numbering accuracy in P10 varies by pipeline, with some single-agent web systems exhibiting steadier numbering.

“whereas numbering accuracy in P10 varies by pipeline, with some single-agent web systems exhibiting steadier numbering.”

• AssumptionRests on an assumption: The ten presentation checklist items (P1-P10) are valid measures of presentation and organization quality.

Probably not a substantial claim: may not pass S3 (load-bearing).

The reports exhibit an error pattern of hallucinated information.

“Figure 21: Error pattern: hallucinated information.”

May combine two claims.

For Presentation & Organization, the human agreement rate for Gemini 2.5 Pro or GPT-5 is 98.3% of cases (Gemini 2.5 Pro alone 95.0%, GPT-5 alone 85.0%).

“The human agreement rate for Gemini 2.5 Pro or GPT-5 is 98.3% of cases (Gemini 2.5 Pro alone 95.0%, GPT-5 alone 85.0%).”

• AssumptionRests on an assumption: Human expert judgments are a reliable ground truth for evaluating deep research reports.

May combine two claims.

Recent deep research benchmarks are often static, domain-specific, coarse-grained, or ambiguous, frequently omitting details such as intended audience, required output format, or scope.

“Recent benchmarks have introduced search-intensive tasks (Gou et al., 2025), but these are often static (and thus prone to contamination) (Bosse et al., 2025), domain-specific (Xu et al., 2025b; Patel et al., 2025), coarse-grained, or ambiguous (Du et al., 2025)-frequently omitting details such as the intended audience, required output format, or scope.”

• cites Gou et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bosse et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Long-form open-ended reports cannot be evaluated with simple string-matching approaches.

“For example, we cannot use simple string-matching approaches, while real-time queries imply that there is time-sensitive variation, without a fixed ground truths.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Tasks in deep research are inherently multi-dimensional, requiring assessment of coverage, reasoning, evidence use, and presentation quality.

“Most critically, tasks are inherently multi-dimensional: we must assess coverage, reasoning, evidence use, and presentation quality.”

May combine two claims.

Deep Research Bench includes search-intensive queries that are static, time-bounded, and require minimal reasoning, limiting their ability to evaluate evolving real-world research.

“For example, Deep Research Bench (Bosse et al., 2025) includes search-intensive queries such as How many IM and GM account closures did chess.com report for 2024? , but these are static, time-bounded, and require minimal reasoning-limiting their ability to evaluate evolving real-world research.”

• cites Bosse et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Two top-tier deep research models (OpenAI o3 Deep Research and Gemini Deep Research) were prompted to generate clarification questions, and human experts reviewed and refined each query.

“To make the questions unambiguous and consistently interpretable across both models and human evaluators, we then prompt two top-tier deep research models, OpenAI o3 Deep Research and Gemini Deep Research, to generate possible clarification questions. Relying on a single model risks bias or omission, so using multiple models provides complementary perspectives. Human experts then review these clar”

Classified as probably not a substantial claim. May combine two claims.

For each system, settings were configured to maximize performance, including setting output length to the maximum token limit and enabling retries for low-quality or failed generations.

“For each system, we configure settings to maximize performance. For example, setting output length to the maximum token limit (128,000 for GPT-5 models) and enabling retries for low-quality or failed generations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

As reports grow longer, they are more likely to contain factual and logical inconsistencies.

“As reports grow longer, they are more likely to contain factual and logical inconsistencies, or even contradictory claims and numbers.”

May combine two claims.

They adopt a multi-judge ensemble protocol using Gemini 2.5 Pro and GPT-5 as independent judges and report final scores as the average of their assessments.

“Based on these findings, we adopt a multi-judge ensemble protocol for all LLM-as-a-Judge evaluations. Specifically, we use Gemini 2.5 Pro and GPT-5 as independent judges and report final scores as the average of their assessments.”

Classified as probably not a substantial claim. May combine two claims.

The clarity of LiveResearchBench queries ensures consistent interpretation across models and imposes rigorous demands on information retrieval and in-depth reasoning.

“This level of clarity ensures consistent interpretation across models and imposes rigorous demands on information retrieval and in-depth reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Presentation quality is evaluated with a checklist of 10 common error patterns from their pilot study, covering structure, grammar, citations, duplicates, and formatting.

“We evaluate presentation quality with a checklist of 10 common error patterns from our pilot study (Table 2), covering structure, grammar, citations, duplicates, and formatting.”

May combine two claims.

Existing deep research benchmarks are domain-specific, restricted to short-form or closed-ended answers, and static, which hinders fair comparison across agents and prevents evaluation on evolving information.

“While valuable, existing benchmarks exhibit some combination of the following limitations: they are domain-specific, restricted to short-form or closed-ended answers, and static, which hinders fair comparison across agents and prevents evaluation on evolving information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-5 was used to generate initial checklists that decompose each query into unit questions testing specific aspects of the required output.

“After the queries are finalized, we use GPT-5 to generate initial checklists. Each checklist decomposes a query into unit questions that test specific aspects of the required output.”

May combine two claims.

In multi-agent systems, increasing planning steps and subagents can easily push retrieval into thousands (>3,000), where the context problem compounds.

“In multi-agent systems, increasing planning steps and subagents can easily push retrieval into thousands ( > 3,000), where the problem compounds.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Perplexity Sonar Reasoning benefits from producing relatively concise reports, which lowers the probability of factual or logical mistakes while keeping reasoning aligned.

“Similarly, Perplexity Sonar Reasoning benefits from producing relatively concise reports, which lowers the probability of factual or logical mistakes while keeping reasoning aligned.”

Probably not a substantial claim: may not pass S2 (checkable).

The open challenge is not the synthesizer alone, but the design of system-wide memory architectures and compression strategies that allow agents to retain breadth while maintaining fidelity.

“Thus, the open challenge is not the synthesizer alone, but the design of system-wide memory architectures and compression strategies that allow agents to retain breadth while maintaining fidelity.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Human evaluation is costly and hard to scale, while naive LLM-as-a-judge setups produce inconsistent results.

“Human evaluation is costly and hard to scale, while naive LLM-as-a-judge setups produce inconsistent results.”

Classified as probably not a substantial claim.

They employ an agentic judge with web access that performs a structured validation process for Citation Accuracy.

“To capture this, we employ an agentic judge with web access that performs a structured validation process: (1) identify each claim with its linked source URL, (2) check whether each URL is accessible, (3) if accessible, assess whether the content sufficiently supports the claim.”

May combine two claims.

DeepResearch Bench often omits crucial elements such as target audience, content and format requirements, or a clearly defined scope, leaving queries open to multiple interpretations and undermining evaluation reliability.

“Likewise, DeepResearch Bench (Du et al., 2025) often omits crucial elements such as the target audience, content and format requirements, or a clearly defined scope. Such omissions leave queries open to multiple interpretations, even across runs of the same model, undermining evaluation reliability.”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Domain experts from enterprise and academia drafted initial research questions aligned with the distributions.

“Next, we hire domain experts from enterprise and academia to draft initial research questions aligned with these distributions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The quality-control process ensures that overall quality deviates by no more than 5% from the true data quality.

“A second quality-control stage then samples annotation results to detect potential systematic issues and to ensure that overall quality deviates by no more than 5% from the true data quality.”

May combine two claims.

LiveResearchBench tasks are more realistic and unambiguous than prior work.

“Compared to prior work, tasks in LiveResearchBench are realistic and unambiguous .”

• Also stated in: § 3.2 BENCHMARK OVERVIEW · p. 4 · plain

  “Compared to prior deep research benchmarks, LiveResearchBench tasks are designed to be more unambiguous and realistic (Figure 1).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Real-time queries have time-sensitive variation without fixed ground truths.

“For example, we cannot use simple string-matching approaches, while real-time queries imply that there is time-sensitive variation, without a fixed ground truths.”

Probably not a substantial claim: may not pass S3 (load-bearing).

They use position-swap averaging to mitigate positional bias in pairwise comparisons.

“To mitigate positional bias, we adopt the position-swap averaging method of Wang et al. (2023), scoring each pair twice with reversed order and averaging results.”

• cites Wang et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

For the collected questions, the authors distilled them into ten representative task categories.

“For the collected questions, we distilled them into ten representative task categories (Figure 2).”

Classified as probably not a substantial claim. May combine two claims.

Analysis Depth, Factual & Logical Consistency, and Citation Traceability are central components of LiveResearchBench's six-dimensional DeepEval framework.

“all of which are central components of our six-dimensional DeepEval framework.”

May combine two claims.

LiveResearchBench adopts an ensemble of Gemini 2.5 Pro and GPT-5 for Coverage & Comprehensiveness evaluation, with each question independently scored by both judge models and the final result averaged.

“Based on these results, we adopt an ensemble of Gemini 2.5 Pro and GPT-5 for Coverage & Comprehensiveness evaluation, where each question is independently scored by both judge models and the final result is averaged.”

• AssumptionRests on an assumption: Human expert judgments are a reliable ground truth for evaluating deep research reports.

May combine two claims.

DeepScholar-Bench's evaluation relies on a single LLM judge (GPT-4.1 or GPT-4o) and lacks human-verified rubrics.

“Its evaluation relies on a single LLM judge (GPT-4.1 or GPT-4o) and lacks humanverified rubrics.”

• cites Patel et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Sustaining both high coverage and depth will require long-horizon memory with update operations, hierarchical information compression without information loss, and explicit synthesis/argumentation modules.

“Sustaining both high coverage and depth will require: (i) long-horizon memory with update operations , (ii) hierarchical information compression without information loss , and (iii) explicit synthesis/argumentation modules .”

May combine two claims.

LiveResearchBench adopts an ensemble of Gemini 2.5 Pro and GPT-5 as final judges for Presentation & Organization, with each question independently scored by both judge models and the average reported.

“Based on these results, we adopt an ensemble of Gemini 2.5 Pro and GPT-5 as our final judges for Presentation & Organization where each question is independently scored by both judge models and we report the average results.”

• AssumptionRests on an assumption: Human expert judgments are a reliable ground truth for evaluating deep research reports.

May combine two claims.

DeepScholar-Bench does not assess Analysis Depth, Factual & Logical Consistency, or Citation Traceability.

“DeepScholar-Bench does not assess Analysis Depth, Factual & Logical Consistency, or Citation Traceability”

• cites Patel et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The annotators' expertise spans multi-modal evaluation (text, image, and audio) as well as quality control for advanced annotation pipelines.

“Their expertise spans multi-modal evaluation (text, image, and audio) as well as quality control for advanced annotation pipelines.”

Classified as probably not a substantial claim.

These representative cases illustrate the diverse range of structural and citation-related issues that can undermine the clarity and overall reliability of generated research reports.

“These representative cases illustrate the diverse range of structural and citation-related issues that can undermine the clarity and overall reliability of generated research reports.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For data annotation and verification, the authors worked with professional annotators from Salesforce.

“For data annotation and verification, we worked with professional annotators from Salesforce.”

May combine two claims.

Adding inline citations both simplifies verification and reduces hallucinations, improving overall report quality.

“Therefore, adding inline citations both simplifies verification and reduces hallucinations, improving overall report quality .”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepEval includes an Analysis Depth dimension that evaluates a model's ability to perform nontrivial analytical reasoning beyond surface-level retrieval or summarization.

“2) Analysis Depth. Evaluates a model's ability to perform nontrivial analytical reasoning beyond surface-level retrieval or summarization.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All annotators hold advanced degrees in fields including English Literature, Social Science, Computer Science, and Finance, with 2-6 years of experience in data annotation and analysis.

“All annotators hold advanced degrees in fields including English Literature, Social Science, Computer Science, and Finance, with 2-6 years of experience in data annotation and analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepEval includes a Factual & Logical Consistency dimension that measures global coherence across the entire report, capturing contradictions, reasoning gaps, and logical discontinuities that local checks may miss.

“3) Factual & Logical Consistency. Measures global coherence across the entire report-capturing contradictions, reasoning gaps, and logical discontinuities that local checks may miss.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepEval includes a Presentation & Organization dimension that assesses a report's structural clarity, stylistic coherence, formatting hygiene, and citation organization.

“1) Presentation & Organization. Assesses a report's structural clarity, stylistic coherence, formatting hygiene, and citation organization.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Treating presentation as its own axis improves clarity, granularity, and diagnostic usefulness.

“Treating presentation as its own axis improves clarity, granularity, and diagnostic usefulness.”