DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation

May combine two claims.

Models exhibit genuine specialization across domains, with cross-model agreement of only ρ = 0.61 and per-case disagreement reaching 18.8 percentage points.

“models exhibit genuine specialization across domains, with cross-model agreement of only ρ = 0.61 and per-case disagreement reaching 18.8 percentage points”

May combine two claims.

Strong and weak models fail in qualitatively different ways, with strong models' errors dominated by incomplete derivation and weak models' by hallucinated precision.

“strong and weak models fail in qualitatively different ways, with strong models' errors dominated by incomplete derivation and weak models' by hallucinated precision”

May combine two claims.

Retrieval failures account for only 12-14% of errors while derivation and calibration failures account for over 70%.

“retrieval failures account for only 12-14% of errors while derivation and calibration failures account for over 70%”

May combine two claims.

In the top-4 group, incomplete derivation accounts for 31% of failures: strong models retrieve the correct intermediate values but misapply a composition step.

“In the top-4 group, incomplete derivation accounts for 31% of failures: strong models retrieve the correct intermediate values but misapply a composition step (e.g., applying a gross-margin rate to total revenue instead of segment revenue).”

May combine two claims.

GPT 5.5 retrieves all three numbers from the correct authoritative sources but applies the margin to total company revenue (CNY 777B) instead of segment revenue, producing a value 34% below the reference.

“GPT 5.5 retrieves all three numbers from the correct authoritative sources but applies the margin to total company revenue (CNY 777B) instead of segment revenue, producing a value 34% below the reference.”

May combine two claims.

The mean pairwise Spearman ρ across 100 case scores is 0.61, with no pair exceeding ρ = 0.79, indicating that models make genuinely different errors rather than failing on a shared set of hard cases.

“The mean is ρ = 0.61, with no pair exceeding ρ = 0.79, indicating that models make genuinely different errors rather than failing on a shared set of hard cases.”

Classified as probably not a substantial claim.

A hallucination-resistance cell asks for a metric that the target company does not publicly disclose.

“A hallucination-resistance cell asks for a metric that the target company does not publicly disclose.”

May combine two claims.

Qwen 3.6 Plus returns a precise number citing an anonymous blog and scores 0; Claude Opus 4.7 returns not available with a justification and scores 1; DeepSeek V4 Flash returns a wide range with an estimation method and scores 0.5.

“Qwen 3.6 Plus returns a precise number citing an anonymous blog and scores 0; Claude Opus 4.7 returns not available with a justification and scores 1; DeepSeek V4 Flash returns a wide range with an estimation method and scores 0.5.”

May combine two claims.

Derivation and Reasoning together cover six dimensions per task and dominate the benchmark score.

“By contrast, Derivation and Reasoning together cover six dimensions per task and dominate the benchmark score.”

May combine two claims.

The strongest two models are not merely better retrievers: their largest advantage over the middle cluster appears in Derivation and Calibration, where they more often preserve scope, carry intermediate quantities through a computation, and abstain when disclosure is insufficient.

“The strongest two models are not merely better retrievers: their largest advantage over the middle cluster appears in Derivation and Calibration, where they more often preserve scope, carry intermediate quantities through a computation, and abstain when disclosure is insufficient.”

May combine two claims.

Codex CLI + GPT-5.5 is the strongest model, with an overall score of 33.37%.

“Codex CLI + GPT-5.5 is the strongest model, with an overall score of 33.37%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of 874 scored model-task pairs, a total of 55,936 cell-level judgments are recorded.

“Of 874 scored model-task pairs, each producing 64 cell scores, a total of 55,936 cell-level judgments are recorded.”

Classified as probably not a substantial claim.

A chain-derivation cell asks for BYD's per-vehicle gross profit, with a reference answer derived from three numbers: automotive-segment revenue (CNY 602B), segment gross margin (20.2%), and passenger-vehicle deliveries (4.27M).

“A chainderivation cell asks for BYD's per-vehicle gross profit, with a reference answer derived from three numbers: automotive-segment revenue (CNY 602B), segment gross margin (20.2%), and passengervehicle deliveries (4.27M).”

May combine two claims.

DeepSeek V4 Flash returned a range of [40%, 55%] and scored 0.5.

“DeepSeek V4 Flash returns a range of [40%, 55%] with a stated estimation method and scores 0.5.”

May combine two claims.

In the remaining-model group, hallucinated precision rises to 38%: weaker models commit to a confident precise number even when no authoritative source supports it, which the scoring rule penalizes to zero.

“In the remaining-model group, hallucinated precision rises to 38%: weaker models commit to a confident precise number even when no authoritative source supports it, which the scoring rule penalizes to zero.”

May combine two claims.

Disagreements concentrated on range-type cells where the tolerance boundary is ambiguous; on precise-value and not available cells, human-machine agreement exceeded κ = 0.90.

“Disagreements concentrated on range-type cells where the tolerance boundary is ambiguous; on precise-value and not available cells, human-machine agreement exceeded κ = 0.90.”

May combine two claims.

MiniMax M2.7 scores 24.06%, while Kimi K2.6 trails at 16.79%.

“MiniMax M2.7 scores 24.06%, while Kimi K2.6 trails at 16.79%.”

May combine two claims.

Among models hosted through Claude Code CLI, Claude Opus 4.7 leads at 31.84%, followed by a middle cluster between 26% and 29%: DeepSeek V4 Pro, GLM 5.1, Claude Sonnet 4.6, DeepSeek V4 Flash, and Qwen 3.6 Plus.

“Among models hosted through Claude Code CLI, Claude Opus 4.7 leads at 31.84%, followed by a middle cluster between 26% and 29%: DeepSeek V4 Pro, GLM 5.1, Claude Sonnet 4.6, DeepSeek V4 Flash, and Qwen 3.6 Plus.”

May combine two claims.

The per-cell rubric assigns 0.5 to GPT 5.5's answer, while Claude Opus 4.7 and DeepSeek V4 Pro correctly scope the margin to segment revenue and receive 1.0.

“The per-cell rubric assigns 0.5 (correct direction and order of magnitude, outside tight tolerance), while Claude Opus 4.7 and DeepSeek V4 Pro correctly scope the margin to segment revenue and receive 1.0.”

May combine two claims.

Per-case cross-model averages range from 14.67% (the hardest case, mortgage REITs) to 83.07% (the easiest, luxury goods), with the hardest cases concentrated in domains requiring reconciliation of non-standardized financial disclosures and the easiest in domains where primary filings are abundant and uniform.

“Per-case cross-model averages range from 14.67% (the hardest case, mortgage REITs) to 83.07%(the easiest, luxury goods), with the hardest cases concentrated in domains requiring reconciliation of non-standardized financial disclosures and the easiest in domains where primary filings are abundant and uniform.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The task matrix scale in DEEPWEB-BENCH is 100 tasks, 8 entities per task, 8 dimensions per task, 64 cells per task.

“Count Tasks Entities per task Dimensions per task Cells per task 100 8 8 64 (c) Task matrix scale”

May combine two claims.

Claude Opus 4.7 returned an explicit not available answer and scored 1.

“Claude Opus 4.7 returns an explicit not available answer with a one-sentence justification and scores 1.”

May combine two claims.

Qwen 3.6 Plus returned a precise value of 52.3% and scored 0.

“Qwen 3.6 Plus returns a precise value of 52.3% and cites a named blog post as its source; under the not available rule any precise value scores 0.”

May combine two claims.

The three candidates issued comparable queries against comparable sources; the difference is how each translated an absence of authoritative evidence into a final answer.

“According to the browsing logs, the three candidates issued comparable queries against comparable sources; the difference is how each candidate translated an absence of authoritative evidence into a final answer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every released task uses the same capability-family split: one Retrieval dimension, four Derivation dimensions, one Calibration dimension, and two Reasoning dimensions.

“Every released task uses the same capability-family split: one Retrieval dimension, four Derivation dimensions, one Calibration dimension, and two Reasoning dimensions.”

• AssumptionRests on an assumption: The capability-family split (one Retrieval, four Derivation, one Calibration, two Reasoning) covers the necessary capabilities for deep research.

May combine two claims.

Only a small share of cells in any task fall in the Retrieval family; the majority are Derivation, Reasoning, or Calibration cells, which pushes difficulty beyond retrieval.

“Only a small share of cells in any task fall in the Retrieval family; the majority are Derivation, Reasoning, or Calibration cells, which pushes difficulty beyond retrieval.”

May combine two claims.

All three models searched comparable sources, but they differed in whether they committed to a precise value or acknowledged the absence of authoritative evidence.

“All three models searched comparable sources, but they differed in whether they committed to a precise value or acknowledged the absence of authoritative evidence.”

May combine two claims.

The reference-record density in DEEPWEB-BENCH is 3.2 source URLs per cell, 2.4 publishers per cell, and 2.8 derivation steps per derivation cell.

“Average count Source URLs per cell Publishers per cell Derivation steps per derivation cell 3.2 2.4 2.8 (d) Reference-record density”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each task in DEEPWEB-BENCH contains eight dimensions.

“Each task contains eight dimensions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The candidate scores 0.5 under the per-cell partial-credit rule.

“The candidate scores 0.5 under the per-cell partial-credit rule: the direction and order of magnitude are correct, but the value falls outside the tight tolerance required for full credit.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For each of the 64 cells, the construction record stores a reference answer, unit, answer type, derivation note, source list, source-provenance labels, cross-source agreement label, and scoring rule.

“For each of the 64 cells, the construction record stores a reference answer, unit, answer type, derivation note, source list, source-provenance labels, cross-source agreement label, and scoring rule.”

May combine two claims.

The top-2 models together achieve the highest score on 79 of 100 cases, but on the most-disagreed cases the cross-model standard deviation reaches 18.8 percentage points, meaning that a model ranked near the top on one domain can rank near the bottom on another.

“The top-2 models (Codex CLI + GPT-5.5 and Claude Opus 4.7) together achieve the highest score on 79 of 100 cases, but on the most-disagreed cases the cross-model standard deviation reaches 18.8 percentage points, meaning that a model ranked near the top on one domain can rank near the bottom on another.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The dimension axis covers four capability families, and every task spans multiple families.

“The dimension axis covers four capability families, and every task spans multiple families.”

Classified as probably not a substantial claim. May combine two claims.

Every cell is scored independently using a four-tier rubric ({1, 0.5, 0.25, 0}) and carries a reference answer with source-provenance labels and cross-source agreement.

“every cell is scored independently using a four-tier rubric ({1, 0.5, 0.25, 0}) and carries a reference answer with source-provenance labels and cross-source agreement.”

May combine two claims.

Models exhibit domain-specific strengths: Codex CLI + GPT-5.5 is strongest in Healthcare and Technology, while Kimi is notably weak in Energy & Materials.

“Models exhibit domain-specific strengths: Codex CLI + GPT-5.5 is strongest in Healthcare and Technology, while Kimi is notably weak in Energy & Materials.”

May combine two claims.

Energy & Materials is the hardest domain (cross-model mean 24.6%) and Healthcare the easiest (30.7%).

“Energy & Materials is the hardest domain (cross-model mean 24.6%) and Healthcare the easiest (30.7%).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The average number of tool calls per task ranges from 24.8 (Claude Opus 4.7) to 41.5 (Codex CLI + GPT-5.5).

“The average number of tool calls per task ranges from 24.8 (Claude Opus 4.7) to 41.5 (Codex CLI + GPT-5.5)”

Probably not a substantial claim: may not pass S3 (load-bearing).

The average answer length ranges from 12,685 characters (Codex CLI + GPT-5.5) to 32,948 characters (MiniMax M2.7).

“the average answer length ranges from 12,685 characters (Codex CLI + GPT-5.5) to 32,948 characters (MiniMax M2.7).”

May combine two claims.

DeepWeb-Bench currently comprises 100 tasks spanning six domain categories: Technology (25), Energy & Materials (20), Industrials & Transport (18), Consumer (16), Finance (12), and Healthcare & Pharma (9).

“Figure: -BENCH currently comprises 100 tasks spanning six domain categories: Technology (25), Energy & Materials (20), Industrials & Transport (18), Consumer (16), Finance (12), and Healthcare & Pharma(9).”

May combine two claims.

Codex CLI + GPT-5.5 and Opus show the most right-shifted cumulative score profiles, while Kimi is left-shifted with many low-scoring cases.

“Codex CLI + GPT-5.5 and Opus show the most right-shifted curves, while Kimi is left-shifted with many low-scoring cases.”

May combine two claims.

The distribution of pairwise Kendall τ across all 4,950 case pairs has mean τ = 0.32 with 10% negative values.

“Distribution of pairwise Kendall τ across all 4,950 case pairs.Mean τ = 0.32 with 10% negative values”

May combine two claims.

Cases below the 20% pass-band floor show compressed spread, while the 20-35% band contains most of the benchmark and still exposes substantial model-to-model variation.

“Cases below the 20% pass-band floor show compressed spread, while the 20-35% band contains most of the benchmark and still exposes substantial model-to-model variation.”

May combine two claims.

Reference derivations cite on average 3.2 distinct URLs per cell and 22 per task, span 2.4 independent publishers per cell, and require 2.8 derivation steps per derivation-type cell.

“reference derivations cite on average 3.2 distinct URLs per cell and 22 per task, span 2.4 independent publishers per cell, and require 2.8 derivation steps per derivation-type cell.”

May combine two claims.

Qwen and MiniMax occupy the middle and lower-middle of the cumulative score distribution, respectively.

“Qwen and MiniMax occupy the middle and lower-middle of the distribution, respectively.”

May combine two claims.

Codex CLI + GPT-5.5 produces the shortest answers (12,685 characters on average) but scores highest (33.37%).

“Codex CLI + GPT-5.5 produces the shortest answers (12,685 characters on average) but scores highest (33.37%)”

May combine two claims.

Codex CLI + GPT-5.5 is relatively stronger in Finance and Industrials; DeepSeek V4 Pro is comparatively strongest in Finance; MiniMax M2.7 remains below Opus across all domains.

“Codex CLI + GPT-5.5 is relatively stronger in Finance and Industrials, DeepSeek V4 Pro is comparatively strongest in Finance, and MiniMax M2.7 remains below Opus across all domains.”

May combine two claims.

Across the 100 released tasks, 61% of supporting source records are T1 primary sources, 24% are T2 methodology-published research or formal industry and statistical datasets, and 15% are T3 or lower.

“Across the 100 released tasks, 61% of supporting source records are T1 primary sources, 24% are T2 methodology-published research or formal industry and statistical datasets, and 15% are T3 or lower.”

• AssumptionRests on an assumption: The source-provenance labels correctly classify sources into T1-T4 categories.

May combine two claims.

Derivation accounts for 50% of dimensions per task, Reasoning 25%, and Calibration and Retrieval each 12.5%.

“Derivation accounts for 50% of dimensions per task (4 of 8), Reasoning 25%, and Calibration and Retrieval each 12.5%”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The divergent cells are concentrated in Calibration-type dimensions, where detecting source disagreement is part of the intended skill.

“The divergent cells are concentrated in Calibration-type dimensions, where detecting source disagreement is part of the intended skill.”

• AssumptionRests on an assumption: The source-provenance labels correctly classify sources into T1-T4 categories.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

In 500 failing cells, the most common failure mode for the top four models is incomplete derivation (31%), while for the other five models it is hallucinated precision (38%).

“| Failure mode | Top four models | Other five models | | --- | --- | --- | | Hallucinated precision | 22% | 38% | | Silent source choice | 18% | 14% | | Incomplete derivation | 31% | 24% | | Scope drift | 15% | 12% | | Retrieval gap | 14% | 12% |”

• AssumptionRests on an assumption: The human-labeled failure modes accurately categorize the reasons for failing cells.

May combine two claims.

Each task uses an 8 × 8 matrix, yielding 100 × 8 × 8 = 6,400 atomic scoring cells.

“Each task targets a distinct industry segment and uses an 8 × 8 matrix, yielding 100 × 8 × 8 = 6,400 atomic scoring cells.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each task is an 8 × 8 matrix of entities against research dimensions.

“Each task is an 8 × 8 matrix of entities against research dimensions”

The quote only partly states this claim.

On DeepWeb-Bench, Codex CLI + GPT-5.5 achieved the highest overall score (33.37%) among the evaluated models.

“| Codex CLI + GPT-5.5 | 33.37 | 37.84 | 32.55 | 34.16 | 32.38 | 16.80 | 92.19 |”

• AssumptionRests on an assumption: The benchmark's per-cell scoring and rubrics validly measure the intended deep research capabilities.
• AssumptionRests on an assumption: The independent grader's scores are a valid measure of answer quality.

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearchGym offers a reproducible sandbox on frozen corpora.

“DeepResearchGym (11) offers a reproducible sandbox on frozen corpora.”

• cites [11]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearch-9K provides 9,000 multi-hop questions at three difficulty levels with search trajectories.

“DeepResearch-9K (47) provides 9,000 multi-hop questions at three difficulty levels with search trajectories.”

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

DRACO evaluates accuracy, completeness, and objectivity across 10 domains using expert-crafted rubrics.

“DRACO (56) evaluates accuracy, completeness, and objectivity across 10 domains using expert-crafted rubrics.”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Deep research tasks require extensive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation.

“deep research tasks require not only extensive evidence collection and cross-source reconciliation but also long-horizon multi-step derivation in which the agent must compose retrieved numbers into a final quantitative answer through explicit arithmetic and modeling assumptions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearch Bench and DeepResearch Bench II evaluate deep research agents across broad sets of research tasks requiring long-horizon browsing and multi-step synthesis.

“DeepResearch Bench (13) and DeepResearch Bench II (21) evaluate deep research agents across broad sets of research tasks requiring long-horizon browsing and multi-step synthesis.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Deep research is currently deployed along two tracks: vertically integrated commercial products and command-line coding-agent harnesses.

“It is currently deployed along two tracks: vertically integrated commercial products such as OpenAI Deep Research (34), Claude Research (3), Gemini Deep Research (15), Perplexity Research-Pro (37), and Kimi Researcher (31); and command-line coding-agent harnesses such as Claude Code (2) and Codex (33), which practitioners increasingly pair with frontier backbones for open-web research.”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

BrowseComp and BrowseComp-ZH require answers to be assembled from many web pages rather than read from a single source, but remain focused on short-answer questions whose answers require no further derivation.

“BrowseComp (44) and BrowseComp-ZH (57) raise the bar by requiring answers to be assembled from many web pages rather than read from a single source, but they remain focused on short-answer questions whose answers, once found, require no further derivation.”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [57]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The difficulty of DEEPWEB-BENCH comes from three properties of the data: massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation.

“Difficulty comes from three properties of the data itself: each task requires massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation.”

• AssumptionRests on an assumption: The three properties of massive evidence collection, cross-source reconciliation, and long-horizon multi-step derivation are the right properties to make a benchmark substantially harder than existing benchmarks for the current frontier.

Probably not a substantial claim: may not pass S3 (load-bearing).

OpenResearcher builds an open offline pipeline for synthesizing long-horizon trajectories.

“OpenResearcher (24) builds an open offline pipeline for synthesizing long-horizon trajectories.”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

WideSearch benchmarks broad information-seeking in which an agent populates a structured table from web sources.

“WideSearch (45) benchmarks broad information-seeking in which an agent populates a structured table from web sources.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

AutoResearchBench benchmarks scientific literature discovery where even the strongest models achieve below 10%.

“AutoResearchBench (51) benchmarks scientific literature discovery where even the strongest models achieve below 10%.”

• cites [51]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Several of these benchmarks were difficult at introduction, yet frontier deep research products are now reported to score strongly on them as well.

“Several of these benchmarks were difficult at introduction, yet frontier deep research products are now reported to score strongly on them as well (34; 3).”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

LiveResearchBench provides live user-centric evaluation.

“LiveResearchBench (41) provides live user-centric evaluation”

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The public benchmark release includes the data, rubrics, and evaluation code.

“The public benchmark release includes the data, rubrics, and evaluation code.”

Classified as probably not a substantial claim.

Frontier deep research products are now reported to score strongly on existing benchmarks.

“yet frontier deep research products are now reported to score strongly on them (34; 3).”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepSearchQA targets comprehensiveness in multi-step information-seeking across 17 fields.

“DeepSearchQA (17) targets comprehensiveness in multi-step information-seeking across 17 fields.”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The three sources of difficulty are represented as four capability families: Retrieval, Derivation, Reasoning, and Calibration.

“We represent these three sources of difficulty as four capability families (Retrieval, Derivation, Reasoning, and Calibration) and report results sliced by family.”

• AssumptionRests on an assumption: The four capability families (Retrieval, Derivation, Reasoning, and Calibration) adequately represent the three sources of difficulty.

Probably not a substantial claim: may not pass S3 (load-bearing).

Mind2Web 2 evaluates 130 long-horizon agentic search tasks with an agent-as-a-judge framework.

“Mind2Web 2 (16) evaluates 130 long-horizon agentic search tasks with an agent-as-a-judge framework.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Scoring is performed by an automated GPT-5.5 grader applying the per-cell rubric; a human validation on a stratified sample of 200 cells yields κ = 0.82 agreement with the automated grader.

“Scoring is performed by an automated GPT-5.5 grader applying the per-cell rubric; a human validation on a stratified sample of 200 cells yields κ = 0.82 agreement with the automated grader.”

Classified as probably not a substantial claim. May combine two claims.

Each cell carries supporting sources, a source-provenance label for each source, and a marker indicating whether the sources agree, disagree, or provide only a single independent view.

“Each cell also carries supporting sources, a source-provenance label for each source, and a marker indicating whether the sources agree, disagree, or provide only a single independent view.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The error falls under the incomplete derivation failure mode.

“The error falls under the incomplete derivation failure mode: two of three intermediate numbers are correct, the third is misapplied.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The final audit checks five conditions before a task is admitted.

“The final audit checks five conditions before a task is admitted.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of 900 model-task pairs (9 models × 100 tasks), 874 produced valid responses and were scored; the remaining 26 pairs are listed in the full result tables.

“Of 900 model-task pairs (9 models × 100 tasks), 874 produced valid responses and were scored; the remaining 26 pairs are listed in the full result tables.”

May combine two claims.

This difficulty is not uniform across domains: models develop domain-specific strengths, so a single aggregate score does not fully capture a model's deep research profile.

“Finding 3 adds that this difficulty is not uniform across domains: models develop domain-specific strengths, so a single aggregate score does not fully capture a model's deep research profile.”

Classified as probably not a substantial claim.

DEEPWEB-BENCH adapts distinctions from recent evaluation work to quantitative deep research by making derivation, source reconciliation, and calibrated abstention explicit per-cell capabilities rather than implicit properties of a whole answer.

“DEEPWEB-BENCH adapts these distinctions to quantitative deep research by making derivation, source reconciliation, and calibrated abstention explicit per-cell capabilities rather than implicit properties of a whole answer.”

Probably not a substantial claim: may not pass S4 (based).

A benchmark task that separates frontier deep research agents from simpler retrieval pipelines should reduce the chance that a lucky search query, a single authoritative page, or a narrow skill is enough for a high score.

“A benchmark task that separates frontier deep research agents from simpler retrieval pipelines should reduce the chance that a lucky search query, a single authoritative page, or a narrow skill is enough for a high score.”

May combine two claims.

The per-cell rule-based scoring used in DEEPWEB-BENCH trades flexibility for reproducibility and is intended to make human cross-checking tractable when disputes arise.

“The per-cell rule-based scoring used in DEEPWEB-BENCH trades flexibility for reproducibility and is intended to make human cross-checking tractable when disputes arise.”

May combine two claims.

The per-cell design isolates the calibration skill from retrieval skill, and the three-tier answer-type system captures meaningful gradations in agent behavior that a binary correct/incorrect scheme would collapse.

“This case shows that the per-cell design isolates the calibration skill from retrieval skill, and that the three-tier answer-type system (precise / range / not-available) captures meaningful gradations in agent behavior that a binary correct/incorrect scheme would collapse.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All models are accessed through configured backend endpoints and paired with the same benchmark browsing tools.

“All models are accessed through configured backend endpoints and paired with the same benchmark browsing tools.”

May combine two claims.

A model can improve its Retrieval score by finding primary documents more reliably, but that improvement affects only one dimension per task.

“A model can improve its Retrieval score by finding primary documents more reliably, but that improvement affects only one dimension per task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 16.58-point gap leaves substantial headroom for current frontier agents.

“the 16.58-point gap between the strongest and weakest models leaves substantial headroom for current frontier agents.”

May combine two claims.

DEEPWEB-BENCH avoids an aggregate free-form judge as the primary metric; each answer is scored at the cell level against a stored reference value and explicit rubric.

“DEEPWEB-BENCH avoids an aggregate free-form judge as the primary metric; each answer is scored at the cell level against a stored reference value and explicit rubric.”

Classified as probably not a substantial claim.

Each evaluated agent is placed in an isolated session per task, receiving the entity list, the dimension list, and the output-format specification.

“Each evaluated agent is placed in an isolated session per task, receiving the entity list, the dimension list, and the output-format specification.”

Classified as probably not a substantial claim.

No reference answer, source-provenance label, scoring rule, or any hint about the expected value is ever included in the prompt.

“No reference answer, source-provenance label, scoring rule, or any hint about the expected value is ever included in the prompt.”

Classified as probably not a substantial claim.

The reference answer relies on T1 or T2 evidence whenever such evidence exists.

“A lower-provenance source can provide context, but the reference answer relies on T1 or T2 evidence whenever such evidence exists.”

• AssumptionRests on an assumption: The T1-T4 source-provenance scheme is a valid measure of source quality for the benchmark's purposes.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The benchmark stores URLs and short factual excerpts for scoring purposes; it does not redistribute full documents.

“The benchmark stores URLs and short factual excerpts for scoring purposes; it does not redistribute full documents.”

Classified as probably not a substantial claim.

The task score is the mean of independent cell scores rather than a single holistic judgment on a long report.

“This format fixes the scope of the task and the atomic unit of grading: a task score is the mean of independent cell scores rather than a single holistic judgment on a long report.”

Classified as probably not a substantial claim.

No tool returns source-provenance information, cross-source hints, or any metadata beyond the raw page content.

“No tool returns source-provenance information, cross-source hints, or any metadata beyond the raw page content.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Variation in disclosure is part of the task state: on some dimensions a calibrated agent should return not available for entities whose public filings do not support a precise answer, rather than forcing a number for every row.

“Variation in disclosure is part of the task state: on some dimensions a calibrated agent should return not available for entities whose public filings do not support a precise answer, rather than forcing a number for every row.”

Classified as probably not a substantial claim.

We evaluate nine frontier model configurations on the 100-task release of DEEPWEB-BENCH.

“We evaluate nine frontier model configurations on the 100-task release of DEEPWEB-BENCH, report aggregate and capability-family scores, and analyze the main failure patterns.”

Classified as probably not a substantial claim. May combine two claims.

Producing a conclusion in DEEPWEB-BENCH requires reading many authoritative documents, reconciling values that disagree, and composing the result through multi-step derivation; producing the full set of conclusions amplifies this load.

“Producing such a conclusion requires reading many authoritative documents, reconciling values that disagree, and composing the result through multi-step derivation; producing the full set of conclusions amplifies this load, because the same body of evidence feeds many conclusions and the agent has to work with it coherently across the whole task.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The harness enforces a per-task budget of 200 tool calls and a wall-clock cap of 30 minutes.

“The harness enforces a per-task budget of 200 tool calls (any mix of the three tools) and a wall-clock cap of 30 minutes.”

• AssumptionRests on an assumption: The 200 tool call budget and 30-minute wall-clock cap are sufficient for agents to complete tasks.

Classified as probably not a substantial claim. May combine two claims.

No reference answer, source hint, or scoring rule is disclosed to the agent.

“No reference answer, source hint, or scoring rule is disclosed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All three benchmark tools are available to every model configuration.

“All three benchmark tools are available to every model configuration.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Native search and browsing tools in the host command-line interfaces are disabled.

“Native search and browsing tools in the host command-line interfaces are disabled.”

Classified as probably not a substantial claim. The quote only partly states this claim.

The dimension axis spans four capability families: Retrieval, Derivation, Reasoning, and Calibration.

“The dimension axis spans four capability families.”

May combine two claims.

Because the rubric requires an explicit derivation alongside the final value, the misapplied step is directly identifiable in the agent's output, which would not be possible under a scoring rule that compares only the final number.

“Because the rubric requires an explicit derivation alongside the final value, the misapplied step is directly identifiable in the agent's output, which would not be possible under a scoring rule that compares only the final number.”

Classified as probably not a substantial claim. May combine two claims.

The task score is the mean of cell scores; the benchmark score is the mean across tasks.

“The task score is the mean of cell scores; the benchmark score is the mean across tasks.”

Classified as probably not a substantial claim. May combine two claims.

The budget is 200 tool calls and 30 minutes wall-clock per task; cells without an answer at budget exhaustion score 0.

“The budget is 200 tool calls and 30 minutes wall-clock per task; cells without an answer at budget exhaustion score 0.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All agents access the same three benchmark browsing tools: web_search, page_visit, and pdf_fetch.

“All agents access the same three benchmark browsing tools: web_search for retrieving candidate pages, page_visit for reading web pages, and pdf_fetch for downloading and reading PDF documents.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The provenance labels are a record of disclosure provenance rather than a learned quality score.

“These labels are a record of disclosure provenance rather than a learned quality score.”

Classified as probably not a substantial claim. May combine two claims.

A DEEPWEB-BENCH task covers one subject domain and asks an agent to complete an 8 × 8 matrix of quantitative research answers; each cell is graded independently against a reference answer, a derivation record, and supporting sources.

“A task covers one subject domain and asks an agent to complete an 8 × 8 matrix of quantitative research answers; each cell is graded independently against a reference answer, a derivation record, and supporting sources.”

May combine two claims.

DEEPWEB-BENCH sits at the deep research end of the spectrum and differs from prior work along two axes: difficulty and auditability.

“DEEPWEB-BENCH sits at the deep research end of this spectrum and differs from prior work along two axes.”

Classified as probably not a substantial claim.

Each cell is scored by a fixed four-tier rubric ({1, 0.5, 0.25, 0}): 1 for a value and derivation within tolerance, 0.5 for a partially correct answer, 0.25 for a marginally relevant attempt, and 0 otherwise.

“Each cell is scored by a fixed four-tier rubric ({1, 0.5, 0.25, 0}): 1 for a value and derivation within tolerance, 0.5 for a partially correct answer (e.g., the correct direction of change but a value outside tolerance, or a correct range without a derivation), 0.25 for a marginally relevant attempt, and 0 otherwise.”

Classified as probably not a substantial claim.

Each cell's reference answer is either a precise value, a range estimate with a stated confidence and derivation method, or an explicit not available marker when the quantity is not disclosed by an authoritative source.

“Each cell's reference answer is either a precise value, a range estimate with a stated confidence and derivation method, or an explicit not available marker when the quantity is not disclosed by an authoritative source.”

Classified as probably not a substantial claim. May combine two claims.

DEEPWEB-BENCH is organized around task matrices, cell-level reference records, and a shared evaluation protocol.

“DEEPWEB-BENCH is organized around task matrices, cell-level reference records, and a shared evaluation protocol.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All entities are publicly traded companies or publicly known organizations.

“All entities are publicly traded companies or publicly known organizations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cross-source agreement labels (consistent, divergent, single) are assigned during construction to record whether a cell has independent corroboration, visible disagreement, or only one public source.

“Cross-source agreement labels (consistent, divergent, single) are assigned during construction to record whether a cell has independent corroboration, visible disagreement, or only one public source.”

May combine two claims.

The per-cell design supports capability-targeted diagnosis and extension to additional domains.

“The per-cell design supports capability-targeted diagnosis and extension to additional domains.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The release contains 6,400 cells; 874/900 model-task pairs were scored, yielding 55,936 scored cells.

“The release contains 6,400 cells; 874/900 model-task pairs were scored, yielding 55,936 scored cells.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The eight non-Codex models are hosted through Claude Code CLI with native browsing disabled.

“The eight non-Codex models are hosted through Claude Code CLI with native browsing disabled.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Source-provenance labels serve as an audit record during task construction.

“Source-provenance labels serve as an audit record during task construction.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

T3 and T4 sources are retained when they provide useful context, but cell-level reference answers prioritize T1 and T2 evidence whenever such evidence is publicly available.

“T3 and T4 sources are retained when they provide useful context, but cell-level reference answers prioritize T1 and T2 evidence whenever such evidence is publicly available.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Backend identifiers are internal run identifiers and do not imply access to native search.

“Backend identifiers are internal run identifiers and do not imply access to native search.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Usage statistics are diagnostic only; the benchmark score is computed from the submitted cell answers under the rubric, with missing runs excluded from the scored set.

“These usage statistics are diagnostic only; the benchmark score is computed from the submitted cell answers under the rubric, with missing runs excluded from the scored set.”

• AssumptionRests on an assumption: The benchmark's per-cell scoring and rubrics validly measure the intended deep research capabilities.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference answers consist of aggregate financial and operational metrics, not individual-level data.

“Reference answers consist of aggregate financial and operational metrics, not individual-level data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The construction process records cross-source verification whenever independent public sources are available.

“The construction process records cross-source verification whenever independent public sources are available.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Codex CLI + GPT-5.5 is hosted through Codex CLI with native browsing disabled.

“Codex CLI + GPT-5.5 is hosted through Codex CLI with native browsing disabled.”

May combine two claims.

Because the public benchmark release includes data, rubrics, evaluation code, source-provenance labels, and source records, evaluators can reproduce scores and extend the analysis to new models.

“Because the public benchmark release includes data, rubrics, evaluation code, source-provenance labels, and source records, evaluators can reproduce scores and extend the analysis to new models.”

May combine two claims.

Under a binary task-level pass/fail, most or all evaluated agents would fail every task at the present difficulty, and the capability-axis analysis would not be feasible.

“Under this alternative, most or all evaluated agents would fail every task at the present difficulty, and the capability-axis analysis would not be feasible.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Per-model failure-mode breakdowns require human annotation of a stratified sample of failing cells.

“Per-model failure-mode breakdowns require human annotation of a stratified sample of failing cells.”

Classified as probably not a substantial claim.

Every model receives the benchmark-provided web search, page visit, and PDF retrieval tools.

“Every model receives the benchmark-provided web search, page visit, and PDF retrieval tools.”