CoChat
Sign Up
PreprintCited by 0Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

Open access
Leung, Ethan, Lumer, Elias, Feld, Corey, Huber, Austin, Subbiah, Vamse Kumar, Paul, Kevin·2026·arXiv
DOI 10.48550/arxiv.2607.08700arXiv 2607.08700
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training. Before such a signal can be trusted, we need to know how capable the judge must be and how biased it is. We study this calibration question for citation quality in deep-research systems, where a search-grounded LLM must support each claim it writes with a cited source. Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM, source relevance and factual support. On an adversarial long-form benchmark, we score 8 off-the-shelf LLM judges from 3 model families against gold labels over 1,248 rubric decisions, all of which were human-reviewed and 378 of which were hard cases adjudicated from judge disagreements. Cheaper judges remain competitive across both dimensions, with GPT-5-mini attaining the strongest source-relevance pass-class F1 at 0.908 ($κ$=0.636), while on factual support the judges are statistically indistinguishable (overlapping confidence intervals), so no single model dominates. At comparable F1, the judges still differ substantially in pass-rate drift, false positive rate, and false negative rate. Scalar F1 obscures this directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce. Calibrating the judge is therefore a prerequisite for using citation rubrics as reward signals, and our results show that this calibration does not require the most expensive available model.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper benchmarks 8 off-the-shelf LLM judges from 3 model families on a citation-quality rubric for deep-research source attribution, using 1,248 human-reviewed gold decisions. It finds cheaper judges are competitive, no single model dominates both rubric dimensions, and judges with similar F1 still differ substantially in directional bias that would shape an RLVR reward signal.

Key findings

  • Cheaper judges remain competitive: GPT-5-mini achieves the strongest source-relevance pass-class F1 at 0.908 (κ=0.636), while Claude Opus 4.6 has the highest factual-support point estimate at F1=0.750 (κ=0.701).
  • On factual support, all 95% confidence intervals overlap (from F1=0.649 [.56,.72] to 0.750 [.68,.82]), so no judge is statistically distinguishable on that dimension.
  • No single model dominates both dimensions: GPT-5-mini leads source relevance but ranks fourth on factual support (F1=0.710), while Claude Opus 4.6 leads factual support but ranks third on source relevance (F1=0.866).
  • At similar F1, judges differ in directional bias: all 8 judges under-predict source-relevance pass rate relative to the 79.3% gold rate (predicted 42.9%–72.0%), and factual-support false negative rates range from 0.183 (GPT-5.4-mini) to 0.470 (GPT-OSS-120B).

Methods

  • The authors construct the Deep-Research Citation Benchmark: a single long-form report spanning 25 topic domains, yielding 624 attribution-citation pairs; about 60% of attributed claims are adversarially edited using 19 strategies, with the rest left clean.
  • Gold labels cover 1,248 LLM-judged decisions across source relevance and factual support, all human-reviewed, including 378 hard cases adjudicated from judge disagreements (263 source relevance, 115 factual support); link accessibility is deterministic and excluded.
  • They evaluate 8 off-the-shelf LLM judges from Anthropic, Google, and OpenAI on all 624 pairs per dimension, measuring pass-class F1, Cohen’s κ, pass-rate drift, false positive rate, and false negative rate, with 95% bootstrap confidence intervals over 2,000 resamples.

Limitations

  • Stated: findings are limited to a single adversarial document, and sensitivity to prompt design and batching remain open questions.
  • Evident: two evaluated judges (GPT-5-mini and Claude Opus 4.6) also contributed to the gold-label council, though the paper argues this overlap does not drive the cost and quality findings.
  • Evident: factual-support gold pass rate is only 18.4%, a strongly imbalanced positive class that may affect F1 interpretation, and the benchmark is one report across 25 topics rather than a broad multi-document corpus.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.08700

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

25 of 43 cited sources checked · 12 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    In domains where no programmatic verifier exists, the community has converged on prompt-specific rubrics with weighted criteria, each scored by an LLM judge whose aggregate judgment serves as the scalar reward.

    “In domains where no programmatic verifier exists (medical advice, scientific writing, instruction following), the community has converged on prompt-specific rubrics with weighted criteria, each scored by an LLM judge whose aggregate judgment serves as the scalar reward”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Rezaei et al., 2026 source
    • cites Mahmoud et al., 2026 source
    • cites Liu et al., 2023 source

      The Method and Auto Chain-of-Thoughts sections (page not provided) describe an LLM judge using task-specific evaluation criteria to produce a score. But the paper weights possible scores by token probability, not rubric criteria. It does not establish community convergence across the named domains or report aggregate judgments being used as rewards; the Introduction discusses reward use only as a possibility. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    RAG evaluation frameworks such as RAGAS and ARES assess faithfulness at the answer level, treating the full context-answer pair as a unit rather than scoring individual attribution-citation pairs.

    “RAG evaluation frameworks such as RAGAS [Es et al., 2024] and ARES [Saad-Falcon et al., 2024] assess faithfulness at the answer level, treating the full context-answer pair as a unit rather than scoring individual attribution-citation pairs”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Es et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Saad-Falcon et al., 2024 source
  • Premisecitation

    Onweller et al. find that factual support accuracy drops by approximately 42% as retrieval depth scales from 2 to 150 tool calls.

    “Benchmarking 14 LLMs on citation quality in deep research agents, they find that factual support accuracy drops by approximately 42% as retrieval depth scales from 2 to 150 tool calls”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source
  • Interpretationcitation

    This asymmetry maps directly onto the reward hacking mechanism studied in rubric RL, where a judge with high FPR over-rewards bad citations, driving the trained model to exploit the judge's permissive behavior rather than learn the true quality signal.

    “This asymmetry maps directly onto the reward hacking mechanism studied in rubric RL [Mahmoud et al., 2026, Li, 2026], where a judge with high FPR over-rewards bad citations, driving the trained model to exploit the judge's permissive behavior rather than learn the true quality signal.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Mahmoud et al., 2026 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Li, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    Source relevance is better resolved, with GPT-5-mini at 0.908 [.89,.93] separating from lower-ranked models such as Claude Sonnet 4.6 at 0.700 [.66,.74].

    “Source relevance is better resolved, with GPT-5-mini at 0.908 [.89,.93] separating from lower-ranked models such as Claude Sonnet 4.6 at 0.700 [.66,.74].”
    Quote in the paperSubstantial claim
  • May combine two claims. 40
  • Classified as probably not a substantial claim. 43
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Citation grounding benchmarks such as ALCE and FactScore evaluate whether claims are supported by evidence, but operate over closed retrieval corpora rather than live hyperlinks and use single NLI-based scorers rather than comparing judge models.

    “Citation grounding benchmarks such as ALCE [Gao et al., 2023c] and FactScore [Min et al., 2023] evaluate whether claims are supported by evidence, but operate over closed retrieval corpora rather than live hyperlinks and use single NLI-based scorers rather than comparing judge models.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Gao et al., 2023c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Min et al., 2023 source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM: source relevance and factual support.

    “Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM, source relevance and factual support.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training recipe for tasks where output quality can be automatically checked.

    “Reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training recipe for tasks where output quality can be automatically checked”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Tyagi et al., 2026 source

      The Introduction describes RLVR as “a central recipe” for post-training on tasks where success can be cheaply and reliably checked. That supports its importance in this setting, but the source does not establish the stronger claim that it is the dominant recipe. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Mahmoud et al., 2026 source

      The Introduction (p. not provided) says RLVR has been highly effective in mathematics and coding, where correctness can be checked from a final answer or test suite. The paper does not say RLVR is the dominant post-training recipe for such tasks; that comparative claim is unsupported. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites DeepSeek-AI, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training.

    “Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Tyagi et al., 2026 source
    • cites Mahmoud et al., 2026 source
    • cites DeepSeek-AI, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A weaker supervisor's directional signal can still enable strong model improvement, suggesting judge imperfection does not necessarily degrade downstream training quality.

    “weak-to-strong generalization results show that a weaker supervisor's directional signal can still enable strong model improvement”
    • Also stated in: § 1 Introduction · hedged
      “Burns et al. [2023] show that directionally correct reward signals from weaker supervisors still enable strong model improvement, suggesting that judge imperfection does not necessarily degrade downstream training quality.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Burns et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Rubric-based rewards decompose response quality into explicit, independently scored criteria, making the training objective auditable and reducing the surface area for reward hacking.

    “Rubric-based rewards address this by decomposing response quality into explicit, independently scored criteria, making the training objective auditable and reducing the surface area for reward hacking”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Tyagi et al., 2026 source

      The Introduction and §3.2 state that rubric rewards decompose response quality into prompt-specific criteria scored independently by an LLM judge. The Reward judging section also describes a separate verdict and rationale for each criterion. Our copy does not establish that this makes the training objective auditable or reduces reward hacking. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Mahmoud et al., 2026 source

      The Introduction and §2.1 describe explicit, separately judged criteria that make rewards more interpretable than a holistic score. But the Introduction says this structure does not remove reward hacking, and §4 describes policies exploiting both verifier judgments and gaps in the rubric. The claim that rubrics reduce the surface area for reward hacking is therefore not supported by this paper. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Bai et al., 2022 source

      The “Simplicity and Transparency” section says the method encodes training goals in a list of natural-language principles to make objectives more transparent. Section 4.1 describes sampling a principle for each response comparison, not independently scoring a set of criteria. The copy does not establish that this approach reduces reward hacking. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Onweller et al. [2026] introduce the first evaluation framework for citation quality in deep research, pairing an AST-based parser with a three-dimension evaluator checking link accessibility, source relevance, and factual support.

    “Onweller et al. [2026] introduce the first evaluation framework for this task, pairing an AST-based parser that extracts attribution-citation pairs from markdown-formatted LLM output with a three-dimension evaluator”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source

  • Classified as probably not a substantial claim.

    Premisecitation

    We build on the two-stage source attribution evaluation pipeline of Onweller et al. [2026], which parses attribution-citation pairs from long-form model output and then scores each pair against the rubric criteria.

    “We build on the two-stage source attribution evaluation pipeline of Onweller et al. [2026], which parses attribution-citation pairs from long-form model output and then scores each pair against the rubric criteria.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Faithful citation generation is an active RLVR training target.

    “faithful citation generation of this kind is an active RLVR training target”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Nakano et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Menick et al., 2022 source

      The paper trains models to generate answers with supporting quotes, so citation quality is a reinforcement-learning target. But §2.8 (p. 7) and the Conclusion (p. 18) specify rewards learned from human preferences, not RLVR: mechanical checks establish that quotes are verbatim, while humans judge whether they support the answer. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Asai et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Menick et al. [2022] used a dedicated support model that scores whether each quoted passage supports the generated claim, with its binary predictions serving as the RL reward.

    “Menick et al. [2022] made the citation judgment explicit, using a dedicated support model that scores whether each quoted passage supports the generated claim, with its binary predictions serving as the RL reward.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Menick et al., 2022 source

      Section 2.8 (p. 7) describes a reward model that receives the selected evidence and predicts human pairwise preferences; it also predicts a binary Supported&Plausible judgment as an auxiliary task. The paper does not say that it judges each quoted passage separately or that those binary predictions serve as the RL reward: RL maximizes the reward model's score. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The quality of each citation decomposes into criteria applied per attribution-citation pair; two require an LLM judgment (source relevance and factual support) while a third (URL accessibility) is a deterministic check.

    “The quality of each citation decomposes into criteria applied per attribution-citation pair.Two of them require an LLM judgment, whether the source is topically relevant to the claim and whether it factually supports it, while a third, whether the cited URL is accessible, is a deterministic check.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Citation attribution benchmarks such as CiteME and CiteGuard test whether a model attributes a scientific claim to the correct source paper, a matching task over a fixed corpus.

    “Citation attribution benchmarks such as CiteME [Press et al., 2024] and CiteGuard [Choi et al., 2026] test whether a model attributes a scientific claim to the correct source paper, a matching task over a fixed corpus.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Press et al., 2024 source

      The Abstract describes CiteME as testing whether a model can identify the paper referenced by a scientific claim. But the Introduction distinguishes CiteME from benchmarks that retrieve from a small, pre-selected set of papers, and the contributions describe its attribution task as open-ended rather than limited to a fixed corpus. This verdict addresses CiteME, not the separately cited CiteGuard. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Choi et al., 2026 source

      Section 3.1 describes CiteME as a benchmark of 130 excerpts with missing citations and says CiteGuard is an agent evaluated on it. The claim is therefore right about testing citation attribution, but CiteGuard is not itself a benchmark, and the paper does not describe the task as matching papers from a fixed candidate corpus. Section 4.2 also reports that CiteGuard identifies valid citations beyond the original references. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Weak judges produce proxy-reward gains that do not transfer to stronger reference panels.

    “recent work shows that weak judges produce proxy-reward gains that do not transfer to stronger reference panels”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Mahmoud et al., 2026 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.

    “Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zheng et al., 2023 source

      The Introduction (p. 2) lists position, verbosity, and self-enhancement bias among the limitations examined, and §3.3 (pp. 4–5) reports position and verbosity bias in evaluations of chat-assistant responses. The paper studies response evaluation rather than citation-specific rubric tasks. But §3.3 (p. 5) explicitly says the data cannot determine whether the models exhibit self-enhancement bias, so the claim overstates that part as an identified bias. (LLM, conf=0.89)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The OpenStax calculus page covers Taylor series, so the source relevance criterion passes for the example claim.

    “The OpenStax calculus page covers Taylor series, so this criterion passes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Of the 378 non-unanimous decisions, 263 were source relevance and 115 were factual support, with none on link accessibility.

    “Of the 378 non-unanimous decisions, 263 were source relevance and 115 were factual support, with none on link accessibility.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Agreement among the labeling judges was substantial, with mean pairwise Cohen's κ of 0.62 (range 0.50 to 0.72) on source relevance and 0.67 (range 0.54 to 0.83) on factual support.

    “Agreement among the labeling judges was substantial, with mean pairwise Cohen's κ of 0.62 (range 0.50 to 0.72) on source relevance and 0.67 (range 0.54 to 0.83) on factual support.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    We evaluate 8 LLM judges from 3 model families (Anthropic, Google, and OpenAI), accessed through their respective cloud inference APIs.

    “We evaluate 8 LLM judges from 3 model families (Anthropic, Google, and OpenAI), accessed through their respective cloud inference APIs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The gold labels reflect the adversarial design: link accessibility passes at 98.4%, source relevance at 79.3%, and factual support at only 18.4%, the last kept intentionally low to stress-test judge discrimination on the hardest citation quality dimension.

    “The gold labels reflect the adversarial design of the benchmark.Link accessibility passes at 98.4%, source relevance at 79.3%, and factual support at only 18.4%, the last kept intentionally low to stress-test judge discrimination on the hardest citation quality dimension.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Each judge is applied to all 624 source relevance pairs and all 624 factual support pairs in the benchmark, yielding 1,248 decisions.

    “Each judge is applied to all 624 source relevance pairs and all 624 factual support pairs in the benchmark, yielding 1,248 decisions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On factual support, every 95% confidence interval overlaps (from 0.649 [.56,.72] to 0.750 [.68,.82]), so no model is statistically distinguishable and the apparent Opus lead is within noise.

    “but every 95% confidence interval overlaps on this dimension (from 0.649 [.56,.72] to 0.750 [.68,.82]), so no model is statistically distinguishable and the apparent Opus lead is within noise.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most disagreements were near-consensus rather than deep splits, with 44% showing a single dissenting judge and only 12% evenly split, while 29% included at least one partial (0.5) score, indicating that the binary rubric is occasionally applied as a graded one.

    “Most disagreements were near-consensus rather than deep splits, with 44% showing a single dissenting judge and only 12% evenly split, while 29% included at least one partial (0.5) score, indicating that the binary rubric is occasionally applied as a graded one.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On factual support, Claude Opus 4.6 posts the highest point estimate (F1=0.750, κ=0.701) and GPT-OSS-120B the lowest (F1=0.649).

    “On factual support, the hardest dimension with a gold pass rate of only 18.4%, Claude Opus 4.6 posts the highest point estimate (F1=0.750,κ=0.701) and GPT-OSS-120B the lowest (F1=0.649)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    The OpenStax source defines a Taylor series as an infinite sum of terms, contradicting the example claim of a finite sum, so the factual support criterion fails.

    “The source defines a Taylor series as an infinite sum of terms, contradicting the claim of a finite sum, so this criterion fails.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-5-mini leads on source relevance (F1=0.908) but ranks fourth on factual support (F1=0.710), while Claude Opus 4.6 leads on factual support (F1=0.750) but ranks third on source relevance (F1=0.866), so no single model dominates across both dimensions.

    “GPT-5-mini leads on source relevance (F1=0.908)but ranks fourth on factual support (F1=0.710), while Claude Opus 4.6 leads on factual support (F1=0.750) but ranks third on source relevance (F1=0.866),so no single model dominates across both dimensions”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Deep-Research Citation Benchmark is a single long-form report spanning 25 topic domains.

    “The Deep-Research Citation Benchmark is a single long-form report spanning 25 topic domains”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Most attributed claims in the benchmark are left clean, while the rest carry intentional factual errors, live-but-irrelevant sources, and plausible-but-unsupporting sources designed to stress-test each rubric criterion independently.

    “Most attributed claims are left clean, while the rest carry intentional factual errors, live-but-irrelevant sources, and plausiblebut-unsupporting sources designed to stress-test each rubric criterion independently”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The 378 multi-judge disagreement cases are the hardest instances in the benchmark, the pairs where the 6 gold-label judges did not agree and a human adjudicated the final label.

    “The 378 multi-judge disagreement cases are the hardest instances in the benchmark, the pairs where the 6 gold-label judges did not agree and a human adjudicated the final label.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Parsing the report with the pipeline yields 624 attribution-citation pairs.

    “Parsing the report with the pipeline yields 624 attribution-citation pairs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On factual support, all confidence intervals overlap and no judge is statistically distinguishable from the rest.

    “on factual support all confidence intervals overlap and no judge is statistically distinguishable from the rest.”
    • Also stated in: § wrong location incorrect geographic location. · plain
      “On factual support every interval overlaps, so no model is statistically distinguishable on that dimension.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Cost does not predict accuracy or track quality.

    “cost does not predict accuracy.”
    • Also stated in: § 5.3 Cost and Quality · plain
      “Cost does not track quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among the 8 evaluated judges, pairwise agreement ranges from 76.9% (GPT-5.4-mini and Claude Sonnet 4.6, κ=0.535) to 93.6% (GPT-OSS-120B and Gemini 3.1 Pro, κ=0.860), with 21 of 28 pairs between 82% and 88%.

    “Among the 8 evaluated judges, pairwise agreement ranges from 76.9% (GPT-5.4-miniand Claude Sonnet 4.6, κ=0.535) to 93.6% (GPT-OSS-120B and Gemini 3.1 Pro, κ=0.860), with 21 of 28 pairs between 82% and 88%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Deep-Research Citation Benchmark applies adversarial edits to roughly 60% of attributed claims using the 19 strategies below (some claims receive more than one).

    “The Deep-Research Citation Benchmark applies adversarial edits to roughly 60% of attributed claims using the 19 strategies below (some claims receive more than one).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The adversarial edits applied to roughly 60% of attributed claims are representative of the citation errors that occur in practice.

  • Classified as probably not a substantial claim.

    Findingreasoning

    The relevance rubric asks whether the source provides supporting information for the claims, which overlaps with the factual support criterion.

    “The relevance rubric asks whether the source”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On source relevance, judges exhibit high precision but moderate recall, rarely marking irrelevant sources as relevant while frequently missing relevant ones.

    “On source relevance, judges exhibit high precision but moderate recall, rarely marking irrelevant sources as relevant while frequently missing relevant ones.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

  • May combine two claims.

    Findingown data

    On factual support, GPT-5.4-mini leads the subset (F1=0.780), improving from its full-set score of 0.671 and displacing Claude Opus 4.6 (F1=0.672), which led the full set.

    “On factual support, GPT-5.4-minileads the subset (F1=0.780),improving from its full-set score of 0.671 and displacing Claude Opus 4.6 (F1=0.672),which led the full set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The lowest agreement involves the pair with the largest pass-rate difference, where GPT-5.4-mini passes 49.2% of evaluated pairs while Sonnet 4.6 passes 28.4%.

    “The lowest agreement involves the pair with the largest pass-rate difference, where GPT-5.4-minipasses 49.2% of evaluated pairs while Sonnet 4.6 passes 28.4%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

  • May combine two claims.

    Findingown data

    GPT-5-mini, the third-cheapest model, achieves the highest source relevance F1 (0.908).

    “GPT-5-mini, the third-cheapest model, achieves the highest source relevance F1 (0.908)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

  • May combine two claims.

    Findingown data

    On source relevance, GPT-5-mini remains the strongest (F1=0.832) but drops 0.076 from its full-set score, while Claude Sonnet 4.6 degrades most severely, falling from F1=0.700 to 0.420.

    “On source relevance, GPT-5-mini remains the strongest (F1=0.832)but drops 0.076 from its full-set score, while Claude Sonnet 4.6 degrades most severely, falling from F1=0.700 to 0.420.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On the adjudicated cases, no single labeling judge reliably matched the human label, with per-model agreement ranging from 38% to 85% on source relevance and 43% to 67% on factual support.

    “On the adjudicated cases, no single labeling judge reliably matched the human label, with per-model agreement ranging from 38% to 85% on source relevance and 43% to 67% on factual support.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the adjudicated subset, all 8 judges degrade on source relevance, while factual support behavior is mixed, with GPT-5.4-mini, Claude Haiku 4.5, and Gemini 3.1 Flash Lite improving on the adjudicated cases.

    “On this subset, all 8 judges degrade on source relevance, while factual support behavior is mixed, with GPT-5.4-mini,Claude Haiku 4.5, and Gemini 3.1 Flash Lite improving on the adjudicated cases.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    The model that scores each rubric criterion acts as the reward model, so designing the training rubric amounts to designing the operationalized goal.

    “The model that scores each criterion acts as the reward model, so designing the training rubric amounts to designing the operationalized goal.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    A human reviewer examined the full set of council decisions, corrected errors caught on review, and adjudicated the final label for every case in which the council was not unanimous.

    “A human reviewer then examined the full set of council decisions, corrected errors caught on review, and adjudicated the final label for every case in which the council was not unanimous.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Link accessibility labels are produced deterministically by an HTTP check that passes when the cited URL returns a 200 status, so this criterion needs no LLM and no human review and is excluded from the judge comparison.

    “Link accessibility labels are produced deterministically by an HTTP check that passes when the cited URL returns a 200 status, so this criterion needs no LLM and no human review, and we exclude it from the judge comparison.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A 13B model with rubric fine-tuning matches GPT-4 evaluation quality, establishing that rubric-following ability is separable from model scale.

    “Kim et al. [2024] show that a 13B model with rubric fine-tuning matches GPT-4 evaluation quality, establishing that rubric-following ability is separable from model scale”
    • Also stated in: § 1 Introduction · plain
      “Kim et al. [2024] show that a 13B model fine-tuned on rubric evaluation matches GPT-4 quality, so rubric-following ability is separable from model scale.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kim et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Confidence intervals are the 2.5th and 97.5th percentiles of the metric over 2,000 bootstrap resamples of the pairs.

    “Confidence intervals are the 2.5th and 97.5th percentiles of the metric over 2,000 bootstrap resamples of the pairs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Source relevance and factual support labels were produced by a council of 6 LLM judges that independently ran the evaluation pipeline.

    “Source relevance and factual support labels were produced by a council of 6 LLM judges that independently ran the evaluation pipeline.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Our setting scores the quality of citations to live web sources along independent relevance and factual-support criteria, and compares candidate judge models rather than proposing a single attributor.

    “Our setting differs in that we score the quality of citations to live web sources along independent relevance and factual-support criteria, and we compare candidate judge models rather than proposing a single attributor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Self-RAG trains a model to emit per-citation reflection tokens inline with generation, using an LLM judge to produce the training signal.

    “Self-RAG [Asai et al., 2024] internalizes this judgment further, training a model to emit percitation reflection tokens (including [IsSupported]) inline with generation, using an LLM judge to produce the training signal.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Asai et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Our benchmark supplies the source attribution pipeline with a controlled, adversarial input and a set of human-reviewed gold labels.

    “Our benchmark supplies that pipeline with a controlled, adversarial input and a set of human-reviewed gold labels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Domain diversity reduces the risk that a judge's domain knowledge, rather than its rubric-following ability, drives performance.

    “Domain diversity reduces the risk that a judge's domain knowledge, rather than its rubric-following ability, drives performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Faithful citation generation has been an RL training target since WebGPT, which used human preference over cited answers as a reward signal.

    “Faithful citation generation has been an RL training target since WebGPT [Nakano et al., 2022], which used human preference over cited answers as a reward signal.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nakano et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Both judge scores and gold labels are binarized at a threshold of 0.5, where a score ≥ 0.5 is treated as pass, the positive class.

    “Both judge scores and gold labels are binarized at a threshold of 0.5, where a score ≥ 0.5 is treated as pass, the positive class.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Prior citation RL systems use a single, large, or purpose-built model; none study whether off-the-shelf smaller models can provide an equally reliable reward signal or characterize directional bias.

    “Each of these systems uses a single, large, or purpose-built model.None study whether off-the-shelf smaller models can provide an equally reliable reward signal, nor do they characterize the directional bias (pass-rate drift, false positive rate, false negative rate) that determines what the training loop actually reinforces.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The verification bottleneck can be reduced by an order of magnitude through batching and lower-cost judges.

    “the verification bottleneck can be reduced by an order of magnitude through batching and lower-cost judges”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Trivedy et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Two of the 8 judges (GPT-5-mini and Claude Opus 4.6) also contributed to the gold-label council, but because they span the cost range and the human-adjudicated subset offers a council-independent comparison, this overlap does not drive the cost and quality findings.

    “Two of the 8 judges (GPT-5-mini and Claude Opus 4.6) also contributed to the gold-label council, but because they span the cost range and the human-adjudicated subset offers a council-independent comparison, this overlap does not drive the cost and quality findings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    A judge with high FNR under-rewards genuinely supported claims, producing signal sparsity and potentially training models to over-hedge or under-cite.

    “A judge with high FNR instead under-rewards genuinely supported claims, producing signal sparsity and potentially training models to over-hedge or under-cite.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Batch scoring and prompt caching would reduce costs further.

    “Batch scoring, which evaluates all criteria in a single call, and prompt caching [Lumer et al., 2026b], which amortizes shared context across tool calls in long-horizon agentic workloads, would reduce costs further [Trivedy et al., 2026].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Lumer et al., 2026b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Trivedy et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    These findings are limited to a single adversarial document.

    “These findings are limited to a single adversarial document”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Scalar F1 obscures directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce.

    “Scalar F1 obscures this directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Judge selection is a component of the agent design problem in production systems, where the choice of model, tool, and scoring strategy must be made jointly.

    “More broadly, judge selection is a component of the agent design problem in production systems, where the choice of model, tool, and scoring strategy must be made jointly [Lumer et al., 2026a].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lumer et al., 2026a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Selecting judges with high pairwise agreement, or ensembling across disagreeing judges, can reduce this noise without a stronger model.

    “Selecting judges with high pairwise agreement, or ensembling across disagreeing judges, can reduce this noise without a stronger model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Human adjudication is essential.

    “and human adjudication is essential.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.
63 of 63

  • The paper relies on this without support.

    The human-reviewed rubric decisions in the Deep-Research Citation Benchmark are a reliable ground truth for evaluating judge accuracy.

    “using the Deep-Research Citation Benchmark of 1,248 human-reviewed rubric decisions”

  • The paper relies on this without support.

    The adversarial edits applied to roughly 60% of attributed claims are representative of the citation errors that occur in practice.

    “The Deep-Research Citation Benchmark applies adversarial edits to roughly 60% of attributed claims using the 19 strategies below”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting descriptions of how many claims were edited

    Major

    Section 3.1 says most attributed claims were left clean, but the construction description says about 60% were adversarially edited. If about 60% were edited, most were not left clean.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2024.

Authors
Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K.
Affiliations
PricewaterhouseCoopers (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.08700
arXiv
2607.08700
OpenAlex
W7167960347
Cited by
0
References
34
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
citation verification, source attribution, LLM-as-a-judge, rubric-based assessment, calibration, source relevance, benchmarking, false positive and false negative rates

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review