CoChat
Sign Up
PreprintCited by 0Topic ModelingInformation Retrieval and Search BehaviorSoftware Engineering Research

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research

Open access
Hirsch, Eran, Wan, David, Wang, Han, Stengel-Eskin, Elias, Bansal, Mohit, Dagan, Ido·2026·arXiv
DOI 10.48550/arxiv.2608.24306arXiv 2608.24306
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web. Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall. Moreover, improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way. We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs. Furthermore, we propose a four-type taxonomy to categorize the discovered errors: hallucination, uncited input reliance, uncited output, or insufficient citations. Applying our method to three top-ranked open-source DR systems, we obtain actionable diagnostics. Almost every agent makes a lot of mistakes with the exception being those that summarize a single document. We find that the dominant error type varies systematically across agents, where the orchestrator mistakes are mostly citation-related. We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes. Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper proposes a local, human-validated evaluation method that traces citation and faithfulness errors in multi-agent deep research systems to the specific agent invocation that introduced them. Applying it to three open-source systems, it finds that most agents make substantial mistakes, that final-report errors often originate at the orchestrator, and that two simple interventions raise citation recall by 5% without reducing output quality.

Key findings

  • Almost every agent makes significant mistakes, except agents that summarize a single document: AI-Q searcher snippets 3.8% mistakes, MS-Agent searcher snippets 6.4% and synthesized summaries 0.9%, TrajectoryKit searcher synthesized summaries 14.9%; by contrast AI-Q researcher 70.8%, AI-Q searcher synthesized 62.0%, MS-Agent reporter 61.6%, TrajectoryKit orchestrator 92.5%.
  • The dominant error type varies systematically: deeper agents hallucinate less in stronger-model systems, with only 31% of AI-Q orchestrator mistakes being hallucinations versus 79% for its researcher, and none of MS-Agent orchestrator mistakes being hallucinations versus 78% for its reporter and 92% for its researcher.
  • At the system level, 84.7% of AI-Q final-report errors are attributed to the orchestrator, 52.6% for MS-Agent, and 100% for TrajectoryKit; for AI-Q roughly 31% of final-report errors are hallucinations, while for TrajectoryKit 95% are hallucinations.
  • Two interventions on AI-Q—replacing synthesized researcher notes with raw source snippets and instructing the orchestrator not to use uncited information—raise citation recall by 5% and citation precision by 3% to 7% percentage points without significantly changing RACE output quality.

Methods

  • The framework evaluates a single agent invocation locally against its own inputs, using LLM-as-judge attribution and entailment tests to classify synthesized outputs into hallucination, uncited input reliance, uncited output, or insufficient citations, and snippet outputs into snippet missing context.
  • It applies agent-level evaluation (sampling up to 10 sentences per agent across 20 DeepResearch Bench examples, roughly 200 sentences per agent) and system-level tracing that recursively follows final-report errors back to the responsible agent.
  • The study evaluates Nvidia AI-Q, MS-Agent, and TrajectoryKit; human validation used 50 AI-Q sentences with two annotators and arbitration, yielding Cohen’s κ=0.71 before arbitration and 76% exact matches with the method (κ=0.62), plus 75% agreement on supporting spans. Interventions were tested on 50 English DeepResearch Bench examples.

Limitations

  • The paper states that faithfulness and verifiability checks rely on LLM-as-judge prompts, mitigated by using prior-work attribution and entailment tests and by the human annotation study.
  • The paper states that the observation that dominant error type tracks model capability is observational rather than controlled; isolating the factor would require running the same system with the same model across roles.
  • The paper states that evaluation covers running-text sentences only, excluding tabular content from recall and system-level analysis; it also notes in the appendix that experiments used only English examples and leaves multilingual evaluation to future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2608.24306

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 11 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Deep research systems can hallucinate URLs.

    “Because DR systems can hallucinate URLs (Onweller et al., 2026), we use the tracing log to build a ground-truth list of retrieved URLs and filter out any hallucinated citations.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Onweller et al., 2026 source

      The Introduction says deep research agents risk producing hallucinated references, and §2.1 says their citations may link to inaccessible URLs. Neither passage establishes that the systems fabricate URLs specifically; an inaccessible URL could instead be a broken or blocked link, as §3.3.1 explains. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Premisecitation

    Current deep research systems exhibit poor citation recall.

    “current DR systems exhibit poor citation recall.”
    • Also stated in: § 1 Introduction · plain
      “The standard metric for citation quality, citation recall -the fraction of claims supported by their cited sources (Zhang et al., 2025;Gao et al., 2023) -reveals that current DR systems perform poorly (Han et al., 2026;Patel et al., 2025).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Han et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Patel et al., 2025 source
  • Premisecitation

    Improving citation recall in deep research systems is challenging because their complex multi-agent architectures pass information through multiple agents, and both content and citations can get corrupted along the way.

    “improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way.”
    • Also stated in: § 1 Introduction · plain
      “As content crosses agent boundaries, and often gets compressed, both core information as well as citations may get corrupted (Laban et al., 2026).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Laban et al., 2026 source

      The Benchmarking AI Systems for Document Editing section says errors can compound across long delegated workflows, and the Deletion vs. Corruption section describes distortion of document content. The paper does not establish that deep research systems pass information between multiple agents or that citations become corrupted, so it does not support the claimed explanation for difficulty improving citation recall. Page numbers were not available in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingreasoning

    Researcher notes consolidate several documents at once.

    “and consolidate several documents at once”
    Quote in the paperSubstantial claim
  • Findingown data

    Only 4% of TrajectoryKit's orchestrator errors are citation-related.

    “TrajectoryKit is the exception: only 4% of its orchestrator errors are citation-related, as discussed below.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 28
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    TrajectoryKit runs its agents on the substantially smaller open-weights model gpt-oss-20b, so its model capability does not vary with the depth of the agent.

    “Unlike the other two systems, TrajectoryKit runs its agents on a substantially smaller open-weights model gpt-oss-20b (OpenAI et al., 2025), so its model capability does not vary with the depth of the agent.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites OpenAI et al., 2025 source

  • May combine two claims.

    Findingnone given

    In the searcher hallucination example, the searcher's answer summarizes the pages retrieved for the query, but the highlighted data-sharing and database-integration commitments appear in none of them.

    “The searcher's answer summarizes the pages retrieved for the query, but the highlighted data-sharing and database-integration commitments appear in none of them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    TrajectoryKit runs the same model at every stage and shows no shift in hallucination rate, with hallucinations accounting for most mistakes at each of its agents.

    “TrajectoryKit is consistent with this explanation: it runs the same model at every stage and correspondingly shows no shift, with hallucinations accounting for most of the mistakes”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Only 31% of the AI-Q orchestrator's mistakes are hallucinations, against 79% for its researcher.

    “Only 31% of the AI-Q orchestrator's mistakes are hallucinations, against 79% for its researcher.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    84.7% of final-report errors in AI-Q originate at the orchestrator, and roughly 31% of them are hallucinations and the rest citation mistakes.

    “We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes.”
    • Also stated in: § 1 Introduction · plain
      “At the system level, 84.7% of final-report errors in AI-Q are attributed to the orchestrator (main) agent, where the majority of these are citation-related and a sizable 31% of the orchestrator's own errors are hallucinations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Error propagation chains include a searcher fabricating a claim that the researcher and orchestrator faithfully propagate, the orchestrator abstracting a researcher's fabricated claim while keeping its citations, and unsupported statistics traveling verbatim with a citation that never supported them.

    “Table 6 in the Appendix walks through three representative chains that illustrate example patterns: a searcher fabricates a claim that the researcher and the orchestrator faithfully propagate; the orchestrator abstracts a researcher's fabricated claim while keeping its citations; and unsupported statistics travel verbatim with a citation that never supported them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Even a one-sentence prompt intervention raises citation recall by 5% and citation precision by 3-7% without reducing output quality.

    “even a one-sentence prompt intervention raises citation recall by 5% and citation precision by 3-7% without reducing output quality”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both interventions substantially increase citation recall by 5% without significantly changing output quality.

    “both interventions substantially increase citation recall by 5% without significantly changing output quality”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The majority of final-report mistakes start at the orchestrator: 84.7% for AI-Q, 52.6% for MS-Agent, and 100% for TrajectoryKit.

    “The majority of mistakes start at the orchestrator.For AI-Q, 84.7% of final-report errors were attributed to the orchestrator, for MS-Agent 52.6%, and for TrajectoryKit 100%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    None of the MS-Agent orchestrator's mistakes are hallucinations, against 78% for the reporter and 92% for the researcher.

    “MS-Agent shows the same contrast even more sharply: none of the orchestrator's mistakes are hallucinations, against 78% for the reporter and 92% for the researcher.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    In TrajectoryKit, unlike MS-Agent, there is no shared file system and no source registry, so the tool-result channel is the only route by which subagent content reaches the orchestrator.

    “Unlike MS-Agent, there is no shared file system and no source registry, so this tool-result channel is the only route by which subagent content reaches the orchestrator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the two stronger-model systems, the dominant failure mode shifts from hallucination at shallower agents to citation errors at the orchestrator, while the smaller-model system hallucinates at every stage.

    “Also, that in the two strongermodel systems the dominant failure mode shifts from hallucination at the shallower agents to citation errors at the orchestrator, while the smallermodel system hallucinates at every stage.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    TrajectoryKit's orchestrator is also its weakest agent (92.5% error rate), so agent-level and system-level evaluations agree for TrajectoryKit.

    “TrajectoryKit is the opposite case: its orchestrator is also its weakest agent (92.5%), so the two levels of evaluation agree.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Two simple interventions-instructing the orchestrator to avoid using uncited information and replacing synthesized researcher notes with raw source snippets-raise citation recall by 5% and citation precision by 3 to 7 percentage points without reducing output quality.

    “Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.”
    • Also stated in: § 1 Introduction · plain
      “These interventions successfully raise citation recall by 5% and citation precision by 3% to 7% percentage points, without reducing output quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The searcher components that compress one document at a time have low mistake rates: AI-Q searcher snippets 3.8%, MS-Agent searcher snippets 6.4% and synthesized summaries 0.9%, and TrajectoryKit searcher synthesized summaries 14.9%.

    “The clear exceptions are the searcher components that compress one document at a time: the AI-Q searcher snippets (3.8%), the MS-Agent searcher snippets and synthesized summaries (6.4% and 0.9%, respectively), and the TrajectoryKit searcher synthesized summaries (14.9%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Although the orchestrator is the most common origin, a substantial share of final-report errors starts elsewhere, meaning several subagents propagate mistakes received as input up to the final report.

    “Although the orchestrator is the most common origin, a substantial share of final-report errors starts elsewhere (Table 2), which means several subagents propagate mistakes received as input up to the final report.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Final output mistakes are mostly citation-related errors: 70% of AI-Q's orchestrator-level mistakes and 99% of MS-Agent's are citation-related.

    “Final output mistakes are mostly citationrelated errors.For AI-Q 70% of the mistakes are citation-related mistakes at the orchestrator level, and for MS-Agent 99%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Hallucinations account for roughly 31% of AI-Q's final-report errors, with most originating at the orchestrator.

    “Hallucinations account for roughly 31% of AI-Q's final-report errors (30% of the orchestrator's 200 errors and 37% of the researcher's 35), with most originating at the orchestrator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The global citation recall for AI-Q is 58.7%, for MS-Agent is 28.5%, and for TrajectoryKit is 7.1%.

    “The global citation recall for AI-Q is 58.7%, for MS-Agent is 28.5%, and for TrajectoryKit is 7.1%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The researcher's note restates the fabricated commitment and attaches the URLs that the searcher provided for this claim.

    “The researcher's note restates the fabricated commitment and attaches the URLs that the searcher provided for this claim.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The orchestrator abstracts the fabricated procedure into a hedged, higher-level claim that is fully supported by the researcher's notes.

    “The orchestrator abstracts the fabricated procedure into a hedged, higher-level claim that is fully supported by the researcher's notes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In Chain 3, the highlighted figures were available to the researcher from other input documents, but the note cites only the OECD report, which does not support them.

    “The highlighted figures were available to the researcher from other input documents, but the note cites only the OECD report, which does not support them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The final-report sentence faithfully summarizes the researcher's notes and propagates their citations, so the orchestrator's local test passes as well.

    “The final-report sentence faithfully summarizes the researcher's notes and propagates their citations, so the orchestrator's local test passes as well.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    TrajectoryKit's chainanalysis call runs before any retrieval, consumes no documents, and produces no citations, so it is not an evaluation target and TrajectoryKit contributes no planner row to our analyses.

    “Because it runs before any retrieval, it consumes no documents and produces no citations, so it is not an evaluation target and TrajectoryKit contributes no planner row to our analyses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The figures and their citation travel together across the compression boundary into the report.

    “The figures and their citation travel together across the compression boundary into the report.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    System-level evaluation reveals errors that made it to the final report but can hide mistakes that specific agents made, while agent-level evaluation reveals errors an agent makes but not whether these errors affected the final report.

    “In analogy to system vs. unit testing, system-level evaluation tells us about the errors that made it to the final report, but can hide mistakes that specific agents made.Agent-level evaluation tells us about the errors that an agent makes, but not whether these errors affected the final report.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A researcher's output does not have [N] markers, so it is attributed structurally to the URLs that researcher surfaced through its own search and fetch results.

    “A researcher's output does not have [N] markers, so it is attributed structurally to the URLs that researcher surfaced through its own search and fetch results.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    TrajectoryKit is ranked #3 on the same DeepResearch Bench open-source leaderboard.

    “ranked #3 on the same leaderboard, which instantiates the same logical roles.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A snippet extracted from a larger source can remove necessary context, creating uninterpretable or even hallucinated information.

    “Zhang et al. (2023) identified that a snippet extracted from a larger source can remove necessary context, creating uninterpretable or even hallucinated information.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zhang et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The no-crawl design, extracting raw documents from the tracing log instead of re-crawling, sidesteps crawl-fragility issues.

    “We also adopt a no-crawl design, extracting raw documents from the tracing log instead of re-crawling, which sidesteps crawl-fragility issues (disagreement between crawlers, paywalls, dynamic content).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Nvidia AI-Q and MS-Agent are ranked #1 and #2, respectively, in the DeepResearch Bench open-source leaderboard as of May 2026.

    “These systems are ranked #1 and #2, respectively, in the dominant DeepResearch Bench open-source leaderboard (Du et al., 2025) as of May 2026.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Du et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    For AI-Q and MS-Agent, the orchestrator has one of the lowest agent-level error rates despite being the most common origin of final-report errors.

    “For the first two systems this is despite the orchestrator having one of the lowest agent-level error rates (30.9% for AI-Q and 16.8% for MS-Agent in Table 1), which illustrates the value of system-level evaluation on top of agent-level evaluation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The descent itself involves no additional model calls.

    “the descent itself involves no additional model calls”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    In the experiments, only English examples were used, and multilingual evaluation is left to future work.

    “In our experiments, we used only English examples and leave multilingual evaluation to future work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The global citation recall implementation follows standard practice of checking whether a target sentence needs a citation and whether it has citations that support it.

    “The implementation of the global citation recall in Step 1 follows standard practice (Zhang et al., 2025) of answering two questions: (1) whether the target sentence needs a citation, and (2) whether it has citations that support it.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisenone given

    The faithfulness and verifiability checks of the algorithm rely on LLM-as-judge prompts adapted from prior work.

    “The faithfulness and verifiability checks of our algorithm (Section 4) rely on LLM-as-judge prompts adapted from prior work.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The evaluation algorithm itself runs identically on all three systems.

    “the evaluation algorithm itself (Section 4.2) runs identically on all three systems”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    All checks are attribution and entailment tests drawn from prior work.

    “all checks are attribution and entailment tests drawn from prior work (Section 4.2)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The searcher nodes are evaluated exactly as agents are evaluated, because each performs the same compression of retrieved documents whose citation behavior the framework measures.

    “We nevertheless evaluate these nodes exactly as we evaluate agents, because each performs the same compression of retrieved documents whose citation behavior our framework measures.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The observation that the dominant error type tracks the capability of the model an agent runs is observational rather than controlled.

    “Our observation that the dominant error type tracks the capability of the model an agent runs (Sections 5 and 6.2) is observational rather than controlled.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    We do not treat search_web entries as an evaluation target because we don't have access to the raw document that the system retrieved, so the test that scores extracted content has no reliable reference.

    “We do not treat these entries as an evaluation target, because unlike the snippet leaves of AI-Q and MS-Agent we don't have access to the raw document that the system retrieved, so the test that scores extracted content has no reliable reference.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    An entailment check verifies that the union of spans supports the target sentence.

    “an entailment check verifies that the union of spans supports the target sentence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Structural attribution is the conservative choice in both cases: crediting a sentence with every source its author had at hand errs toward finding support rather than toward flagging a citation error.

    “Structural attribution is the conservative choice in both cases: crediting a sentence with every source its author had at hand errs toward finding support rather than toward flagging a citation error.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Because the orchestrator's sentence is faithful to the researcher note it cites, only recursing into the researcher reveals that the attached citation never supported the figures.

    “Because the orchestrator's sentence is faithful to the researcher note it cites, only recursing into the researcher reveals that the attached citation never supported the figures.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Study design and conductreasoning

    On failure of span validation, the method retries with self-critique feedback.

    “On failure, we retry with self-critique feedback.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The abstraction propagates the fabrication, masks it behind vaguer language, and keeps its citations.

    “The abstraction propagates the fabrication, masks it behind vaguer language, and keeps its citations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    If the final report cites a note marker, dropping this citation could result in citation-related errors if the note did support the content.

    “This could result in citation-related errors if the note did support the content.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    If the retry budget is exhausted, the case is recorded as evaluation failed and filtered out.

    “If the retry budget is exhausted, we record the case as evaluation failed and filter it out.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The mode of citation attribution follows from the output format rather than from the agent's position in the information flow, so the same role can be attributed differently across systems.

    “The mode follows from the output format rather than from the agent's position in the information flow, so the same role can be attributed differently across systems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Extracted spans are validated by requiring them to appear verbatim in the input texts.

    “We include two validations for the extracted spans: (a) spans must appear verbatim in the input texts”
    Quote in the paperSubstantial claim
50 of 50

  • The authors state this as an assumption.

    The inputs to an agent invocation are faithful to the sources, and each input information unit is supported by the citations associated with it.

    “From the agent's perspective, it is assumed that the inputs are faithful to the sources, and that each input info unit is supported by the citations associated with it.”

  • The authors state this as an assumption.

    Mistakes inherited from prior agent invocations are not attributed to the current agent.

    “The method is not concerned with mistakes that were possibly inherited from prior agent invocations.”

  • The paper relies on this without support.

    Deep research systems typically attach citations at the sentence level.

    “Because DR systems typically attach citations at the sentence level, we segment each synthesized output information unit into sentences and run the algorithm on each sentence independently”

  • The paper relies on this without support.

    The tracing log provides a complete ground-truth list of retrieved URLs.

    “we use the tracing log to build a ground-truth list of retrieved URLs and filter out any hallucinated citations.”

  • The paper relies on this without support.

    The 20 examples from DeepResearch Bench are representative of deep research tasks.

    “We evaluate on 20 examples from DeepResearch Bench (Du et al., 2025).”

  • The paper relies on this without support.

    Sampled sentences are representative of all agent invocations for each agent.

    “For each example, we sample up to 10 sentences per agent, yielding roughly 200 sentences per agent”

  • The paper relies on this without support.

    The LLM judge gpt-5-mini-2025-08-07 is a reliable evaluator for the classification tasks.

    “The LLM used as the judge is gpt-5-mini-2025-08-07 (Singh et al., 2026).”

  • The paper relies on this without support.

    Human annotator consensus is a valid ground truth for citation error classification.

    “a human annotation study (Section 4.3) shows substantial agreement between annotator consensus and algorithm verdicts.”

  • The paper relies on this without support.

    Note markers are meaningless to end users reading the final report.

    “These note markers are not standard URL citations and would be meaningless to end users reading the final report.”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Snippet evaluation is described as both entailment and verbatim testing

    Major

    Section 4.2.2 says snippets are evaluated with an entailment test against the source document, while the citation-attribution description says their claims are checked by a verbatim test rather than an entailment test.

  • Partially supported sentences receive conflicting verdicts

    Major

    The method says that spans supporting only part of a sentence produce a hallucination verdict. The pseudocode instead retries when the spans do not entail the sentence and returns EVAL_FAILED if the retries are exhausted; such cases are subsequently filtered out.

  • Figure 6 assigns span extraction to the wrong step

    Minor

    Figure 6 calls span extraction Step 2, whereas the method and appendix identify span extraction as Step 1 and citation classification as Step 2.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 7

  1. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint. To appear in Proceedings of ICLR 2026.
  2. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465-6488, Singapore. Associa- tion for Computational Linguistics. · DOI · Paper page
  3. Jacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, and Luke Zettlemoyer. 2025. Precise information control in long-form text gen- eration. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  4. Eran Hirsch, Aviv Slobodkin, David Wan, Elias Stengel- Eskin, Mohit Bansal, and Ido Dagan. 2025. LAQuer: Localized attribution queries in content-grounded generation. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15355-15370, Vi- enna, Austria. Association for Computational Lin- guistics. · DOI
  5. Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. 2025a. Webthinker: Empowering large reasoning models with deep research capability. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  6. Miriam Redi, Besnik Fetahu, Jonathan Morgan, and Dario Taraborelli. 2019. Citation needed: A tax- onomy and algorithmic assessment of wikipedia's verifiability. In The World Wide Web Conference, WWW '19, page 1567-1578, New York, NY, USA. Association for Computing Machinery. · DOI
  7. Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2025. LongCite: En- abling LLMs to generate fine-grained citations in long-context QA. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5098-5122, Vienna, Austria. Association for Compu- tational Linguistics. · DOI

Authors
Hirsch, E., Wan, D., Wang, H., Stengel-Eskin, E., Bansal, M., & Dagan, I.
Affiliations
  • Bar-Ilan University (IL)
  • University of North Carolina at Chapel Hill (US)
  • The University of Texas at Austin (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2608.24306
arXiv
2608.24306
OpenAlex
W7204232084
Semantic Scholar
862b20a73298511a9ce620e430037f418b8fdc9a
Cited by
0
References
0
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1846185, 2112635, CAREER)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
deep research, multi-agent systems, hallucinations, faithfulness evaluation, error taxonomy, citation errors, error diagnostics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review