DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

May combine two claims.

The granularity of rubrics in ResearcherBench and ResearchRubrics is insufficient, and correctly evaluating them usually requires the model to have specific internal knowledge.

“However, the granularity of their rubrics is insufficient, and correctly evaluating them usually requires the model having specific internal knowledge.”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sharma et al., 2025 sourceDisputed

  The Overview of RESEARCHRUBRICS (p. 4) describes 20–43 expert-refined criteria per prompt as enabling fine-grained assessment, contrary to the claim that its rubrics lack granularity. The evaluator instructions (p. 25) direct the judge to rely solely on evidence in the document, not specific internal knowledge. The cited paper does discuss coarse metrics in other benchmarks (p. 4), but does not characterize ResearchRubrics that way. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

BrowseComp, XBench, Deep Research Bench, and WideSearch focus on tasks that have fixed answers, such as specific entities or numbers.

“BrowseComp [Wei et al., 2025], XBench [Chen et al., 2025], Deep Research Bench [FutureSearch et al., 2025], WideSearch [Wong et al., 2025] focus on tasks that have fixed answers, such as specific entities or numbers.”

• cites Wei et al., 2025 sourceSupported
• cites Chen et al., 2025 sourcePartly supported

  The recruitment tasks use verification answers and, for Info-to-People, a target individual (3.2 BENCHMARKING RECRUITMENT, p. 6), supporting the description of some xbench tasks as having fixed answers. But the marketing benchmark scores how well recommended influencers fit a client-specific ideal persona (Influencer search, p. 10), rather than requiring a fixed entity or number, so the characterization does not cover xbench as a whole. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites FutureSearch et al., 2025 sourceNot checked — no full text of the source

May combine two claims.

Fixed-answer setups capture whether an agent can locate and extract relevant information, but only partially reflect real-world user needs: they rarely test whether the system can decide what to look for under open-ended goals, or how to integrate findings into a coherent narrative.

“These setups capture whether an agent can locate and extract relevant information, but they only partially reflect real-world user needs: they rarely test whether the system can decide what to look for under open-ended goals, or how to integrate findings into a coherent narrative.”

• cites Wei et al., 2025 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Chen et al., 2025 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites FutureSearch et al., 2025 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These fixed-answer benchmarks primarily assess whether the agent can locate and retrieve the correct information, testing planning, reasoning, and discerning information sources.

“These benchmarks primarily assess whether the agent can locate and retrieve the correct information, testing the agent's abilities in planning, reasoning, and discerning information sources.”

• cites Wei et al., 2025 sourceSupported
• cites Chen et al., 2025 sourcePartly supported

  The recruitment benchmark describes agents searching for talent information and being scored against verification questions or annotated sources (Section 3.2, pp. 5–7). That supports the information-finding part of the claim, but the paper does not characterize these as fixed-answer benchmarks that primarily test planning, reasoning, and discernment of information sources; it describes open-ended responses assessed for accuracy and hallucinations. (LLM, conf=0.81)

  Experimental — accuracy not yet measured
• cites FutureSearch et al., 2025 sourceNot checked — no full text of the source

May combine two claims.

In these full-report benchmarks, the rubrics are overly coarse and weakly interpretable, and may require judge LLMs to rely on out-of-distribution or unverifiable internal knowledge, leading to inaccurate and hard-to-trust scores.

“moreover, the rubrics are overly coarse and weakly interpretable, and may require judge LLMs to rely on out-of-distribution or unverifiable internal knowledge, leading to inaccurate and hard-to-trust scores”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourcePartly supported

  Section 2.3.4 (page not provided) says that evaluating DeepResearch reports is difficult because the criteria lack clear evaluation standards, lending some support to the concern about interpretable rubrics. It does not say that full-report benchmark rubrics are overly coarse or that judges rely on out-of-distribution or unverifiable internal knowledge, producing inaccurate scores. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites Wang et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

This LLM-defined criteria approach may introduce anchoring bias.

“This approach may introduce anchoring bias [Du et al., 2025, Fan et al., 2025, Wang et al., 2025a, Sharma et al., 2025].”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourceCould not verify

  Section 2.3.4 describes LLM-as-a-Judge criteria and prompt refinement to align scores with human experts, but does not discuss anchoring bias. Searches for “anchoring bias” and bias associated with evaluation criteria found no passage addressing the claimed risk. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites Wang et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The second category of benchmarks evaluates full research reports along dimensions such as comprehensiveness, insight, and citation quality.

“The second category evaluates full research reports, typically along dimensions such as comprehensiveness, insight, and citation quality [Du et al., 2025, Xu et al., 2025, Fan et al., 2025].”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourceSupported

May combine two claims.

These full-report benchmarks suffer from structural issues: their evaluation criteria are often defined directly by LLMs, which can introduce systematic misalignment with human expert judgments.

“However, these benchmarks suffer from structural issues: their evaluation criteria are often defined directly by LLMs, which can introduce systematic misalignment with human expert judgments;”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourcePartly supported

  Section 2.3.4 says report evaluation lacks clear standards and describes a process to align LLM-as-a-Judge scores with human expert judgments, supporting the concern about alignment. It does not say that full-report benchmarks often have criteria defined directly by LLMs; for this paper's framework, it describes previously defined scoring guidelines and manual prompt refinement. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Wang et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Other existing deep-research benchmarks rely on evaluation criteria that are overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret.

“Others rely on evaluation criteria that are either overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret.”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourcePartly supported

  The “Evaluation of DeepResearch” section says existing benchmarks are inadequate for assessing comprehensive reports, and §2.3.4 describes unclear evaluation standards and the need to align LLM scores with human experts. Our copy does not establish that other benchmarks use overly coarse or LLM-defined criteria, or that those criteria cause biased, hard-to-verify or hard-to-interpret scores. Page numbers were not available in the indexed passages. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

May combine two claims.

Since LLM generations are prone to hallucinations, especially in long-context settings, the authors adopt a self-evaluation strategy to mitigate this issue.

“Since LLM generations are prone to hallucinations [Simhi et al., 2025, Huang et al., 2025]-especially in long-context settings [Farquhar et al., 2024, Liu et al., 2025]-we adopt a self-evaluation strategy to mitigate this issue.”

• cites Simhi et al., 2025 sourceSupported
• cites Huang et al., 2025 sourceSupported
• cites Farquhar et al., 2024 sourcePartly supported

  The Main section calls hallucinations a critical problem for LLMs, supporting the general concern. But Farquhar et al. do not say hallucinations are especially prevalent in long-context settings; the Methods section instead says context passages were omitted from certain tests because providing them made accuracy too high to meaningfully study confabulations. The citing authors’ decision to adopt self-evaluation is their own, not a claim attributed to this source. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In ResearcherBench and ResearchRubrics, the evaluation criteria design involves human experts, which can better reflect human expert cognition to some extent.

“In contrast, the evaluation criteria designing in ResearcherBench and ResearchRubrics involve human expert, which can better reflect human expert cognition to some extent [Xu et al., 2025, Sharma et al., 2025].”

• cites Xu et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sharma et al., 2025 sourceSupported

Classified as probably not a substantial claim.

Rubrics designed by human experts help ensure the consistency of LLM judge scores with human expert preferences to some extent.

“Rubrics designed by human experts also help ensure the consistency of LLM judge scores with human expert preferences to some extent [Pathak et al., 2025, Hashemi et al., 2024].”

• cites Pathak et al., 2025 sourcePartly supported

  The Introduction (p. 2) reports that question-specific rubrics improve code-evaluation accuracy, and Data Preprocessing (p. 8) describes comparing LLM marks with human-grader marks for agreement. The paper supports some alignment in grading student code, but it does not establish that human-designed rubrics ensure LLM judge scores match human expert preferences more generally. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Hashemi et al., 2024 sourcePartly supported

  The Evaluation Rubric Construction section (page not provided) says the authors wrote the rubric questions, but does not identify their designers as human experts. The Introduction and §2 (pages not provided) report that using those questions with judge-specific calibration improves agreement with human judgments; they do not show that expert design of the rubric itself ensures that agreement. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

DeepResearch Bench, DeepResearch-ReportEval, LiveResearchBench, and Researcher Bench evaluate whether citations support the claims, but this only measures citation accuracy and does not necessarily confirm that the information is correct.

“It is also noteworthy that DeepResearch Bench, DeepResearch-ReportEval, LiveResearchBench, and Researcher Bench evaluate whether the citations support the claims, but this only measures the accuracy of the citations and does not necessarily confirm that the information is correct (e.g., false information from unofficial sources)”

• cites Du et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fan et al., 2025 sourcePartly supported

  The paper’s ‘Ensuring Source Reliability and Factuality’ section distinguishes citation accuracy from source reliability, supporting the point that a citation can accurately reflect an unreliable source. Section 2.3.3 assesses whether cited sources support claims. But the ‘Evaluation of DeepResearch’ section describes DeepResearch Bench as comparing reports with references and ResearcherBench as evaluating insight quality; it does not establish that all four named benchmarks evaluate claim–citation support. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Wang et al., 2025aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Each rubric is derived directly from expert reports and encodes a concrete factual or inferential requirement.

“Each rubric is derived directly from expert reports and encodes a concrete factual or inferential requirement.”

May combine two claims.

The strongest systems fail to pass more than 50% of the rubrics, with especially large deficits in Information Recall and Analysis.

“even the strongest systems fail to pass more than 50% of the rubrics, with especially large deficits in Information Recall and Analysis.”

May combine two claims.

Even the best-performing system still fails to satisfy more than half of the rubrics, indicating a substantial remaining gap between current DRSs and human experts.

“However, even the best-performing system still fails to satisfy more than half of the rubrics, indicating a substantial remaining gap between current DRSs and human experts.”

Classified as probably not a substantial claim.

After filtering, 132 articles were retained.

“After filtering, 132 articles were retained (see Appendix C for the full list).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

DRB-v2 contains 132 tasks, with 66 in English and 66 in Chinese.

“| Total | 66 | 66 | 132 |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On average, each task contains 52.902 rubrics for InfoRecall, 12.773 rubrics for Analysis, and 5.652 rubrics for Presentation.

“On average, each task contains 52.902 rubrics for InfoRecall, 12.773 rubrics for Analysis, and 5.652 rubrics for Presentation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Batch size 50 achieves the best F1-score while maintaining reasonable cost, making it the optimal choice for our evaluation pipeline.

“Batch size 50 achieves the best F1-score while maintaining reasonable cost, making it the optimal choice for our evaluation pipeline.”

May combine two claims.

A batch size of 50 offers the best balance, delivering strong accuracy while keeping evaluation overhead manageable.

“The cost-accuracy tradeoff curve (Figure ??) illustrates that a batch size of 50 offers the best balance, delivering strong accuracy while keeping evaluation overhead manageable.”

May combine two claims.

Performance remains stable across research topics for all evaluated systems, with all models showing no significant topic-based variation (p > 0.05).

“Performance remains stable across research topics for all evaluated systems, with all models (OpenAI, Gemini, Doubao, Qwen, Grok, Perplexity) showing no significant topic-based variation (p > 0.05).”

May combine two claims.

Only Grok shows a marginally significant difference (p = 0.034), suggesting a slight sensitivity to language variation.

“Only Grok shows a marginally significant difference (p = 0.034), suggesting a slight sensitivity to language variation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rubric counts per task are concentrated within specific ranges for each dimension (InfoRecall, Analysis, Presentation).

“The distributions show the concentration of rubric counts within specific ranges for each dimension.”

The quote only partly states this claim.

There is a clear, consistent, substantial gap between current DRSs and human experts.

“revealing a substantial gap between current DRSs and human experts.”

May combine two claims.

The construction pipeline ensures that the criteria are atomic, verifiable, and aligned with human expert judgment.

“ensuring that the criteria are atomic, verifiable, and aligned with human expert judgment.”

• AssumptionRests on an assumption: Human expert judgment is the appropriate gold standard for evaluating deep research reports.

May combine two claims.

DeepResearch Bench II derives fine-grained, content-bearing rubrics from human expert reports, enabling the LLM judge to reject seemingly correct hallucinations and provide unbiased, verifiable evaluations aligned with human judgment.

“DeepResearch Bench II derives fine-grained, content-bearing rubrics from human expert reports, enabling the LLM judge to reject seemingly correct hallucinations and provide unbiased, verifiable evaluations aligned with human judgment.”

• Also stated in: § E.2 Rubric Scoring Prompt · plain

  “DeepResearch Bench II derives fine-grained, content-bearing rubrics from human expert reports, enabling the LLM judge to reject seemingly correct hallucinations and provide unbiased, verifiable evaluations aligned with human judgment.”

• AssumptionRests on an assumption: Human expert judgment is the appropriate gold standard for evaluating deep research reports.
• AssumptionRests on an assumption: Human expert reports provide a valid reference standard for evaluating deep research outputs.
• AssumptionRests on an assumption: An LLM judge can reliably evaluate model outputs against rubrics derived from human expert reports.

Probably not a substantial claim: may not pass S3 (load-bearing).

Human annotators' subjective judgment could still introduce bias, affecting the final results of the benchmark.

“Although we invited human experts to annotate the benchmark and implemented multiple rounds of review to ensure the quality of annotations, the subjective judgment of human annotators could still introduce bias, affecting the final results of the benchmark.”

May combine two claims.

This four-stage pipeline ensures each task and rubric is grounded in expert-authored literature, verifiable, fine-grained, and aligned with human evaluation standards.

“This four-stage pipeline ensures that each task and rubric is not only grounded in expert-authored literature but also verifiable, fine-grained, and aligned with human evaluation standards.”

• AssumptionRests on an assumption: Human expert judgment is the appropriate gold standard for evaluating deep research reports.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Even research reports authored by human experts are unlikely to satisfy all reviewers in a single pass.

“Even research reports authored by human experts are unlikely to satisfy all reviewers in a single pass, highlighting the inherently challenging nature of evaluating deep research systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

As most evaluated models are closed-source, they can only prevent the model from viewing source articles through the prompt as a suboptimal workaround.

“As most of the models evaluated are closed-source, we can only prevent the model from viewing the source articles through the prompt as a suboptimal workaround.”

Classified as probably not a substantial claim. May combine two claims.

The paper attempted to prevent the model from accessing source articles solely through the prompt, but could not fully eliminate the possibility of the model viewing the source articles.

“we attempted to prevent the model from accessing source articles solely through the prompt, but we could not fully eliminate the possibility of the model viewing the source articles.”

Probably not a substantial claim: may not pass S4 (based).

Benchmarks that rely on LLM-defined criteria can be misaligned with human experts.

“Benchmarks that rely on LLM-defined criteria can be misaligned with human experts.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

This expert curation step is crucial for aligning the rubric set with human evaluation standards and for improving both validity and reliability of the benchmark.

“This expert curation step is crucial for aligning the rubric set with human evaluation standards and for improving both the validity and reliability of our benchmark.”

• AssumptionRests on an assumption: Human expert judgment is the appropriate gold standard for evaluating deep research reports.

Classified as probably not a substantial claim.

They included a blocked list in the prompt to prevent the model from accessing source articles.

“To mitigate this risk, we included a blocked list in the prompt to prevent the model from accessing these source articles.”

May combine two claims.

They recruited domain experts across fields; over more than 400 human-hours, experts conducted in-depth review, discussion, and iterative refinement of all tasks and rubrics.

“Over the course of more than 400 human-hours, these experts conducted in-depth review, discussion, and iterative refinement of all tasks and rubrics.”

May combine two claims.

Overall, the leakage rate is relatively low, indicating that the blocked list in the prompt has been somewhat effective in limiting access to source articles.

“Overall, the leakage rate is relatively low, indicating that the blocked list in the prompt has been somewhat effective in limiting access to source articles.”

Classified as probably not a substantial claim. May combine two claims.

Each initially generated rubric is applied to evaluate its own source article; if the resulting Information Recall or Analysis dimension achieves an accuracy below 90%, the generation is considered unreliable and regenerated until the threshold is met.

“Each initially generated rubric is applied to evaluate its own source article using the evaluation script.If the resulting Information Recall or Analysis dimension achieves an accuracy below 90%, the generation is considered unreliable due to potential hallucination, and the LLM is prompted again to regenerate the task and rubrics until the threshold is met.”

Classified as probably not a substantial claim. May combine two claims.

This approach cannot guarantee 100% prevention of model access to source articles, and it may potentially affect the model's actual performance.

“However, this approach cannot guarantee 100% prevention of model access to the source articles, and it may potentially affect the model's actual performance.”

Classified as probably not a substantial claim. May combine two claims.

After self-evaluation, human annotators manually inspect all generated tasks and rubrics to ensure compliance with requirements, eliminating logically inconsistent or redundant rubrics and refining ambiguous phrasing.

“After the self-evaluation stage, human annotators manually inspect all generated tasks and rubrics to ensure compliance with the requirements in Section 3.1.2.This step primarily eliminates logically inconsistent or redundant rubrics and refines ambiguous phrasing to improve clarity and precision.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The source reports are typically written by domain experts over weeks or months and undergo multiple layers of validation, reflecting a high degree of credibility and expert consensus.

“These reports are typically written by domain experts over the course of weeks or months and undergo multiple layers of validation-from peer reviewers, editors, practitioners, and the general public-thus reflecting a high degree of credibility and expert consensus.”

Classified as probably not a substantial claim. May combine two claims.

During evaluation, they performed a secondary inspection of generated reports; if a report referenced the source article and correctly answered a question, they excluded that from the score and recorded the leakage rate.

“Additionally, during the evaluation, we performed a secondary inspection of the generated reports.If a report referenced the source article and correctly answered a question, we excluded that from the score and recorded the leakage rate.”

Classified as probably not a substantial claim.

By decomposing human-expert articles into fine-grained, verifiable rubrics, they enable a comparison between human-expert articles and model-generated articles.

“By decomposing human-expert articles into finegrained, verifiable rubrics, we enable a comparison between human-expert articles and model-generated articles.”

• AssumptionRests on an assumption: Human expert reports provide a valid reference standard for evaluating deep research outputs.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

During evaluation, they conducted a secondary check of generated reports and calculated the leakage rate of the articles.

“during the evaluation, we conducted a secondary check of the generated reports and calculated the leakage rate of the articles.”