Understanding DeepResearch via Reports

In DeepResearch Bench, DeepResearch-ReportEval, and LiveResearchBench, the specific evaluation criteria are defined by the LLM itself, with human expert involvement only during the review phase or final consistency assessment.

“In DeepResearch Bench, DeepResearch-ReportEval, and LiveResearchBench, the specific evaluation criteria are defined by the LLM itself, with human expert involvement only during the review phase or final consistency assessment.”

In these full-report benchmarks, the rubrics are overly coarse and weakly interpretable, and may require judge LLMs to rely on out-of-distribution or unverifiable internal knowledge, leading to inaccurate and hard-to-trust scores.

“moreover, the rubrics are overly coarse and weakly interpretable, and may require judge LLMs to rely on out-of-distribution or unverifiable internal knowledge, leading to inaccurate and hard-to-trust scores”

These full-report benchmarks suffer from structural issues: their evaluation criteria are often defined directly by LLMs, which can introduce systematic misalignment with human expert judgments.

“However, these benchmarks suffer from structural issues: their evaluation criteria are often defined directly by LLMs, which can introduce systematic misalignment with human expert judgments;”

Other existing deep-research benchmarks rely on evaluation criteria that are overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret.

“Others rely on evaluation criteria that are either overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret.”

DeepResearch Bench, DeepResearch-ReportEval, LiveResearchBench, and Researcher Bench evaluate whether citations support the claims, but this only measures citation accuracy and does not necessarily confirm that the information is correct.

“It is also noteworthy that DeepResearch Bench, DeepResearch-ReportEval, LiveResearchBench, and Researcher Bench evaluate whether the citations support the claims, but this only measures the accuracy of the citations and does not necessarily confirm that the information is correct (e.g., false information from unofficial sources)”

These benchmarks still do not verify the accuracy of recalled information in the same way as the first category benchmarks.

“Essentially, these benchmarks still do not verify the accuracy of recalled information in the same way as the first category benchmarks.”

Compared to fixed-answer benchmarks, open-ended benchmarks are better at assessing whether the Deep Research system knows which information should be recalled and its ability to analyze the information.

“Compared to the first category, these open-ended benchmarks are better at assessing whether the Deep Research system knows which information should be recalled and its ability to analyze the information.”

Some existing deep-research benchmarks do not adequately test a system's ability to analyze evidence and write coherent reports.

“Some do not adequately test a system's ability to analyze evidence and write coherent reports.”

The second category of benchmarks evaluates full research reports along dimensions such as comprehensiveness, insight, and citation quality.

“The second category evaluates full research reports, typically along dimensions such as comprehensiveness, insight, and citation quality [Du et al., 2025, Xu et al., 2025, Fan et al., 2025].”

This LLM-defined criteria approach may introduce anchoring bias.

“This approach may introduce anchoring bias [Du et al., 2025, Fan et al., 2025, Wang et al., 2025a, Sharma et al., 2025].”