ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The accompanying SurveyBench evaluates generated surveys across reference, outline, and content quality, showing significant improvements over prior methods.

“The accompanying SurveyBench evaluates generated surveys across reference, outline, and content quality, showing significant improvements over prior methods.”

• cites Yan et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Early works leveraged LLMs to improve literature comprehension and survey writing, achieving better coherence compared to sentence extraction methods.

“Early works leveraged LLMs to improve literature comprehension and survey writing Wang et al. (2024); Hu et al. (2025), achieving better coherence compared to sentence extraction methods.”

• cites Wang et al. (2024) sourcePartly supported

  The Ablation study reports that AutoSurvey-generated surveys improved answers to literature questions, and the Results report better structure than naive RAG-based LLM generation. The cited paper does not compare coherence with sentence-extraction methods, so that comparator is unsupported. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Hu et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Several benchmarks have been proposed to evaluate the performance of fact checking in both the general domain and the scientific domain.

“Building on this pipeline, several benchmarks have been proposed to evaluate the performance of fact checking in both the general domain Thorne et al. (2018); Ma et al. (2024) and the scientific domain Wadden et al. (2020; 2022); Ho et al. (2025).”

• cites Thorne et al. (2018) sourceSupported
• cites Ma et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wadden et al. (2020; 2022)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Research reports often include a substantial amount of citation-related content, and evaluating the precision and standardization of these citations plays a crucial role in assessing the overall quality of the report.

“Research reports often include a substantial amount of citation-related content, and evaluating the precision and standardization of these citations plays a crucial role in assessing the overall quality of the report Sarol et al. (2024).”

• cites Sarol et al. (2024) sourcePartly supported

  The Introduction says articles cite about 45 sources on average and explains why citation accuracy matters to the integrity of scientific evidence. Section 2.2 assesses whether citations accurately reflect their sources and identifies ambiguous citation style as an error. The paper does not establish that citation standardization is crucial to assessing a report’s overall quality. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

SurveyForge combines human outline structure analysis with high-quality literature retrieval, generating and refining full survey content through a scholar navigation agent.

“More recently, SurveyForge Yan et al. (2025) combines human outline structure analysis with high-quality literature retrieval, generating and refining full survey content through a scholar navigation agent.”

• cites Yan et al. (2025) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Given a report with citation content, tasks such as cited context identification, evidence sentence retrieval, and citation accuracy classification are commonly used to analyze citation quality.

“Given a report with citation content, tasks such as cited context identification, evidence sentence retrieval, and citation accuracy classification are commonly used to analyze citation quality Sarol et al. (2024).”

• cites Sarol et al. (2024) sourcePartly supported

  Section 2.2 (Citation accuracy categories) formulates citation accuracy assessment as the three tasks named in the claim, and the Abstract (Results) describes using them to assess citations. The paper establishes that these tasks are used in its own approach, but does not establish that they are commonly used across citation-quality research. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Other approaches focused on modeling paper relationships via citation networks, including AutoSurvey with a two-stage LLM pipeline and HiReview with a taxonomy-driven framework, though both faced limitations in capturing human writing styles or relying on restricted citation scopes.

“Other approaches focused on modeling paper relationships via citation networks, including AutoSurvey Wang et al. (2024) with a two-stage LLM pipeline and HiReview Hu et al. (2025) with a taxonomy-driven framework, though both faced limitations in capturing human writing styles or relying on restricted citation scopes.”

• cites Wang et al. (2024) sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Hu et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Automated fact checking has evolved into a well-established multistage pipeline, which has become the dominant research paradigm in the field.

“Driven by efforts from both academia and industry, automated fact checking has evolved into a well-established multistage pipeline, which has become the dominant research paradigm in the field Eldifrawi et al. (2024).”

• cites Eldifrawi et al. (2024) sourcePartly supported

  Section 3 describes automated fact-checking as a pipeline with multiple stages, and Section 6.3 classifies existing pipeline architectures. The paper does not establish that this pipeline is the dominant research paradigm or attribute its development to efforts from both academia and industry. The indexed passages provide no page numbers. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

OpenAI Deep Research and o3 exhibit similar retrieval performance, with precision 0.385 vs. 0.299 and recall 0.033 vs. 0.031.

“OpenAI Deep Research and o3 exhibit similar retrieval performance, with precision (0.385 vs. 0.299) and recall (0.033 vs. 0.031) showing only slight differences.”

May combine two claims.

OpenAI Deep Research maintains a high average alignment score (88.2) despite citing fewer sources.

“Despite citing fewer sources, OpenAI maintains a high average alignment score (88.2), suggesting stronger precision in citation usage.”

May combine two claims.

Gemini Deep Research trades off some precision (0.145 vs. 0.269) to achieve much higher recall (0.036 vs. 0.010) and generates far more references per report (32.42 vs. 4.27) than gemini-2.5-pro.

“Gemini Deep Research trades off some precision (0.145 vs. 0.269) to achieve much higher recall (0.036 vs. 0.010) and generates far more references per report (32.42 vs. 4.27).”

May combine two claims.

Two representative failure types were identified: statement hallucination, where content deviates from the cited source, and citation hallucination, where the reference itself is fabricated.

“In our manual inspection of evaluation results, we identified two representative failure types: statement hallucination , where the content deviates from the cited source, and citation hallucination , where the reference itself is fabricated.”

May combine two claims.

Even advanced Deep Research agents remain susceptible to hallucinating author names, misaligning citations, and fabricating links.

“These examples demonstrate that even advanced Deep Research agents remain susceptible to hallucinating author names, misaligning citations, and fabricating links.”

Probably not a substantial claim: may not pass S4 (based).

Most reports generated by Deep Research products use URL links to cite web pages.

“Since most reports generated by the Deep Research products use URL links to cite web pages, we adopt the same citation format throughout our evaluation, including when assessing the base model.”

May combine two claims.

Kulkarni did publish a paper on reinforcement learning and Text-to-SQL, but it was not among the references cited in the generated report.

“In fact, Kulkarni did publish a paper on reinforcement learning and Text-to-SQL, but it was not among the references cited in the generated report.”

May combine two claims.

Deep Research products still face challenges in hallucination and over-citation.

“Through large-scale experiments on leading LLM-based research agents and the base models, we demonstrate that Deep Research products can outperform base models in content overage and factual grounding, but still face challenges in hallucination, over-citation, etc.”

May combine two claims.

gemini-2.5-pro attains higher precision (0.269) at the expense of recall (0.010) and generates fewer reference documents on average (4.27 per report) than claude4-sonnet.

“In contrast, gemini-2.5-pro attains higher precision (0.269) at the expense of recall (0.010) and generates fewer reference documents on average (4.27 per report), limiting its coverage.”

May combine two claims.

The URL cited for the in-vehicle crowd monitoring system does not exist and appears to be entirely fabricated by the model.

“However, the cited URL does not exist and appears to be entirely fabricated by the model.”

Classified as probably not a substantial claim.

ReportBench evaluates generated reports comprehensively along two axes: content quality and statement factuality.

“It evaluates generated reports comprehensively along two axes: content quality and statement factuality.”

May combine two claims.

gemini-2.5-flash underperforms on both precision (0.237) and recall (0.012), with lower citation semantic consistency (44.88%).

“gemini-2.5-flash underperforms on both precision (0.237) and recall (0.012), with lower citation semantic consistency (44.88%), indicating poorer citation relevance.”

May combine two claims.

The benchmark primarily draws from peer-reviewed survey papers on arXiv, most of which are concentrated in STEM fields.

“The benchmark primarily draws from peer-reviewed survey papers on arXiv, most of which are concentrated in STEM fields.”

May combine two claims.

o3 produces the most references (12.26 per report) and moderate recall (0.031), but its citation semantic consistency (31.43%) and non-cited statement accuracy (82.22%) lag behind.

“Meanwhile, o3 produces the most references (12.26 per report) and moderate recall (0.031), but its citation semantic consistency (31.43%) and non-cited statement accuracy (82.22%) lag behind.”

May combine two claims.

Among the four base models, claude4-sonnet demonstrates the most balanced performance, achieving precision 0.337, recall 0.021, average 6.74 reference documents per report, citation semantic consistency 73.67%, and non-cited statement factual accuracy 92.64%.

“Among the four base models, claude4-sonnet demonstrates the most balanced performance-achieving a precision of 0.337, a recall of 0.021, an average of 6.74 reference documents per report, a high citation semantic consistency (73.67%), and a strong non-cited statement factual accuracy (92.64%).”

May combine two claims.

OpenAI Deep Research is not a direct output of o3, but likely incorporates an additional writing module, possibly optimized via fine-tuning or structured pipelines.

“This suggests that Deep Research is not a direct output of o3 , but rather likely incorporates an additional writing module, possibly optimized via fine-tuning or structured pipelines.”

Classified as probably not a substantial claim.

Each generated prompt is required to include a cutoff date corresponding to the most recent update of the paper.

“Besides, to ensure temporal consistency, we require each generated prompt to include a cutoff date corresponding to the most recent update of the paper.”

Classified as probably not a substantial claim.

ReportBench is specifically designed to assess two critical pillars of trustworthy and practical DRA outputs: factual accuracy and citation behavior.

“In contrast, our ReportBench is specifically designed to assess two critical pillars of trustworthy and practical DRA outputs: factual accuracy and citation behavior.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The rise of deep research agents (DRAs), driven by powerful models such as ChatGPT and Gemini, has underscored the urgent need for robust and targeted evaluation methodologies.

“The rise of deep research agents (DRAs), driven by powerful models such as ChatGPT OpenAI (2025) and Gemini Google (2025), has underscored the urgent need for robust and targeted evaluation methodologies.”

• cites OpenAI (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Google (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The pronounced gaps in precision/recall trade-off, citation count, and overall coverage between Gemini Deep Research and gemini-2.5-pro mirror the contrast observed between OpenAI Deep Research and o3, suggesting the system has undergone targeted optimization for thorough research and report generation.

“These pronounced gaps-in precision/recall trade-off, citation count, and overall coverage-mirror the contrast observed between OpenAI Deep Research and o3 , and suggest that the system has undergone targeted optimization for thorough research and report generation.”

May combine two claims.

ReportBench incorporates large language models and search tools to evaluate and verify the completeness of related work coverage, offering a more comprehensive perspective for research report quality assessment.

“To address this gap, our ReportBench incorporates large language models and search tools to evaluate and verify the completeness of related work coverage, offering a more comprehensive perspective for research report quality assessment.”

• AssumptionRests on an assumption: Large language models and search tools can reliably evaluate and verify the completeness of related work coverage.

Classified as probably not a substantial claim.

The prompt is augmented with an explicit instruction stipulating that the model must refrain from citing the original paper corresponding to the given prompt.

“To address this issue, we augment the prompt with an additional explicit instruction, stipulating that the model must refrain from citing the original paper corresponding to the given prompt.”

Classified as probably not a substantial claim.

For Gemini, both the 'Gemini 2.5 Pro' and 'Deep Research' toggles were enabled on the web interface to activate its full research capabilities.

“For Gemini, we made sure that both the 'Gemini 2.5 Pro' and 'Deep Research' toggles were enabled on the web interface to activate its full research capabilities.”

Classified as probably not a substantial claim. May combine two claims.

For fact-checking non-cited statements, two web-connected models (gemini-2.5-pro and gemini-2.5-flash) each perform three independent judgments per statement, totaling six verdicts, with final decision by majority voting and vote proportion recorded as confidence.

“For the fact-checking of non-cited statements, we employ two web-connected models: gemini-2.5-pro and gemini-2.5-flash . Each model performs three independent judgments per statement, resulting in a total of six verdicts. The final decision is determined by majority voting, and the proportion of votes is recorded as a confidence score.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The content quality of research reports hinges on two critical factors: (1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of generated statements.

“Specifically, we assert that the content quality of research reports hinges on two critical factors: (1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of generated statements, whether derived from cited references or produced by the model.”

May combine two claims.

Overall, Deep Research products significantly outperform their base models in coverage and factual grounding, pointing to the value of task-specific model fine-tuning or pipeline design beyond standalone LLM capabilities.

“Overall, Deep Research products significantly outperform their base models in coverage and factual grounding, pointing to the value of task-specific model fine-tuning or pipeline design beyond standalone LLM capabilities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Responses from OpenAI and Gemini web interfaces and base models were manually collected during July 14 to July 25.

“To evaluate the performance of both Deep Research agents and base models, we manually collected responses from the web-based interfaces of OpenAI and Gemini, as well as batchexecuted outputs from the base models, during the period from July 14 to July 25.”

May combine two claims.

These benchmarks focus solely on fact-checking components, rather than evaluating the synthesized information as a whole, limiting their ability to assess recent long-form output from LLMs, such as full research reports.

“However, these benchmarks focus solely on fact-checking components, rather than evaluating the synthesized information as a whole, limiting their ability to assess recent long-form output from large language models (LLMs), such as full research reports.”

Classified as probably not a substantial claim. May combine two claims.

ReportBench introduces a fine-grained validation workflow that separately assesses cited and non-cited statements, combining citation semantic consistency checks and web-based factual verification.

“Our framework introduces a fine-grained validation workflow that separately assesses cited and non-cited statements, combining citation semantic consistency checks and web-based factual verification.”

• AssumptionRests on an assumption: Web-based factual verification can determine the factual correctness of statements in generated reports.

Classified as probably not a substantial claim. May combine two claims.

Because the cited URL cannot be resolved, no supporting text or evidence can be retrieved to validate the statement, resulting in a citation mismatch.

“Because the link cannot be resolved, no supporting text or evidence can be retrieved to validate the statement, resulting in a citation mismatch.”

Classified as probably not a substantial claim.

ReportBench is a comprehensive benchmark for evaluating the quality of references and the factual accuracy of all statements in reports generated by Deep Research agents.

“In this paper, we present ReportBench , a comprehensive benchmark for evaluating the quality of references and the factual accuracy of all statements in reports generated by Deep Research agents.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Subsequent research explored structured and hierarchical organization, such as hierarchical catalogue generation with semantic and structural metrics, though these remained limited to outline generation with fixed references.

“Subsequent research explored structured and hierarchical organization, such as hierarchical catalogue generation with semantic and structural metrics, though these remained limited to outline generation with fixed references.”

Classified as probably not a substantial claim.

During data collection, OpenAI was using the standard version of Deep Research, powered by the o3 model.

“During data collection, we ensured that OpenAI was using the standard version of Deep Research, powered by the o3 model.”

Probably not a substantial claim: may not pass S4 (based).

Existing evaluation benchmarks focus primarily on individual aspects such as fact checking, citation judgment, or overall writing quality.

“Existing evaluation benchmarks focus primarily on individual aspects such as fact checking, citation judgment, or overall writing quality.”

Probably not a substantial claim: may not pass S4 (based).

Compared with SurveyBench, ReportBench focuses solely on well-defined and automatically verifiable dimensions of evaluation-namely, factual faithfulness and correctness.

“Compared with SurveyBench, ReportBench focuses solely on well-defined and automatically verifiable dimensions of evaluation-namely, factual faithfulness and correctness.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

While citation correctness and existence have been well-studied, the aspect of citation completeness-whether all relevant prior work on a given research topic has been cited-remains underexplored.

“While citation correctness and existence have been well-studied, the aspect of citation completeness-i.e., whether all relevant prior work on a given research topic has been cited-remains underexplored.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The quality of the classification of the 100-paper subset was reviewed and validated by four research experts.

“The quality of the classification of this subset was then reviewed and validated by four research experts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Through an automated construction pipeline, ReportBench ensures data quality while offering clear scalability advantages.

“In addition, through an automated construction pipeline, it ensures data quality while offering clear scalability advantages, enabling”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Published survey papers are typically written by domain experts and have undergone a peer review process that provides additional expert-level validation, considered among the highest-quality research reports currently available.

“Published survey papers are typically written by domain experts and have undergone a peer review process that provides additional expert-level validation, considered among the highest-quality research reports currently available.”