CoChat
Sign Up
PreprintCited by 4Topic ModelingInformation Retrieval and Search BehaviorRecommender Systems and Techniques

DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis

Open access
Patel, Liana, Arabzadeh, Negar, Gupta, Harshit, Sundar, Ankita, Stoica, Ion, Zaharia, Matei, Guestrin, Carlos·2025·arXiv
DOI 10.48550/arxiv.2508.20033arXiv 2508.20033
Open this paper in CoChat — free accountOpen PDF

Abstract

The ability to research and synthesize knowledge is central to human expertise and progress. A new class of AI systems--designed for generative research synthesis--aims to automate this process by retrieving information from the live web and producing long-form, cited reports. Yet, evaluating such systems remains an open challenge: existing question-answering benchmarks focus on short, factual answers, while expert-curated datasets risk staleness and data contamination. Neither captures the complexity and evolving nature of real research synthesis tasks. We introduce DeepScholar-bench, a live benchmark and automated evaluation framework for generative research synthesis. DeepScholar-bench draws queries and human-written exemplars from recent, high-quality ArXiv papers and evaluates a real synthesis task: generating a related work section by retrieving, synthesizing, and citing prior work. Our automated framework holistically measures performance across three key dimensions--knowledge synthesis, retrieval quality, and verifiability. To further future work, we also contribute DeepScholar-ref, a simple, open-source reference pipeline, which is implemented on the LOTUS framework and provides a strong baseline. Using DeepScholar-bench, we systematically evaluate prior open-source systems, search agents with strong models, OpenAI's DeepResearch, and DeepScholar-ref. We find DeepScholar-bench is far from saturated: no system surpasses a geometric mean of $31\%$ across all metrics. These results highlight both the difficulty and importance of DeepScholar-bench as a foundation for advancing AI systems capable of generative research synthesis. We make our benchmark code and data available at https://github.com/guestrin-lab/deepscholar-bench.

AI snapshot

Written from the paper’s full text.

TL;DR

DeepScholar-bench is a live benchmark and automated evaluation framework for generative research synthesis, using recent ArXiv papers to task systems with generating related-work sections and scoring knowledge synthesis, retrieval quality, and verifiability. Across 14 baselines, no system surpasses a 31% geometric mean across all metrics, though DeepScholar-ref is a strong open-source baseline.

Key findings

  • No system surpasses a geometric mean of 31% across all metrics; OpenAI DeepResearch obtains the highest geometric mean at .309, with scores of .392 on Nugget Coverage, .187 on Reference Coverage, and .124 on Document Importance. On Nugget Coverage, Reference Coverage, and Document Importance, every baseline remains below .40.
  • DeepScholar-ref provides a strong baseline: DeepScholar-ref (GPT-4.1, o3) achieves similar or higher Organization, Nugget Coverage, Relevance Rate, Reference Coverage, Citation Precision, and Claim Coverage compared with OpenAI DeepResearch, and DeepScholar-ref achieves up to 6.3× higher verifiability; DeepScholar-ref (GPT-4.1, o3) is also 4.3× cheaper and 2.28× faster than OpenAI DeepResearch.
  • The DeepScholar-June-2025 dataset contains 63 ArXiv papers across 18 distinct ArXiv domains; human-written related-work exemplars average 23 unique references, and over 63% of all cited references are on ArXiv. A later DeepScholar-Nov-2025 slice contains 200 queries from over 75 distinct arXiv subject areas.
  • Human evaluation with 11 Computer Science PhD annotators found 71.43% agreement for Organization pairwise comparisons, 83.33% for nugget labeling, and 65.9% for reference-importance labeling; manual validation of LLM-based evaluation showed above 70% agreement across each metric.

Methods

  • Task and data: given a paper’s description (the abstract), systems retrieve sources and generate a related-work section. The automated pipeline scrapes recent v1 ArXiv papers accepted at conferences, excludes papers without an explicit Related Work section or .bib file, and excludes related-work sections longer than 1,000 words.
  • Evaluation: seven automated metrics across three dimensions—knowledge synthesis (Organization, Nugget Coverage), retrieval quality (Relevance Rate, Reference Coverage, Document Importance), and verifiability (Citation Precision, Claim Coverage)—using LLM-as-a-judge with GPT-4.1 and GPT-4o judges, plus human validation.
  • Baselines: open-source systems DeepResearcher, STORM, and OpenScholar; search agents with Llama-4-Scout-17B-16E-Instruct, GPT-4.1, o3, Claude-opus-4, and Gemini-2.5-pro; OpenAI DeepResearch; and DeepScholar-ref. Retrieval is controlled by allowing systems to access the Web only through the ArXiv API, and search results published after the query paper’s publication date are filtered out.

Limitations

  • The paper states that its automated verifiability metrics under-estimate the actual verifiability of human writing; Reference Coverage is conservative because the LLM under-labels essential references; and results show sensitivity to query formulation for Document Importance and Citation Precision, and to judge choice for Nugget Coverage, Relevance Rate, and Citation Precision.
  • The paper states that DeepScholar-bench remains far from saturated, with no system surpassing a 31% geometric mean, and that even oracle retrievers still far from saturate Nugget Coverage.
  • Evident from the text: the main DeepScholar-June-2025 slice contains only 63 Computer Science papers; retrieval is restricted to the ArXiv API rather than the full live web; human validation uses 11 CS PhD annotators and roughly 300 annotations; and the expanded DeepScholar-Nov-2025 slice evaluates only a subset of systems.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.20033

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

25 of 30 cited sources checked · 15 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 2

    Expert-curated datasets with open-ended research questions and exemplar answers quickly become stale and outdated as new information emerges.

    “To address this limitation, several recent works instead leverage expert-curated datasets with open-ended research questions and exemplar answers [7, 65, 62, 60, 9, 45, 17]. Unfortunately, these benchmarks quickly become stale and outdated as new information emerges.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [7] [7] Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs, November 2024. URL http://arxiv.org/abs/2411.14199 . arXiv:2411.14199 [cs]. -
    • cites [65] [65] Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, Yang Xu, Qingkai Min, Zizhao Zhang, Yiwen Wang, Wenjie Li, and Pengfei Liu. OpenResearcher: Unleashing AI for Accelerated Scientific Research. In Delia Irazu Hernandez Farias, Tom Hope, and Manling Li, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 209-218, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-demo.22. URL https: //aclantho source

      Section 5.1 (page unavailable) says the paper collected 109 research questions from graduate students, supporting its use as an example of a research-question evaluation set. It explicitly says the authors used pairwise comparisons instead of annotated ground-truth answers, so it does not support the exemplar-answers detail. The cited passage also does not say that such benchmarks quickly become stale. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [62] [62] you.com. Su-Sea/ydc-deep-research-evals: you.com's framework for evaluating deep research systems., 2025. URL https://github.com/Su-Sea/ydc-deep-research-evals . -

      Not checked in the free run.

  • Interpretationcitation
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    The strong relative performance of DeepScholar-ref likely reflects the efficiency of the data-processing semantic operators [37] that DeepScholar-ref uses to perform LLM-based filtering, ranking and summarization of sources to generate its report.

    “Overall, the strong relative performance of DeepScholar-ref likely reflects the efficiency of the dataprocessing semantic operators [37] that DeepScholar-ref uses to perform LLM-based filtering, ranking and summarization of sources to generate its report.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [37] [37] Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. Semantic Operators: A Declarative Model for Rich, AI-based Data Processing, March 2025. URL http://arxiv.org/abs/2407.11418 . arXiv:2407.11418 [cs]. - source
  • Premisecitation
    § 6 Related Work · p. 10

    Several prior works provide expert-curated datasets for long-form research synthesis tasks, including ScholarQABench [7], OpenResearcher [65], DeepConsult [62], ResearcherBench [60], DeepResearch Bench [9], Deep Research Bench [11], SurGE [45], and LiveDRBench [17].

    “While our work proposes a continually-updated, live benchmark using an automated data pipeline, several prior works instead provide expert-curated datasets for long-form research synthesis tasks, including ScholarQABench [7], OpenResearcher [65], DeepConsult [62], ResearcherBench [60], DeepResearch Bench [9], Deep Research Bench [11], SurGE [45], and LiveDRBench [17].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [7] [7] Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs, November 2024. URL http://arxiv.org/abs/2411.14199 . arXiv:2411.14199 [cs]. -
    • cites [65] [65] Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang, Yun Luo, Renjie Pan, Yang Xu, Qingkai Min, Zizhao Zhang, Yiwen Wang, Wenjie Li, and Pengfei Liu. OpenResearcher: Unleashing AI for Accelerated Scientific Research. In Delia Irazu Hernandez Farias, Tom Hope, and Manling Li, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 209-218, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-demo.22. URL https: //aclantho source

      Section 5.1 (Evaluation Data; page unavailable) reports 109 research questions collected from graduate students, with answers that are often complex and lengthy. It does not describe an expert-curated dataset for long-form research synthesis; the authors say they used pairwise comparisons rather than providing annotated ground-truth answers. Section 3 describes OpenResearcher primarily as an AI research assistant. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [62] [62] you.com. Su-Sea/ydc-deep-research-evals: you.com's framework for evaluating deep research systems., 2025. URL https://github.com/Su-Sea/ydc-deep-research-evals . -

      Not checked in the free run.

  • Premisecitation
    § 6 Related Work · p. 10

    Several benchmarks develop automated dataset curation pipelines for live benchmarks, but their task focuses on short-form question-answering, as opposed to long-form report generation [36, 29, 18].

    “Additionally, several benchmark develop automated dataset curation pipelines for live benchmark; however, their task focuses on short-form question-answering, as opposed to long-form report generation [36, 29, 18].”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [36] [36] Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, and Qi Liu. HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation, July 2025. URL http://arxiv.org/abs/2503.04800 . arXiv:2503.04800 [cs]. - source

      The Introduction (p. 2) describes an automated pipeline for a continuously evolving QA dataset, rather than a report-generation benchmark. However, Appendix D.1 (p. 16) says HOH evaluates both short and long answers, so describing its task as exclusively short-form QA overstates the source. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [29] [29] Jannat Ara Meem, Muhammad Shihab Rashid, Yue Dong, and Vagelis Hristidis. PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering, June 2024. URL http://arxiv.org/abs/2402.11034 . arXiv:2402.11034 [cs]. - source
    • cites [18] [18] Mohan Jiang, Jin Gao, Jiahao Zhan, and Dequan Wang. MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding, August 2025. URL http://arxiv.org/abs/2508.15802 . arXiv:2508.15802 [cs]. - source
  • Premisecitation
    § 6 Related Work · p. 10

    Several recent benchmarks, including AcademicEval [63], LongBench-Cite [64] and SciIG [13], evaluate long-form generation tasks that do not require search over the live web.

    “Alternatively, several recent benchmarks, including AcademicEval [63], LongBench-Cite [64] and SciIG [13], evaluate long-form generation tasks that do not require search over the live web , which is a key component of generative research synthesis and our benchmark.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [63] [63] Haozhen Zhang, Tao Feng, Pengrui Han, and Jiaxuan You. AcademicEval: Live Long-Context LLM Benchmark. October 2024. URL https://openreview.net/forum?id=iRYExPKnxm . - source
    • cites [64] [64] Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA, September 2024. URL http://arxiv.org/abs/2409.02897 . arXiv:2409.02897 [cs]. - source
    • cites [13] [13] Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay, and Cornelia Caragea. Let's Use ChatGPT To Write Our Paper! Benchmarking LLMs To Write the Introduction of a Research Paper, August 2025. URL http://arxiv.org/abs/2508.14273 . arXiv:2508.14273 [cs]. - source
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 26
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation
    § 1 Introduction · p. 2

    Existing question-answering benchmarks focus on short-form, easily-verifiable answers and do not reflect realistic research synthesis tasks.

    “Many prior works evaluate generative research synthesis systems using existing question answering benchmarks, which do not reflect realistic research synthesis tasks and instead focus on questions with short-form, easily-verifiable answers, making them severely limited for this setting [56, 55, 31, 21, 58, 53, 19, 61, 20, 22, 16, 49, 24].”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [56] [56] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents, April 2025. URL http://arxiv.org/abs/2504.12516 . arXiv:2504.12516 [cs]. - source

      In the section describing BrowseComp’s design (page unavailable), the paper says its questions and answers are short and easy to verify, and that the benchmark does not measure long responses or resolving ambiguous queries. This supports BrowseComp as an example of a benchmark limited for research-synthesis evaluation. The paper does not establish the sentence’s broader claim that many prior works evaluate generative research synthesis systems using such benchmarks. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [55] [55] Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, November 2024. URL http://arxiv.org/abs/2411.04368 . arXiv:2411.04368 [cs]. - source
    • cites [31] [31] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. URL http://arxiv.org/abs/2311. 12983 . arXiv:2311.12983 [cs]. - source

      Section 3.1 describes GAIA as a question-answering benchmark whose questions have a short, single correct answer that is easy to verify. The paper does not establish that question-answering benchmarks generally fail to reflect realistic research synthesis tasks, or that prior work uses them to evaluate research synthesis systems. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Study design and conductcitation
    § 1 Introduction · p. 2

    The automated LLM-based evaluation metrics show strong agreement with human judgments and serve as a strong surrogate for human preferences.

    “using metrics that show strong agreement with human judgments.”
    • Also stated in: § 3.1 Knowledge Synthesis · p. 5 · boosted
      “This model-based evaluation provides scalability while also serving as a strong surrogate for human preferences [42, 26, 25, 5], which we validate in our experiments in Appendix A.3.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [42] [42] Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. LLMJudge: LLMs for Relevance Judgments, August 2024. URL http://arxiv.org/abs/2408.08896 . arXiv:2408.08896 [cs]. - source

      The Submissions and Results section (page unavailable) reports that LLM-generated judgments generally agree with human judgments on search-system rankings. It also says their exact relevance labels are less consistent, so the source does not establish strong agreement across the automated evaluations or show that they are a strong surrogate for human preferences. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [26] [26] Ruosen Li, Teerth Patel, and Xinya Du. PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations, December 2024. URL http://arxiv.org/abs/2307.02762 . arXiv:2307.02762 [cs]. - source

      Section 4.2 reports that the paper’s peer-ranking method agreed better with human judgments than the individual LLM reviewers. But it describes example-level agreement as only “fair to moderate” (Fleiss’ κ around 0.40), with its best accuracy at 67.3%. That supports agreement with human judgments, but not the unqualified description of the metrics as showing strong agreement or serving as a strong surrogate for human preferences. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [25] [25] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge, February 2025. URL http: //arxiv.org/abs/2411.16594 . arXiv:2411.16594 [cs]. - source

      The Logic section reports that, in a self-driving benchmark, LLM-based evaluations aligned better with human preferences than LVLM-based evaluations. Section 6.1 describes measuring agreement between LLM predictions and human judgments, but neither passage establishes strong agreement across LLM-based evaluation metrics or that they are a strong surrogate for human preferences. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 3.1 Knowledge Synthesis · p. 5

    Permuting each evaluated pair of reports avoids position bias.

    “We permute each evaluated pair of reports to avoid position bias [25], and we report the win-rate of each baseline.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] [25] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge, February 2025. URL http: //arxiv.org/abs/2411.16594 . arXiv:2411.16594 [cs]. - source

  • May combine two claims.

    Findingown data
    § 5.3 Human Evaluation · p. 9

    The agreement score is 71.43% for pairwise comparisons judging Organization, 83.33% for nugget labeling to compute Nugget Coverage, and 65.9% for labeling reference importance to compute Reference Coverage.

    “Specifically, we observe a 71 . 43% agreement score for pairwise comparisons judging Organization, a 83 . 33% agreement score for nugget labeling to compute Nugget Coverage, and a 65 . 9% agreement score for labeling reference importance to compute Reference Coverage.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The majority vote of the 11 human annotators is a reliable ground truth for validating LLM judges.

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 8

    The Claude search agent offers the highest Citation Precision (score .701) and Claim Coverage (score .760).

    “The Claude search agent offers the highest Citation Precision, a score of . 701 and Claim Coverage, a score of . 760 .”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    Averaged across the 5 baselines, using the same primary model, DeepScholar-ref increases Organization by 1.18×, Nugget Coverage by 1.17×, the Relevance Rate by 1.06×, Reference Coverage by 2.03×, Citation Precision by 1.83× and Claim Coverage by 1.86×.

    “Next, we compare the performance of DeepScholar-ref and search agents, finding DeepScholar-ref offers competitive and often stronger performance-specifically, averaged across the 5 baselines, using the same primary model, DeepScholar-ref increases Organization by 1 . 18 × , Nugget Coverage by 1 . 17 × , the Relevance Rate by 1 . 06 × , Reference Coverage by 2 . 03 × , Citation Precision by 1 . 83”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2 Dataset Description and Statistics · p. 4

    Over 63% of all cited references in the dataset are on ArXiv.

    “We analyze the human-written exemplars from our dataset, and we find that, on average, each related work section contains 23 unique references, and we find over 63% of all cited references on ArXiv.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2 Dataset Description and Statistics · p. 4

    The DeepScholar-June-2025 dataset scrapes papers from 18 distinct ArXiv domains.

    “This instantiation scrapes papers from a diverse set of 18 distinct ArXiv domains-including, cs.IR, cs.CV, cs.AI, cs.CL, cs.LG, cs.DC, cs.DB, cs.AR, cs.SD, cs.CR, cs.ET, cs.GR, cs.PL, cs.SY, cs.OS, cs.PF, cs.SE, cs.MM-and selects papers marked as accepted at a conference.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 7

    No system surpasses a geometric mean of .31 across all metrics on DeepScholar-June-2025, with OpenAI DeepResearch obtaining the highest geometric mean.

    “From Table 2, we see that no system surpasses a geometric mean of .31 across all metrics, with OpenAI DeepResearch obtaining the highest geometric mean.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    DeepScholar-ref achieves up to 6.3× higher Verifiability scores but its Document Importance remains relatively low compared to OpenAI's DeepResearch.

    “Notably, DeepScholar-ref achieves up to 6 . 3 × higher Verifiability scores but its Document Importance remain relatively low compared to OpenAI's DeepResearch.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Understanding Opportunities for Improvement. · p. 9

    DeepScholar-ref (GPT-4.1, Claude) with either oracle retriever nearly saturates performance on Retrieval Quality and Verifiability metrics, whereas the same method using arxiv.org, parallel.ai or taviliy.com retrievers scores much lower.

    “First, we see that DeepScholar-ref (GPT-4.1, Claude) with either oracle retriever nearly saturates performance on Retrieval Quality and Verifiability metrics, whereas the same method, using the arxiv.org, parallel.ai or taviliy.com retrievers, score much lower.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The set of important references cited in exemplars is a valid gold standard for evaluating Reference Coverage.

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 7

    OpenAI DeepResearch offers the best performance compared to all other prior methods on Organization (score .857) and Nugget Coverage (score .392).

    “On Knowledge Synthesis, we see that OpenAI DeepResearch offers the best performance compared to all other prior methods on both Organization, with a score of . 857 , and Nugget Coverage, with a score of . 392 .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2 Dataset Description and Statistics · p. 4

    On average, each related work section in the dataset contains 23 unique references.

    “We analyze the human-written exemplars from our dataset, and we find that, on average, each related work section contains 23 unique references, and we find over 63% of all cited references on ArXiv.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    In comparison to OpenAI DeepResearch, DeepScholar-ref (GPT-4.1, o3) achieves a similar or higher Organization, Nugget Coverage, Relevance Rate, Reference Coverage, Citation Precision and Claim Coverage score.

    “In comparison to OpenAI DeepResearch, DeepScholar-ref (GPT-4.1, o3) achieves a similar or higher Organization, Nugget Coverage, Relevance Rate, Reference Coverage, Citation Precision and Claim Coverage score.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    Compared to the best-performing prior open-source methods for each metric, DeepScholar-ref offers competitive Knowledge Synthesis performance, 1.09× higher Relevance Rates, 2.18× higher Reference Coverage, 2.08× higher Citation Precision and 1.41× higher Claim Coverage.

    “Compared to the best-performing prior open-source methods for each metric, DeepScholar-ref offers competitive Knowledge Synthesis performance, 1 . 09 × higher Relevance Rates, 2 . 18 × higher Reference Coverage, 2 . 08 × higher Citation Precision and 1 . 41 × higher Claim Coverage.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Human Evaluation · p. 10

    For Reference Importance, the overall agreement score is 65.9% and the false negative rate is 9.8%.

    “Finally, for Reference Importance we observe an overall agreement score of 65 . 9% , and importantly false negative rate of 9 . 8% , i.e., when the LLM incorrectly labels a reference as important.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The majority vote of the 11 human annotators is a reliable ground truth for validating LLM judges.

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 8

    OpenAI DeepResearch's Relevance Rate (.629) exceeds that of the human exemplars, while its Reference Coverage (.187) and Document Importance (.124) scores remain exceedingly low.

    “While it's Relevance Rate shows strong performance, exceeding that of the human exemplars with a score of . 629 , it's Reference Coverage and Document Importance scores remain exceedingly low: . 187 and . 124 respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human-written exemplars are a valid reference standard for evaluating generative research synthesis systems.

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 8

    OpenAI's DeepResearch and all other prior methods are unable to achieve a Citation Precision score beyond .50 and a Claim Coverage score beyond .60.

    “Meanwhile, OpenAI's DeepResearch as well as the all other prior methods are unable to achieve a Citation Precision score beyond . 50 and a Claim Coverage score beyond . 60 .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 8

    OpenAI DeepResearch offers the strongest performance among the other benchmarked prior methods on Relevance Rate, Reference Coverage and Document Importance, but still far from saturates performance.

    “Once again, OpenAI DeepResearch offers the strongest performance among the other benchmarked prior methods on Relevance Rate, Reference Coverage and Document Importance, but still far from saturates performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement. · p. 8

    OpenAI DeepResearch is outperformed on both Citation Precision and Claim Coverage by the search agents with GPT4.1, o3, Claude and Gemini models.

    “We see that OpenAI DeepResearch is outperformed on both Citation Precision and Claim Coverage by the search agents with GPT4.1, o3, Claude and Gemini models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    DeepScholar-ref (GPT-4.1, o3) offers an efficient reference pipeline that is 4.3× cheaper and 2.28× faster than OpenAI DeepResearch.

    “In Appendix Table 6, we provide additional cost analysis, finding that DeepScholar-ref (GPT-4.1, o3) offers an efficient reference pipeline that is 4 . 3 × cheaper and 2 . 28 × faster than OpenAI DeepResearch.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Human Evaluation · p. 10

    The false positive and false negative rates of the LLM-judge for Nugget Importance are similar and both less than 10%.

    “Moreover we see that the false postive and false negative rate of the LLMjudge are similar, and both rather small, less than 10% , once again indicating that severe LLM mis-labeling is rather rare.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    DeepScholar-ref (GPT-4.1) has an average of 20 unique references and 89 inline citations.

    “DeepScholar-ref (GPT-4.1) | 7863 | 735 | 115 | 20 | 89 | 234 | 1.66”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    The Pearson correlation between system-level scores under abstract and Key Idea query formulations is 0.997 for Organization, 0.980 for Nugget Coverage, 0.979 for Relevance Rate, 0.992 for Reference Coverage, 0.992 for Document Importance, 0.589 for Citation Precision, and 0.841 for Claim Coverage.

    “Abstract vs. Key Idea | 0.997 ∗ | 0.980 ∗ | 0.979 ∗ | 0.992 ∗ | 0.992 | 0.589 | 0.841 ∗”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.4.3 Document Importance Across Human Exemplars · p. 23

    Outliers inflate the mean citation values, resulting in relatively high averages compared to typical references (478.3 citations across all references and 647.6 for ArXiv-only references).

    “These outliers inflate the mean citation values, resulting in relatively high averages compared to typical references (478.3 citations across all references and 647.6 for ArXiv-only references).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.3.6 Agreement Between Different LLM Judges · p. 20

    Most metrics remain significantly correlated, but the main sensitivity to choice of judge is on nugget coverage, which is non-significant for both GPT-4o vs. Llama-4-17b-16e-Instruct and DeepSeek-R1-Distill-Qwen-32B vs. Llama-4-17b-16e-Instruct.

    “Most metrics also remain significantly correlated, but we observe the main sensitivity to the choice of judge on nugget coverage (non-significant for both GPT-4o vs. Llama-4-17b-16e-Instruct and DeepSeek-R1-Distill-Qwen-32B vs. Llama-4-17b-16e-Instruct).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingreasoning
    § Background information · p. 26

    The ReAct-based agent generates shorter, more effective searchable queries.

    “which generates shorter, more effective searchable queries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 30

    The Pearson correlation between system-level scores produced by GPT-4o and DeepSeek judges is 0.966 for Organization, 0.920 for Nugget Coverage, 0.980 for Relevance Rate, 0.990 for Reference Coverage, 0.843 for Citation Precision, and 0.996 for Claim Coverage.

    “GPT-4o vs. DeepSeek | 0.966 ∗ | 0.920 ∗ | 0.980 ∗ | 0.990 ∗ | 0.843 ∗ | 0.996 ∗”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    On DeepScholar-Nov-2025, DeepScholar-ref (GPT-4.1 + o3) achieves a Claim Coverage of 0.578.

    “DeepScholar-ref (GPT-4.1 + o3) | 0.578 | 0.479 | 0.568 | 0.087 | 0.054 | 0.563 | 0.578”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 30

    The Pearson correlation between system-level scores produced by DeepSeek and Llama-4 judges is 0.991 for Organization, 0.668 for Nugget Coverage, 0.942 for Relevance Rate, 0.994 for Reference Coverage, 0.986 for Citation Precision, and 0.963 for Claim Coverage.

    “DeepSeek vs. Llama-4 | 0.991 ∗ | 0.668 | 0.942 ∗ | 0.994 ∗ | 0.986 | 0.963 ∗”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    The Pearson correlation between system-level scores under abstract and RQ query formulations is 0.988 for Organization, 0.979 for Nugget Coverage, 0.772 for Relevance Rate, 0.951 for Reference Coverage, 0.707 for Document Importance, 0.766 for Citation Precision, and 0.877 for Claim Coverage.

    “Abstract vs. RQ | 0.988 ∗ | 0.979 ∗ | 0.772 ∗ | 0.951 ∗ | 0.707 | 0.766 ∗ | 0.877 ∗”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    The Pearson correlation between system-level scores under Key Idea and RQ query formulations is 0.991 for Organization, 0.983 for Nugget Coverage, 0.836 for Relevance Rate, 0.958 for Reference Coverage, 0.766 for Document Importance, 0.942 for Citation Precision, and 0.981 for Claim Coverage.

    “Key Idea vs. RQ | 0.991 ∗ | 0.983 ∗ | 0.836 ∗ | 0.958 ∗ | 0.766 ∗ | 0.942 ∗ | 0.981 ∗”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    Human-written exemplars have an average report length of 4381 characters, 497 words, and 28 sentences.

    “Human-written Exemplars | 4381 | 497 | 28 | 23 | 27 | N/A | N/A”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    On DeepScholar-Nov-2025, DeepScholar-ref (GPT-4.1 + o3) achieves a higher Claim Coverage (0.578) than Search Agent (o3) (0.452).

    “DeepScholar-ref (GPT-4.1 + o3) | 0.578 | 0.479 | 0.568 | 0.087 | 0.054 | 0.563 | 0.578”
    • Also stated in: § A.7 Examples of Generated Reports · p. 29 · plain
      “Search Agent (o3) | 0.608 | 0.480 | 0.623 | 0.078 | 0.763 | 0.475 | 0.452”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 30

    The Pearson correlation between system-level scores produced by GPT-4o and Llama-4 judges is 0.985 for Organization, 0.413 for Nugget Coverage, 0.941 for Relevance Rate, 0.975 for Reference Coverage, 0.817 for Citation Precision, and 0.958 for Claim Coverage.

    “GPT-4o vs. Llama-4 | 0.985 ∗ | 0.413 | 0.941 | 0.975 ∗ | 0.817 ∗ | 0.958 ∗”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    Human-written exemplars have an average of 23 unique references and 27 inline citations.

    “Human-written Exemplars | 4381 | 497 | 28 | 23 | 27 | N/A | N/A”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    The human-agreement score with LLM for Reference Coverage is 82%.

    “Reference Coverage | Reference Importance (Not Imp./ Imp.) | 82%”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    On DeepScholar-Nov-2025, Search Agent (o3) achieves a higher Document Importance (0.763) than DeepScholar-ref (GPT-4.1 + o3) (0.054).

    “DeepScholar-ref (GPT-4.1 + o3) | 0.578 | 0.479 | 0.568 | 0.087 | 0.054 | 0.563 | 0.578”
    • Also stated in: § A.7 Examples of Generated Reports · p. 29 · plain
      “Search Agent (o3) | 0.608 | 0.480 | 0.623 | 0.078 | 0.763 | 0.475 | 0.452”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    The median number of citations per reference from ArXiv.org for human-written exemplars is 647.5.

    “Median number of citations per reference from ArXiv.org | 647.5”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.4.3 Document Importance Across Human Exemplars · p. 23

    Median values are lower (31 for all references and 36 for ArXiv-only).

    “In contrast, the median values are lower (31 for all references and 36 for ArXiv-only).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § Background information · p. 26

    The regular search agent fails when tasked with full abstract queries.

    “The regular search agent fails when tasked with full abstract queries; hence the ReAct-based agent was employed, which generates shorter, more effective searchable queries.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    On DeepScholar-Nov-2025, Search Agent (o3) achieves a Claim Coverage of 0.452.

    “Search Agent (o3) | 0.608 | 0.480 | 0.623 | 0.078 | 0.763 | 0.475 | 0.452”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    Human-written exemplars have an average of 11.47 important references from ArXiv.org.

    “# Important References from ArXiv.org | 11.47”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A.7 Examples of Generated Reports · p. 29

    DeepScholar-ref (GPT-4.1) has an average report length of 7863 characters, 735 words, and 115 sentences.

    “DeepScholar-ref (GPT-4.1) | 7863 | 735 | 115 | 20 | 89 | 234 | 1.66”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Box 3: Prompt for Attribution Validation · p. 25

    Most rollouts converge in fewer than three steps.

    “this limit is generous as most rollouts converge in fewer than three steps”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5.3 Human Evaluation · p. 10

    The larger off-diagonal mass (24.2%) reflects under-labeling of essential references by the LLM, suggesting that Reference Coverage scores provide a rather conservative metric, measuring recall of only a subset of all truly important references for each query.

    “The larger off-diagonal mass (24.2%) reflects under-labeling of essential references by the LLM, suggesting that our Reference Coverage scores provide a rather conservative metric, measuring”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The majority vote of the 11 human annotators is a reliable ground truth for validating LLM judges.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 6 Related Work · p. 10

    Expert-curated benchmarks are expensive to construct and update, can quickly become outdated as new information becomes available, and risk data contamination as new models are trained on publicly available data.

    “Unfortunately, these expert-curated benchmarks, are expensive to construct and update, can quickly become outdated, as new information becomes available, and risk data contamination, as new models are trained on publicly available data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 2.1 Automated Data Collection Framework · p. 4

    The data pipeline filters papers to keep only those listed as accepted or published at a conference to control for paper quality.

    “To control for paper quality, our pipeline then optionally filters papers to keep only those listed as 'accepted' or 'published' at a conference based on the ArXiv metadata.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § 5.3 Human Evaluation · p. 9

    The human evaluation used 11 annotators, all Computer Science PhD students from four research universities across North America.

    “To assess whether LLM-based metrics reflect human expert judgments, we conduct a human evaluation with 11 annotators, all Computer Science PhD students from four research universities across North America.”
    • Also stated in: § A.3.4 Human Evaluation Details · p. 19 · plain
      “To assess whether LLM-based metrics reflect human expert judgments, we conduct a human evaluation with 11 annotators, all Computer Science PhD students from four research universities across North America.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.1.2 DeepScholar-ref Provides a Strong Baseline for Generative Research Synthesis. · p. 8

    DeepScholar-ref still demonstrates significant room for improvement and is far from saturating DeepScholar-Bench, especially on key Knowledge Synthesis and Retrieval Quality metrics.

    “Notably, DeepScholar-ref still demonstrates significant room for improvement and far from saturates DeepScholar-Bench, especially on key Knowledge Synthesis and Retrieval Quality metrics.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 7 Conclusion · p. 10

    By automatically sourcing queries from high-quality, recent ArXiv papers, DeepScholar-bench mitigates the risks of data staleness and training contamination, while offering a real research synthesis task.

    “By automatically sourcing queries from high-quality, recent ArXiv papers, our benchmark mitigates the risks of data staleness and training contamination, while offering a real research synthesis task.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Recent ArXiv papers are high-quality and provide a real research synthesis task.

  • May combine two claims.

    Interpretationown data
    § 5.2 Understanding Opportunities for Improvement. · p. 9

    Key opportunities for improvement lie in both retrieval and knowledge synthesis capabilities.

    “Overall, the results demonstrate that key opportunities for improvement lie in both retrieval and knowledge synthesis capabilities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 7 Conclusion · p. 10

    The systematic evaluation of prior open-source systems, search agents, OpenAI's DeepResearch and DeepScholar-ref demonstrates significant opportunities for future work, with no system surpassing a geometric mean of 31% across all metrics.

    “Overall our systematic evaluation of prior open-source systems, search agents, OpenAI's DeepResearch and DeepScholar-ref demonstrates significant opportunities for future work, with no system surpassing a geometric mean of 31% across all metrics.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 2.1 Automated Data Collection Framework · p. 4

    The data pipeline keeps only v1 ArXiv papers to avoid possible data contamination arising from multiple ArXiv versions.

    “To avoid possible data contamination arising from multiple ArXiv versions, we keep only v1 ArXiv papers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data
    § 7 Conclusion · p. 10

    These results demonstrate both the difficulty of DeepScholar-bench and the opportunity for further advancement in this space.

    “These results demonstrate both the difficulty of DeepScholar-bench and the exciting opportunity for further advancement in this space.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § A.3.6 Agreement Between Different LLM Judges · p. 20

    To test robustness to the choice of LLM judge, all experiments were repeated with three different judges: GPT-4o, Llama-4-17b-16e-Instruct, and DeepSeek-R1-Distill-Qwen-32B.

    “To test the robustness of our evaluation to the choice of LLM judge, we repeat all experiments with three different judges: GPT-4o, Llama-4-17b-16e-Instruct, and DeepSeek-R1-Distill-Qwen-32B.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given
    § A.4.3 Document Importance Across Human Exemplars · p. 23

    Citation counts in OpenAlex may not exactly match those from other sources such as Google Scholar, but relative counts are consistent.

    “While citation counts in OpenAlex may not exactly match those from other sources such as Google Scholar, the relative counts are consistent, making it a reliable open-source alternative.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] [35] OpenAlex. OpenAlex: The open catalog to the global research system | OpenAlex, 2025. URL https: //openalex.org/ . -

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § A.3.2 Results on DeepScholar-Nov-2025 · p. 19

    Instantiating new slices such as DeepScholar-Nov-2025 only requires specifying a set of query papers and a date range; the pipeline then automatically constructs the corresponding benchmark and produces scores under the same evaluation protocol.

    “Instantiating new slices such as DeepScholar-Nov-2025 only requires specifying a set of query papers and a date range; the pipeline then automatically constructs the corresponding benchmark and produces scores under the same evaluation protocol.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § A.4.1 Reference Coverage · p. 21

    ArXiv is a reliable and sufficiently broad source for recovering many of the essential references.

    “indicating that ArXiv is a reliable and sufficiently broad source for recovering many of the essential references.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § A.3.5 Manual Validation of LLM-based Evaluation · p. 20

    The generated nuggets are accurate and do not contain hallucinations.

    “We additionally compute the nugget precision and grounded-ness scores, following prior work [47], observing scores of . 83 and 1 . 0 respectively, demonstrating that the generated nuggets are accurate and do not contain hallucinations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A.3.3 Ablation Study: Understanding the Performance Impact of the Chosen Query · p. 19

    Document Importance and Citation Precision are the two metrics most sensitive to how the query is phrased, likely because small changes in the query can shift which highly-cited papers are retrieved and cited.

    “This suggests that these two metrics are most sensitive to how the query is phrased, likely because small changes in the query can shift which highly-cited papers are retrieved and cited.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § A.3.2 Results on DeepScholar-Nov-2025 · p. 19

    DeepScholar-Bench is defined by an automated data-curation and evaluation pipeline rather than a single fixed dataset.

    “We emphasize that DeepScholar-Bench is defined by an automated data-curation and evaluation pipeline rather than a single fixed dataset.”
    Quote in the paperSubstantial claim
62 of 62

  • The paper relies on this without support.

    Generating a related works section is a fundamental research synthesis task.

    “We study the task of generating a related works section of an academic paper, a fundamental research synthesis task, for which we leverage human-written exemplars from our automated data pipeline to ground our evaluation.”

  • The paper relies on this without support.

    Human-written related work sections from recent ArXiv papers are suitable exemplars or ground truth for evaluating generative research synthesis.

    “for which we leverage human-written exemplars from our automated data pipeline to ground our evaluation.”

  • The paper relies on this without support.

    Expert-written exemplars provide one reasonable reference set, but there may be many possible alternative sets that are likewise high-quality.

    “expert-written exemplars provide one reasonable reference set, but there may be many possible alternative sets that are likewise high-quality.”

  • The paper relies on this without support.

    Citation counts are a valid measure of document importance.

    “We compute the document importance of a retrieved set by considering the number of citations of each of its sources.”

  • The paper relies on this without support.

    Papers accepted or published at a conference are of high quality.

    “To control for paper quality, our pipeline then optionally filters papers to keep only those listed as 'accepted' or 'published' at a conference based on the ArXiv metadata.”

  • The paper relies on this without support.

    The human-written exemplar related-work section is a good source for generating nuggets.

    “For our task, we generate nuggets from the human-written exemplar related-work section for each query”

  • The paper relies on this without support.

    The human-written exemplar is a suitable reference for organization and coherence comparison.

    “perform pairwise comparisons between the system generated report and the human-written exemplar from the dataset.”

  • The paper relies on this without support.

    Recent ArXiv papers are high-quality and provide a real research synthesis task.

    “By automatically sourcing queries from high-quality, recent ArXiv papers, our benchmark mitigates the risks of data staleness and training contamination, while offering a real research synthesis task.”

  • The paper relies on this without support.

    The human-written exemplars are a valid reference standard for evaluating generative research synthesis systems.

    “OpenAI DeepResearch, as well as the o3, Claude and Gemini search agents achieve relatively high Organization scores compared to human-written exemplars.”

  • The paper relies on this without support.

    The set of important references cited in exemplars is a valid gold standard for evaluating Reference Coverage.

    “In addition, our two oracle retrievers include the Oracle Retrieval (arxiv.org) setting and the Oracle Retrieval (All) setting, which provide the system with ArXiv references and all references, respectively, from the set of important references cited in exemplars, following our methodology for evaluating Reference Coverage.”

  • The paper relies on this without support.

    The majority vote of the 11 human annotators is a reliable ground truth for validating LLM judges.

    “Table 4 shows the results of human evaluation study as confusion matrices taken between the majority vote of human annotators and the LLM-judge.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DeepResearchEval, ReportBench, DeepScholar-Bench, and DRBench rely on synthetic task generation.

    “DeepResearchEval [Wang et al., 2026], ReportBench [Li et al., 2025b], DeepScholar-Bench [Patel et al., 2025], and DRBench [Abaskohi et al., 2025], in contrast, rely on synthetic task generation.”

    From DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity (2026) · cites this paper as Patel et al., 2025

    For DeepScholar-Bench, the paper says the opposite of the attributed claim. Section 2 (pp. 3–4) says it sources queries by scraping real arXiv papers and extracts human-written related-work sections as exemplars; it does not describe synthetic task generation. This verdict does not assess the other benchmarks cited in the sentence. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Deep research systems are increasingly relevant to knowledge-intensive domains such as academic research, medical decision support, legal analysis, and financial analysis.

    “Deep research systems are increasingly relevant to knowledge-intensive domains, such as academic research [Patel et al., 2025, Zhou et al., 2025], medical decision support [Chen et al., 2025b, Wu et al., 2025], legal analysis [Li et al., 2025a], and financial analysis [Zhu et al., 2025, Bigeard et al., 2025].”

    From DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity (2026) · cites this paper as Patel et al., 2025, Zhou et al., 2025

  • Current deep research systems exhibit poor citation recall.

    “current DR systems exhibit poor citation recall.”

    From Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research (2026) · cites this paper as Patel et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • November document-importance result is not broadly low

    Major

    The November results describe Document Importance as broadly low across baselines, but Table 8 gives the o3 Search Agent a score of 0.763-far above the other reported scores.

  • ArXiv citation median conflicts with appendix

    Minor

    Table 7 labels 647.5 as the median citations per ArXiv reference. Appendix A.4.3 instead gives the ArXiv-only median as 36 and identifies approximately 647.6 as the average.

  • November o3 variant does not outperform on Claim Coverage

    Minor

    The text says both o3-based variants substantially outperform their Llama-4 counterparts on Claim Coverage. In Table 8, DeepScholar-ref with GPT-4.1 + o3 scores 0.578, slightly below its Llama-4 counterpart's 0.581.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. [6][6] arxiv. arXiv.org e-Print archive, 2025. URL https://arxiv.org/ . -
  2. [24][24] Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. QASA: Advanced Question Answering on Scientific Articles. In Proceedings of the 40th International Conference on Machine Learning , pages 19036-19052. PMLR, July 2023. URL https: //proceedings.mlr.press/v202/lee23n.html . -
  3. [35][35] OpenAlex. OpenAlex: The open catalog to the global research system | OpenAlex, 2025. URL https: //openalex.org/ . -

Cited by

Sorted by citations

Authors
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2508.20033
arXiv
2508.20033
OpenAlex
W7078321442
Semantic Scholar
565864b8d6d7198fbc4cbd7fdbc3c9e6a3525b43
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
automated assessment, information retrieval, citation generation, knowledge synthesis, verifiability, arXiv papers, search agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review