ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

May combine two claims.

Existing benchmarks face two primary limitations: reliance on static datasets or answer keys, and use of non-expert or automated evaluation including coarse metrics or auto-generated rubrics.

“To summarize, existing benchmarks thus face two primary limitations: a reliance on static datasets or answer keys [15, 16], and the use of non-expert or automated evaluation, including coarse metrics [20] or auto-generated rubrics [8, 11, 24].”

• cites [15] [15] M. Li, Y. Zeng, Z. Cheng, C. Ma, and K. Jia. Reportbench: Evaluating deep research agents via academic survey tasks. arXiv preprint arXiv:2508.15804, 2025. - sourcePartly supported

  ReportBench describes an automated evaluation framework in the Introduction (p. 2) and an LLM-based validation pipeline in §2.2 (pp. 4–6), supporting its use as an example of automated evaluation. But the citation is attached specifically to “auto-generated rubrics”; the paper describes evaluation against expert-authored surveys and ground-truth references, not auto-generated rubrics. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [16] [16] Y. Liu, Z. Yang, T. Xie, J. Ni, B. Gao, Y. Li, S. Tang, W. Ouyang, E. Cambria, and D. Zhou. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition. arXiv preprint arXiv:2503.21248, 2025. - sourcePartly supported

  Section 4.2 describes scoring generated hypotheses against ground-truth hypotheses using a six-point scale, supporting the characterization of automated evaluation. Appendix A.4 provides fixed scoring criteria; our copy does not say the rubrics are auto-generated. Section 3.4 also reports expert evaluation of the benchmark’s extracted components, so the source should not be characterized as relying solely on non-expert evaluation. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] [8] M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025. - sourceSupported
• cites [11] [11] B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. J. Gutiérrez, Y. Shu, C. H. Song, J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=AUaW6DS9si . - sourceSupported
• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Some recent deep research benchmarks introduce LLM-generated rubrics and evaluation metrics reliant upon LLM-generated reference reports, raising concerns about circularity and limited oversight.

“some benchmarks introduce LLM-generated rubrics and evaluation metrics reliant upon LLM-generated reference reports [8], thus raising concerns about circularity and limited oversight [7]”

• cites [8] [8] M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025. - sourceSupported
• cites [7] [7] F. E. Dorner, V. Y. Nastl, and M. Hardt. Limits to scalable evaluation at the frontier: Llm as judge won't beat twice the data. In The 13th International Conference on Learning Representations (ICLR) 2025, January 2025. URL https://arxiv.org/abs/2410.13341 . - sourcePartly supported

  The cited paper’s Section 4, “Going beyond binary evaluation” (page unavailable), reports that LLM judges may favor their own outputs and correlate poorly with expert annotations, supporting a general concern about unreliable evaluation. It does not address circularity arising from LLM-generated reference reports or establish that the benchmarks at issue have limited oversight. The benchmark-design claim is attached to a different citation, [8]. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

The Macro F1 score is used to validate the effectiveness of using a model-based grader as a proxy for human judgment, following HealthBench.

“Similar to HealthBench [3], we utilize the Macro F 1 score to validate the effectiveness of using a model-based grader as a proxy for human judgment.”

• cites [3] [3] R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL https://doi.org/10.48550/arXiv.2505.08775 . - sourceSupported

May combine two claims.

Existing QA benchmarks, both general and deep research specific, focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR.

“Typical QA benchmarks, both general [14, 17, 21, 29] and deep research specific [5, 13], focus on short, easily-verifiable factual answers and do not capture the long-form, multi-source synthesis required by DR”

• cites [14] [14] S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2025. URL https://doi.org/10.48550/ arXiv.2409.12941 . - sourcePartly supported

  The annotation instructions (p. 15) require each FRAMES question to have a single, unambiguous answer, and the evaluation (Section 3.1, p. 6) checks responses against gold answers rather than assessing long-form research output. But the Introduction (p. 1) explicitly says FRAMES tests synthesis across multiple retrieved documents, so the claim overstates its failure to capture multi-source synthesis. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [17] [17] G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. Gaia: A benchmark for general ai assistants, 2023. URL https://doi.org/10.48550/arXiv.2311.12983 . <!-- image --> sourcePartly supported

  Section 3.1 describes GAIA as a general-assistant benchmark designed for short, single correct answers that are easy to verify, supporting the claim as an example of that format. The same section says its questions can require information from different sources, but the paper does not assess deep-research-specific benchmarks or establish the broader claim that existing benchmarks fail to capture the long-form synthesis required by deep research. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [5] [5] J. Coelho, J. Ning, J. He, K. Mao, A. Paladugu, P. Setlur, J. Jin, J. Callan, J. Magalhães, B. Martins, and C. Xiong. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025. - sourceSupported
• cites [13] [13] A. Java, A. Khandelwal, S. Midigeshi, A. Halfaker, A. Deshpande, N. Goyal, A. Gupta, N. Natarajan, and A. Sharma. Characterizing deep research: A benchmark and formal definition. arXiv preprint, arXiv:2508.04183, August 2025. preprint. - sourcePartly supported

  The Introduction (p. 3) distinguishes typical multi-hop QA benchmarks from deep research tasks requiring more intensive search and reasoning, and describes some benchmarks as evaluating a single final answer. But Background & Related Work (p. 5) says most deep-research evaluation efforts assess long-form reports. The source therefore does not support the claim that deep-research-specific benchmarks generally focus on short factual answers. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Multi-Agent System Taxonomy (MAST) identified reasoning-action mismatch (13.98%) and disobedience of task specifications (10.98%) as systemic issues.

“This corroborates the findings in Multi-Agent System Taxonomy (MAST) [4], identifying reasoning-action mismatch (13.98%) and disobedience of task specifications (10.98%) as systemic issues.”

• cites [4] [4] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025. URL https://arxiv.org/abs/2503.13657 . - sourcePartly supported

  Section 4 identifies both failure modes as issues in multi-agent systems, but reports different prevalence figures: 13.2% for reasoning-action mismatch and 11.8% for failure to follow task requirements. The claimed 13.98% and 10.98% are not the figures given in our copy. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearch Bench introduced 100 PhD-level research problems across 22 fields, each requiring a long-form report.

“Closer to our setting, DeepResearch Bench [8] introduced 100 PhD-level research problems across 22 fields (e.g., scientific analysis, legal reasoning), each requiring a long-form report.”

• cites [8] [8] M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

ReportBench inherently prioritizes replication of the reference content over creative or divergent but valid answers.

“This provides a concrete correctness signal (since an expert-written literature review is treated as the gold standard), but inherently prioritizes replication of the reference content over creative or divergent but valid answers.”

• cites [15] [15] M. Li, Y. Zeng, Z. Cheng, C. Ma, and K. Jia. Reportbench: Evaluating deep research agents via academic survey tasks. arXiv preprint arXiv:2508.15804, 2025. - sourcePartly supported

  In §2.2 (pp. 4–5), ReportBench scores one aspect of content quality by overlap with the expert survey’s reference list, which favors reports citing the same works. But it also checks cited and uncited statements for factual accuracy; the paper does not establish that the evaluation as a whole penalizes creative or divergent answers that are valid. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

DeepResearch Bench's scoring approach leans heavily on overlap with reference solutions and simple citation counts.

“This benchmark confirmed the difficulty of deep research tasks, where no model exceeded roughly 30% on their overall metrics, yet its scoring approach leans heavily on overlap with reference solutions and simple citation counts.”

• cites [8] [8] M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025. - sourcePartly supported

  Section 3.1 describes RACE as comparing reports with a reference against task-specific, weighted criteria; it does not describe scoring by simple overlap. Section 3.2 and Appendix E say FACT counts citations only after checking whether the cited source supports each statement, and also measures citation accuracy—not simple citation counts. The cited sentence’s roughly 30% ceiling is also inconsistent with Table 1, which reports RACE overall scores above 40 for several models. (LLM, conf=0.89)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The paper's benchmark has a much higher average number of rubrics per task than many existing benchmarks, resulting in superior evaluation granularity.

“Additionally, our benchmark has a much higher average number of rubrics per task than many of these benchmarks, resulting in superior evaluation granularity.”

May combine two claims.

RESEARCHRUBRICS consists of 101 single-turn prompts, each paired with 20-43 prompt-specific rubric criteria.

“RESEARCHRUBRICS consists of 101 single-turn prompts, each paired with a set of 20-43 prompt-specific rubric criteria.”

May combine two claims.

The paper's outcome-based, fine-grained rubrics provide rigorous evaluation of long-form research answers and closely align with expert judgments.

“We introduce outcome-based, fine-grained rubrics that provide rigorous evaluation of long-form research answers and closely align with expert judgments.”

May combine two claims.

The paper's benchmark includes general consumer research queries, whereas ExpertLongBench does not.

“whereas we also include general consumer research queries.”

May combine two claims.

No current system exceeds 70% rubric compliance, with Gemini DR achieving 67.7% under ternary grading and 61.5% under binary evaluation.

“Table 5 reveals that no current system exceeds 70%rubric compliance , with the best-performing Gemini DR achieving only 67.7% under ternary grading and 61.5% under binary evaluation.”

• AssumptionRests on an assumption: The three evaluated commercial Deep Research agents (OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research) are representative of current Deep Research systems, so conclusions about current systems generalize to the field.

Classified as probably not a substantial claim. May combine two claims.

The paper uses powerful LLMs to assess rubric compliance and systematically experiments with binary vs. ternary grading and level of detail in rubrics.

“Following the LLM-as-judge paradigm, we use powerful LLMs to assess rubric compliance, and we systematically experiment with improving this process comparing binary vs. ternary grading for each criterion and the level of detail in the rubrics.”

Classified as probably not a substantial claim.

The paper's task complexity framework formalizes deep research queries along three axes-breadth, depth, and ambiguity-to distinguish them from conventional QA tasks and guide construction of balanced benchmarks.

“We formalize deep research queries along three axesbreadth , depth , and ambiguity -to distinguish them from conventional QA tasks and to guide the construction of balanced benchmarks that reflect real-world deep research queries.”

• AssumptionThe authors state this as an assumption: The three complexity dimensions (conceptual breadth, logical nesting depth, exploration) are independent/orthogonal.

May combine two claims.

The paper introduces practical recommendations for rubric design that improve agreement with human evaluators and are validated through ablation studies.

“We introduce practical recommendations for rubric design that improve agreement with human evaluators and are validated through ablation studies.”

Classified as probably not a substantial claim.

RESEARCHRUBRICS tasks are annotated along three complexity dimensions: conceptual breadth, logical nesting depth, and exploration.

“We categorize each RESEARCHRUBRICS task along three orthogonal complexity dimensions: Conceptual Breadth , Logical Nesting Depth , and Exploration (Table 2).”

• Also stated in: § 3.2 Prompt Complexity Dimensions · p. 5 · plain

  “Every task in RESEARCHRUBRICS is annotated with a triplet of (Breadth, Depth, Ambiguity) labels.”

• AssumptionThe authors state this as an assumption: The three complexity dimensions (conceptual breadth, logical nesting depth, exploration) are independent/orthogonal.

May combine two claims.

Agents excel at explicit factual retrieval and communication quality (failure rates below 20%) but consistently fail to infer unstated requirements or integrate multi-document evidence into coherent arguments.

“While agents excel at explicit factual retrieval and communication quality (failure rates below 20%), they consistently fail to infer unstated requirements or integrate multi-document evidence into coherent arguments.”

May combine two claims.

The benchmark's task distribution is fairly even, with AI/ML and historical analysis queries constituting the largest portions, followed by general consumer research.

“The distribution is fairly even, with AI/ML and historical analysis queries constituting the largest portions closely, followed by domains like general consumer research, reflecting both specialized academic topics and everyday research questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's benchmark includes negative criteria targeting likely pitfalls such as factually incorrect statements, off-topic tangents, or disallowed content.

“We weighted each criterion based on its importance (see Section 3.3) and included negative criteria targeting likely pitfalls, such as factually incorrect statements, off-topic tangents, or disallowed content.”

May combine two claims.

RESEARCHRUBRICS introduces 2,500+ expert-written rubric criteria across 100+ prompts, providing a more granular evaluation than existing frameworks.

“Our benchmark introduces 2,500+ expert-written rubric criteria across 100+ prompts, providing a more granular evaluation than existing frameworks.”

• AssumptionRests on an assumption: The benchmark prompts and expert-written rubric criteria are representative and comprehensive enough to measure deep research capability.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each rubric criterion is assigned a numerical weight in the range [-5, 5], with weights of ±4 or ±5 corresponding to mandatory criteria and weights in [-3, 3] optional.

“Each criterion is assigned a numerical weight in the range [ -5, 5 ] , reflecting its relative importance. Weights of ± 4 or ± 5 correspond to mandatory criteria, while criteria with weights in [ -3, 3 ] are optional.”

May combine two claims.

The benchmark's prompts span both breadth (a wide variety of domains) and depth (challenging multi-step problems).

“The result is a collection of prompts that span both breadth (a wide variety of domains) and depth (challenging multi-step problems).”

Probably not a substantial claim: may not pass S4 (based).

The rubric weights are aligned with a calibrated human preference scale spanning six levels from Critically Detrimental to Critically Important.

“These weights are aligned with a calibrated human preference scale (Table 4) spanning six levels, from Critically Detrimental to Critically Important .”

Probably not a substantial claim: may not pass S3 (load-bearing).

The benchmark includes negative rubrics that penalize extraneous or incorrect content.

“The benchmarks also include negative rubrics that specifically aim to penalize extraneous or incorrect content.”

May combine two claims.

Including concrete examples within rubric criteria improves alignment by 3-4% (binary) and 2-3% (ternary).

“We find, in Table 7, that including concrete examples within rubric criteria improves alignment by 3-4% (binary) and 2-3% (ternary).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The criteria count per task ranges from 20 to 43, with a mean of approximately 26.

“Fig. 10 reports the criteria count per task (20-43; mean ≈ 26).”

May combine two claims.

Conceptual breadth correlates with difficulty, though less steeply; systems handle multi-domain synthesis better than extended inferential chaining.

“Conceptual breadth also correlates with difficulty, though less steeply; systems handle multi-domain synthesis better than extended inferential chaining.”

May combine two claims.

Binary grading achieves substantial agreement (0.72-0.76 Macro F1), approaching the best-performing LLM-judges for rubrics benchmarks in recent literature.

“Our human evaluation study (Table 6) demonstrates that binary grading achieves substantial agreement (0.72-0.76 Macro F 1 ), approaching the best-performing LLM-judges for rubrics benchmarks in recent literature.”

May combine two claims.

Mandatory criteria show systematically higher failure rates across most dimensions, with the notable exception of implicit reasoning, where optional criteria failures dominate.

“Mandatory criteria show systematically higher failure rates across most dimensions, with the notable exception of implicit reasoning, where optional criteria failures dominate.”

Classified as probably not a substantial claim.

RESEARCHRUBRICS is a new benchmark and evaluation framework for deep research agents that emphasizes fine-grained, human-aligned assessment.

“We introduced RESEARCHRUBRICS, a new benchmark and evaluation framework for deep research agents that emphasizes fine-grained, human-aligned assessment.”

May combine two claims.

Shallow reasoning tasks (single-hop or two-step queries) are handled well, while multi-step analytical or evaluative problems see sharp drops, particularly for models relying on retrieval-centric architectures.

“Whereas shallow reasoning tasks (single-hop or two-step queries) are handled well, multi-step analytical or evaluative problems see sharp drops, particularly for models relying on retrieval-centric architectures.”

May combine two claims.

Gemini-2.5-Pro emerges as the most reliable judge, achieving 0.76 agreement on binary grading.

“Gemini-2.5-Pro emerges as the most reliable judge, achieving 0.76 agreement on binary grading, though at least the 12-17 percentage point gap to best human agreement indicates remaining room for improvement.”

May combine two claims.

Gemini DR produces 111 citations with 81% accuracy, while Perplexity achieves 90% accuracy with only 31 citations.

“Gemini DR produces 111 citations with 81% accuracy while Perplexity achieves 90% accuracy with only 31 citations.”

• Also stated in: § scale · p. 12 · plain

  “Gemini's 111-citation breadth sacrifices accuracy (81%) while Perplexity's 90% accuracy comes from restrictive 31-citation coverage.”

May combine two claims.

Even leading systems achieve below 40% on technical nugget coverage despite 85% scores on organizational structure.

“The pattern suggests current architectures inherently favor creative synthesis over systematic execution, explaining why even leading systems achieve below 40% on technical nugget coverage despite 85% scores on organizational structure.”

May combine two claims.

Failure patterns across systems include 45-50% implicit criteria failures, poor multi-hop reasoning, and synthesis bottlenecks.

“The consistency of failure patterns across systems-4550% implicit criteria failures (see Fig. 5), poor multi-hop reasoning, synthesis bottlenecks-indicates fundamental architectural constraints rather than implementation differences.”

May combine two claims.

Agents achieve 76% coverage on open-ended consulting questions but struggle with technical precision tasks, contradicting intuitive difficulty expectations.

“Agents achieve 76% coverage on open-ended consulting questions but struggle with technical precision tasks, contradicting intuitive difficulty expectations.”

May combine two claims.

Gemini DR consistently leads, achieving roughly 70% average rubric compliance across most complexity tiers, followed closely by ChatGPT DR, with Perplexity DR lagging slightly behind.

“Gemini DR consistently leads, achieving roughly 70% average rubric compliance across most complexity tiers, followed closely by ChatGPT DR, and Perplexity DR lagging slightly behind.”

• AssumptionRests on an assumption: The three evaluated commercial Deep Research agents (OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research) are representative of current Deep Research systems, so conclusions about current systems generalize to the field.

May combine two claims. The quote only partly states this claim.

Perplexity DR achieves near 50% average rubric compliance across most complexity tiers under both binary and ternary grading schemes.

“and Perplexity DR lagging near 50%.”

May combine two claims.

Mandatory criteria drive failures in explicit requirements and synthesis of information, while optional criteria account for most implicit reasoning failures.

“we observe (from Fig. 8) that, while mandatory criteria drive failures in explicit requirements and synthesis of information, optional criteria account for most implicit reasoning failures.”

May combine two claims.

Today's best agents achieve only around 67% compliance with these rigorous rubrics, often falling short in integrating information across documents and providing well-justified answers with proper citations.

“Our experiments reveal that today's best agents achieve only around 67% compliance with these rigorous rubrics, often falling short in integrating information across documents and providing well-justified answers with proper citations.”

• AssumptionRests on an assumption: The three evaluated commercial Deep Research agents (OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research) are representative of current Deep Research systems, so conclusions about current systems generalize to the field.

May combine two claims.

Through 101 diverse research challenges and expert-written rubric criteria, RESEARCHRUBRICS provides a multi-dimensional lens on an agent's performance, checking factual recall, completeness, reasoning soundness, source usage, and clarity.

“Through 101 diverse research challenges and expert-written rubric criteria, our benchmark provides a multi-dimensional lens on an agent's performance-checking not just factual recall, but the completeness, reasoning soundness, source usage, and clarity of its responses.”

• AssumptionRests on an assumption: The benchmark prompts and expert-written rubric criteria are representative and comprehensive enough to measure deep research capability.

Probably not a substantial claim: may not pass S3 (load-bearing).

ChatGPT produces responses averaging 6,300-6,400 words on Markdown outputs.

“followed by ChatGPT (6,300-6,400 words)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The differences in response length between models are consistent across words and tokens, and between text and rendered formats.

“These differences are consistent across both words and tokens, and between text and rendered formats.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The mean prompt length across all 101 tasks is 87.6 words (SD 58.6), median 68, range 13-315.

“Across all tasks, prompt lengths are moderately distributed, with a mean of 87.6 ± 58.6 words (median = 68, range = 13-315).”

May combine two claims.

Conceptual breadth correlates with difficulty, though less steeply than logical nesting.

“Conceptual breadth also correlates with difficulty, though less steeply”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gemini produces the longest responses on average (7,500-7,600 words) on Markdown outputs.

“On the Markdown outputs (Table 9), Gemini produces the longest responses on average (7,500-7,600 words)”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Using human-written rubrics with LLM judges avoids simplistic overlap measures while maintaining scalability.

“By using human-written rubrics with LLM judges, we avoid simplistic overlap measures while maintaining scalability.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The paper's rubrics are fully human-written and reviewed from the outset.

“Our rubrics, by comparison, are fully human-written and reviewed from the outset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All participants in the data collection only chose and worked on domains they were familiar with.

“All participants in our data collection only chose and worked on domains they were familiar with.”

Classified as probably not a substantial claim. May combine two claims.

The paper's complexity framework helps ensure balanced mix and allows analysis of where agents struggle most.

“This framework helps ensure our benchmark covers a balanced mix of task types and allows analysis of where agents struggle most.”

May combine two claims.

All prompts and rubric criteria in RESEARCHRUBRICS were written and iteratively refined by human experts, with no criteria seeded or generated by LLMs.

“Every prompt and criterion in RESEARCHRUBRICS was written and iteratively refined by human experts to ensure clarity and relevance (no criteria were seeded or generated by LLMs).”

• Also stated in: § Our contributions are as follows: · p. 2 · plain

  “Crucially, each rubric is both written and reviewed by humans, thereby mitigating potential anchoring biases that may arise when only verifying LLM-generated rubrics.”

Classified as probably not a substantial claim.

Each model response is evaluated against all the rubric criteria using a model as a grader, in an LLM-as-a-judge setup.

“Each model response is evaluated against all the rubric criteria using a model as a grader, in an LLM-as-ajudge setup.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The three-participant setup ensures each component is thoroughly reviewed multiple times, guaranteeing high quality in the final data.

“This three-participant setup ensures that each component is thoroughly reviewed multiple times, guaranteeing high quality in the final data.”

May combine two claims.

ExpertLongBench relies on high-quality existing references for evaluation using the CLEAR framework, which limits the scope of prompts to highly academic or professional queries.

“ExpertLongBench [22] is similar to our benchmark in that it targets expert-level, long-form tasks across nine domains with domain-specific rubrics. However, it relies on high-quality existing references for evaluation using the CLEAR framework, which limits the scope of prompts to highly academic or professional queries”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Human-model alignment is quantified using Macro F1 scores, with nine expert annotators providing ground truth across 303 responses.

“Human-model alignment is quantified using Macro F 1 scores, with nine expert annotators providing ground truth across 303 responses.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The paper's weights encourage more consistent human-model agreement during grading.

“This mapping encourages more consistent human-model agreement during grading.”

May combine two claims.

LiveResearchBench found leading systems score below 74% on comprehensive metrics, and DeepResearch Bench found leading systems score below 50% on comprehensive metrics.

“This aligns with findings from LiveResearchBench, where leading systems score below 74% on comprehensive metrics, DeepResearch Bench, where leading systems score below 50% on comprehensive metrics.”

• cites LiveResearchBenchNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites DeepResearch BenchNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

RESEARCHRUBRICS complements efforts like ExpertLongBench, while emphasizing domain diversity, rubric quality, and a focus on deep-research specific tasks.

“RESEARCHRUBRICS complements efforts like ExpertLongBench, while emphasizing domain diversity, rubric quality, and a focus on deep-research specific tasks.”

May combine two claims.

Multi-hop reasoning studies demonstrate that while agents achieve 80%+ success on first-hop inference, bridge entity resolution in early neural layers creates hard limits on subsequent reasoning depth.

“Multi-hop reasoning studies [29] demonstrate that while agents achieve 80%+ success on first-hop inference, bridge entity resolution in early neural layers creates hard limits on subsequent reasoning depth.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

ExpertLongBench's CLEAR evaluation framework depends on high-quality reference outputs for each task.

“This method enables fine-grained assessment of content requirements, but it depends on high-quality reference outputs for each task.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

DeepScholar-Bench is specialized to academic writing tasks and uses automated metrics which may introduce evaluation circularity.

“However, DeepScholar-Bench is specialized to academic writing tasks, and uses automated metrics (including LLM-generated scores) which may introduce evaluation circularity.”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

RESEARCHRUBRICS's granularity enables identification of specific capability gaps invisible to aggregate metrics, and the mandatory/optional distinction places an agent on the sufficiency-excellence continuum, aiding deployment decisions by focusing on minimum viable performance rather than average scores.

“RESEARCHRUBRICS 's granularity enables us to identify specific capability gaps invisible to aggregate metrics, and the mandatory/optional distinction gives us a way to place an agent on the sufficiency-excellence continuum, aiding deployment decisions by focusing on minimum viable performance rather than average scores.”

May combine two claims.

Current systems meet basic implicit requirements but miss nuanced quality indicators that distinguish professional from adequate research.

“This suggests current systems meet basic implicit requirements but miss nuanced quality indicators that distinguish professional from adequate research.”

Classified as probably not a substantial claim. May combine two claims.

The persistence of the correlation at the criterion level suggests either that comprehensive responses naturally satisfy more criteria or that length bias operates even on supposedly objective checkpoints.

“However, the correlation persists even at the criterion level, suggesting that either (1) comprehensive responses naturally satisfy more criteria, or (2) length bias operates even on supposedly objective checkpoints.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ExpertLongBench targets expert-level, long-form tasks in 9 domains, providing 11 complex prompts each accompanied by a domain-specific checklist or rubric.

“Similarly, ExpertLongBench [22] targets expert-level, long-form tasks in 9 domains (law, finance, healthcare, etc.), providing 11 complex prompts each accompanied by a domain-specific checklist or rubric.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Fully automatic rubric generation can miss domain nuances or implicitly favor certain solution paths.

“One limitation, however, is that fully automatic rubric generation can miss domain nuances or implicitly favor certain solution paths.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearch Arena automatically curates over 10,000 open-ended tasks from transcripts of academic seminars across 12 disciplines.

“Meanwhile, DeepResearch Arena [24] addresses the authenticity of research prompts: it automatically curates over 10,000 open-ended tasks from transcripts of academic seminars across 12 disciplines.”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The length-quality conflation hypothesis is supported: longer reports often perform better because they cover more rubric criteria, not necessarily because evaluators prefer verbosity.

“This supports the length-quality conflation hypothesis: longer reports often perform better because they cover more rubric criteria, not necessarily because evaluators prefer verbosity.”

• AssumptionRests on an assumption: Evaluators show a documented bias toward verbosity independent of content quality.

May combine two claims.

The mandatory/optional distinction is essential for deployment decisions because a 60% overall score might indicate either dangerous gaps in core requirements or merely missing polish on otherwise solid foundations.

“The mandatory/optional distinction proves essential for deployment decisions: a 60% overall score might indicate either dangerous gaps in core requirements or merely missing polish on otherwise solid foundations.”

May combine two claims.

Human-authored concise rubrics with targeted examples outperform machine-generated verbose descriptions, likely because augmentation introduces semantic drift and emphasis distortion.

“Human-authored concise rubrics with targeted examples outperform machine-generated verbose descriptions, likely because augmentation introduces semantic drift and emphasis distortion.”

Classified as probably not a substantial claim. May combine two claims.

ResearcherBench found that systems excel at exploratory reasoning while failing on deterministic requirements.

“This aligns with ResearcherBench [28] findings that systems excel at exploratory reasoning while failing on deterministic requirements.”

• cites [28]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Improving Deep Research agents requires architectural innovation rather than incremental refinement, because systematic failures in implicit reasoning, multi-document synthesis, and sustained sequential reasoning point to fundamental limitations in how current systems represent and manipulate complex information structures.

“Most critically, our findings suggest that improving Deep Research agents requires architectural innovation rather than incremental refinement: systematic failures in implicit reasoning, multi-document synthesis, and sustained sequential reasoning point to fundamental limitations in how current systems represent and manipulate complex information structures.”

May combine two claims.

Expert curation cannot be replaced by automated expansion, and clarity emerges from precision rather than elaboration.

“The implication for RESEARCHRUBRICS' 2,593 criteria is clear: expert curation cannot be replaced by automated expansion, and clarity emerges from precision rather than elaboration .”

May combine two claims.

Systems optimized for comprehensive coverage sacrifice precision, while those targeting accuracy miss crucial perspectives; neither strategy successfully handles the implicit judgment of source relevance and authority.

“Systems optimized for comprehensive coverage sacrifice precision, while those targeting accuracy miss crucial perspectives-neither strategy successfully handles the implicit judgment of source relevance and authority.”

May combine two claims.

The consistency levels validate automated evaluation feasibility for RESEARCHRUBRICS's 2,593 criteria, exceeding HealthBench's 0.709 Macro F1 score.

“The consistency levels validate automated evaluation feasibility for RESEARCHRUBRICS's 2,593 criteria, exceeding HealthBench's 0.709 Macro F 1 score.”

• cites HealthBenchNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Partial credit introduces ambiguity without improving discriminative power.

“The shift from ternary to binary evaluation increases agreement by approximately 20 percentage points, confirming that partial credit introduces ambiguity without improving discriminative power.”