HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

Progressive gating analysis shows that the main bottlenecks arise earlier in the pipeline, especially in evidence identification and intermediate claim construction.

“Further progressive gating analysis shows that the main bottlenecks arise earlier in the pipeline, especially in evidence identification and intermediate claim construction”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Experiments on 16 representative multimodal large language models show that many systems achieve strong report quality but their performance drops markedly on citation accuracy, claim construction, and answer correctness.

“Experiments on 16 representative multimodal large language models show that, although many systems achieve strong report quality, their performance drops markedly on citation accuracy, claim construction, and answer correctness.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Extensive experiments across 16 competitive multimodal large language models under both RAG and deep research paradigms reveal a consistent gap between report quality and faithful evidence aggregation.

“Extensive experiments across 16 competitive multimodal large language models under both the RAG and deep research paradigms reveal a consistent gap between report quality and faithful evidence aggregation.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

HiEviDR-Bench is built from a large multimodal corpus from two complementary domains: Wikipedia and arXiv.

“We build a large multimodal corpus from two complementary domains: Wikipedia, which provides broad open-domain knowledge with text chunks and associated images [1,27], and arXiv, which provides recent RAG-related papers parsed by MoDora [38] into textual and visual content for domain-specific evaluation.”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

While many models obtain relatively strong Report scores, their performance drops substantially on Citation, Claim, and Answer.

“While many models obtain relatively strong Report scores, their performance drops substantially on Citation, Claim, and Answer”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The stage distribution under the progressive gating pipeline is broadly consistent with the final ranking: stronger models tend to perform better at later gates, especially those related to claim composition and answer generation.

“Overall, the stage distribution is broadly consistent with the final ranking: stronger models tend to perform better at later gates, especially those related to claim composition and answer generation.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Alternative judges produce similar system rankings.

“These results indicate that Qwen3-VL-235B-A22B-Instruct achieves great alignment with human judgments, while alternative judges produce similar system rankings, confirming that our main findings are not an artifact of a single judge model.”

Multimodal settings exhibit larger variation than text-only settings in both Wiki and arXiv.

“A related pattern is that multimodal settings exhibit larger variation than text-only settings in both Wiki and arXiv.”

Qwen3-VL-235B-A22B-Instruct achieves great alignment with human judgments.

“These results indicate that Qwen3-VL-235B-A22B-Instruct achieves great alignment with human judgments, while alternative judges produce similar system rankings, confirming that our main findings are not an artifact of a single judge model.”

Experiments on representative MLLMs show that current systems still struggle with faithful answer generation.

“Experiments on representative MLLMs show that current systems still struggle with evidence traceability, claim-level reasoning, and faithful answer generation.”

The benchmark is divided into two modality settings to support evaluation of both text-only and multimodal deep research methods.

“To support evaluation of both text-only and multimodal deep research methods, the benchmark is divided into two modality settings.”

The gap between report quality and evidence-grounded reasoning is consistent across model families.

“This gap is consistent across model families”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Experiments on representative MLLMs show that current systems still struggle with claim-level reasoning.

“Experiments on representative MLLMs show that current systems still struggle with evidence traceability, claim-level reasoning, and faithful answer generation.”

Incorrect graph links are rare.

“Table 4 shows high adjudicated accuracy, raw agreement, and Cohen's 𝜅, indicating that the filtered evidence graphs are reliable; the few remaining errors mainly involve ambiguous questions, incomplete evidence, or mildly over-generalized claims, while incorrect graph links are rare.”

The average graph size increases consistently from Easy to Hard.

“the average graph size increases consistently from Easy to Hard”

Deep Research shows more consistent gains on the Wiki subsets, especially in the text-only setting.

“Deep Research shows more consistent gains on the Wiki subsets, especially in the text-only setting, while its gains on the arXiv subsets are smaller and more uneven.”

Experiments on representative MLLMs show that current systems still struggle with evidence traceability.

“Experiments on representative MLLMs show that current systems still struggle with evidence traceability, claim-level reasoning, and faithful answer generation.”

From 30,000 randomly selected anchor evidence items, only 3,407 pass all filters, resulting in an acceptance rate of 11.36%.

“Our pipeline applies strict rule-based filtering and consistency checks to remove noisy, incomplete, or weakly grounded samples; from 30,000 randomly selected anchor evidence items, only 3,407 pass all filters, resulting in an acceptance rate of 11.36%.”

The results reveal a clear mismatch between report quality and evidence-grounded reasoning.

“However, the results also reveal a clear mismatch between report quality and evidence-grounded reasoning.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The most dominant failure patterns across models are concentrated in No Gate Pass and Claim Gate.

“More importantly, the most dominant failure patterns across models are concentrated in No Gate Pass and Claim Gate.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The proportion at Answer Gate is relatively smaller.

“By comparison, the proportion at Answer Gate is relatively smaller”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Table 4 shows high adjudicated accuracy, raw agreement, and Cohen's κ.

“Table 4 shows high adjudicated accuracy, raw agreement, and Cohen's 𝜅, indicating that the filtered evidence graphs are reliable; the few remaining errors mainly involve ambiguous questions, incomplete evidence, or mildly over-generalized claims, while incorrect graph links are rare.”

Stronger models tend to distinguish themselves more clearly on later evidence-grounded dimensions than on report quality alone.

“Stronger models also tend to distinguish themselves more clearly on these later evidence-grounded dimensions than on report quality alone”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

For several model pairs, Deep Research improves intermediate gate performance even when the final overall gain is limited.

“In addition, for several model pairs, Deep Research improves intermediate gate performance even when the final overall gain is limited.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The results in Table 7 show consistently low standard deviations across all evaluation metrics.

“The results in Table 7 show consistently low standard deviations across all evaluation metrics, demonstrating the stability and reproducibility of our method.”

Multimodal settings exhibit larger variation than text-only settings in both the Wiki and arXiv subsets.

“A related pattern is that multimodal settings exhibit larger variation than text-only settings in both Wiki and arXiv.”

Deep Research generally achieves stronger performance than vanilla RAG on HiEviDR-Bench.

“Overall, Deep Research generally achieves stronger performance than vanilla RAG”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Most existing deep research benchmarks assess primarily final outputs such as answer correctness, report quality, or citation quality.

“Most benchmarks assess primarily final outputs, such as answer correctness, report quality, or citation quality, while treating much of the evidence aggregation process as a black box [8,29].”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The central limitation of existing deep research systems lies in evidence composition and claim-level reasoning, rather than in report fluency alone.

“Taken together, our results show that the central limitation of existing deep research systems lies in evidence composition and claim-level reasoning, rather than in report fluency alone.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Current benchmarks still lack an explicit framework for evaluating whether retrieved evidence is correctly organized into coherent intermediate claims and final conclusions.

“Nevertheless, current benchmarks still lack an explicit framework for evaluating whether retrieved evidence is correctly organized into coherent intermediate claims and final conclusions.”

By requiring that every scoring dimension is anchored to specific nodes and edges in the graph, HiEviDR-Bench can distinguish between methods that build conclusions through faithful evidence aggregation and those that only generate superficially convincing responses.

“By requiring that every scoring dimension is anchored to specific nodes and edges in the graph, our benchmark can distinguish between methods that build conclusions through faithful evidence aggregation and those that only generate superficially convincing responses.”

All constructed evidence graphs and question-answer pairs in HiEviDR-Bench are verified by human annotators before inclusion in the final benchmark.

“Although the pipeline is largely LLM-assisted, all constructed evidence graphs and question-answer pairs are further verified by human annotators before inclusion in the final benchmark.”

The progression in graph complexity directly governs the difficulty of evidence aggregation required to answer the corresponding question.

“This progression in graph complexity directly governs the difficulty of evidence aggregation required to answer the corresponding question”

Multimodal scenarios involving tables, charts, and figures as evidence remain largely unexplored in deep research evaluation.

“Meanwhile, multimodal scenarios involving tables, charts, and figures as evidence remain largely unexplored [8,45].”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Strong surface-level report quality does not necessarily imply grounded multi-stage reasoning on HiEviDR-Bench.

“revealing that the strong surface-level report quality does not necessarily imply grounded multi-stage reasoning on our benchmark.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Most current benchmarks either focus on textual outputs or assess multimodal understanding in relatively narrow settings.

“However, most current benchmarks either focus on textual outputs or assess multimodal understanding in relatively narrow settings [3,45].”

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Existing deep research evaluation does not provide structured metrics for whether retrieved evidence is correctly organized into coherent intermediate claims.

“nor does it provide structured metrics for whether retrieved evidence is correctly organized into coherent intermediate claims [20].”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The main bottlenecks in current deep research systems lie in evidence identification and intermediate claim construction.

“Further analysis shows that the main bottlenecks lie in evidence identification and intermediate claim construction”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The key challenge of multimodal deep research lies less in evidence retrieval itself and more in hierarchical evidence aggregation.

“This further supports the central conclusion of HiEviDR-Bench: the key challenge of multimodal deep research lies less in evidence retrieval itself and more in hierarchical evidence aggregation.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The benefits of Deep Research become less stable when stronger cross-modal grounding and more complex evidence composition are required.

“This indicates that the benefits of Deep Research become less stable when stronger cross-modal grounding and more complex evidence composition are required.”

• Also stated in: § ### Key · hedged

  “This indicates that the benefits of Deep Research become less stable when stronger cross-modal grounding and more complex evidence composition are required.”

The report generation framework supports the integration of retrieved images into the final rendered output rather than limiting the report to text-only summaries.

“This example also illustrates the practical advantage of our report generation framework, which supports the integration of retrieved images into the final rendered output rather than limiting the report to text-only summaries.”

Main findings are not an artifact of a single judge model.

“These results indicate that Qwen3-VL-235B-A22B-Instruct achieves great alignment with human judgments, while alternative judges produce similar system rankings, confirming that our main findings are not an artifact of a single judge model.”

The progressive gating mechanism avoids applying all evaluation dimensions indiscriminately to every sample.

“By doing so, the framework avoids applying all evaluation dimensions indiscriminately to every sample, which both preserves the process-oriented nature of our benchmark for assessing intermediate evidence aggregation and reasoning quality, and improves evaluation efficiency by reducing unnecessary judge calls.”

The filtered evidence graphs are reliable.

“Table 4 shows high adjudicated accuracy, raw agreement, and Cohen's 𝜅, indicating that the filtered evidence graphs are reliable; the few remaining errors mainly involve ambiguous questions, incomplete evidence, or mildly over-generalized claims, while incorrect graph links are rare.”

HiEviDR-Bench enables fine-grained diagnosis of how models retrieve, organize, and compose heterogeneous evidence into grounded reports.

“It enables fine-grained diagnosis of how models retrieve, organize, and compose heterogeneous evidence into grounded reports.”

Prior benchmarks cover browsing, long-form reporting, citation evaluation, interaction, or multimodal reasoning.

“Table 5 shows that prior benchmarks cover browsing, long-form reporting, citation evaluation, interaction, or multimodal reasoning, while our core novelty is the explicit evidence-to-claim-toconclusion graph for process-level grounding evaluation.”

Producing a fluent and well-structured report is considerably easier than faithfully organizing evidence into supported intermediate claims and grounded conclusions.

“indicating that producing a fluent and well-structured report is considerably easier than faithfully organizing evidence into supported intermediate claims and grounded conclusions.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The gains of Deep Research come more from suppressing distracting evidence and preserving more meaningful evidence-to-report links than from improving retrieval coverage alone.

“This indicates that the gains of Deep Research come more from suppressing distracting evidence and preserving more meaningful evidence-to-report links than from improving retrieval coverage alone.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The model can generate a content-rich and well-grounded multimodal deep research report for this case.

“Correspondingly, the model achieves consistently strong scores across all five dimensions, demonstrating that it can generate a content-rich and well-grounded multimodal deep research report for this case.”

The progressive gating mechanism improves evaluation efficiency by reducing unnecessary judge calls.

“By doing so, the framework avoids applying all evaluation dimensions indiscriminately to every sample, which both preserves the process-oriented nature of our benchmark for assessing intermediate evidence aggregation and reasoning quality, and improves evaluation efficiency by reducing unnecessary judge calls.”

The gains of Deep Research come more from suppressing distracting evidence and preserving meaningful evidence-to-report links than from improving retrieval coverage alone.

“This indicates that the gains of Deep Research come more from suppressing distracting evidence and preserving more meaningful evidence-to-report links than from improving retrieval coverage alone.”

The dual-domain setup allows HiEviDR-Bench to evaluate evidence aggregation in both open-domain and specialized academic settings.

“This dual-domain setup allows HiEviDR-Bench to evaluate evidence aggregation in both open-domain and specialized academic settings.”

Human validation was conducted on the filtered samples.

“As shown in Figure 6, we further conduct human validation on the filtered samples, and finally selected 2000 samples evenly from four subdomains.”

The pipeline applies strict rule-based filtering and consistency checks to remove noisy, incomplete, or weakly grounded samples.

“Our pipeline applies strict rule-based filtering and consistency checks to remove noisy, incomplete, or weakly grounded samples; from 30,000 randomly selected anchor evidence items, only 3,407 pass all filters, resulting in an acceptance rate of 11.36%.”

The RAG and Deep Research settings differ substantially in how evidence is gathered, organized, and incorporated before the final rendering stage.

“Although both settings ultimately produce an HTML-formatted multimodal report, they differ substantially in how evidence is gathered, organized, and incorporated before the final rendering stage.”

The key challenge of deep research lies less in evidence retrieval itself and more in organizing heterogeneous evidence into grounded intermediate claims.

“Taken together, these results further support our main conclusion that the key challenge of deep research lies less in evidence retrieval itself and more in organizing heterogeneous evidence into grounded intermediate claims.”

Deep Research is more helpful in controlling and organizing evidence than in merely finding more of it.

“Put differently, Deep Research is more helpful in controlling and organizing evidence than in merely finding more of it.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

2000 samples were selected evenly from four subdomains.

“As shown in Figure 6, we further conduct human validation on the filtered samples, and finally selected 2000 samples evenly from four subdomains.”

A judge reliability study was conducted on the human-validated subset to ensure conclusions do not rely on a single judge.

“To ensure that our conclusions do not rely on a single judge, we conduct a judge reliability study on the humanvalidated subset.”

The key challenge of multimodal deep research lies less in evidence retrieval itself and more in hierarchical evidence aggregation (organizing heterogeneous evidence into grounded intermediate claims).

“central conclusion of HiEviDR-Bench: the key challenge of multimodal deep research lies less in evidence retrieval itself and more in hierarchical evidence aggregation.”

• Also stated in: § ### Key · plain

  “Taken together, these results further support our main conclusion that the key challenge of deep research lies less in evidence retrieval itself and more in organizing heterogeneous evidence into grounded intermediate claims.”

• AssumptionRests on an assumption: The two source domains used to build the benchmark (Wiki and Arxiv) are representative of multimodal deep research tasks in general.

Explicit evidence integration is beneficial for grounded multimodal report generation.

“suggesting that explicit evidence integration is beneficial for grounded multimodal report generation.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

Once evidence has been properly organized, generating the final answer is not necessarily the hardest step.

“indicating that once evidence has been properly organized, generating the final answer is not necessarily the hardest step.”

• AssumptionRests on an assumption: The LLM judge (Qwen3-VL-235B-A22B-Instruct) produces valid and reliable scores for the evaluation dimensions.
• AssumptionRests on an assumption: The 16 evaluated models are representative of state-of-the-art multimodal large language models.

The effectiveness of Deep Research depends not only on the retrieval paradigm itself, but also on the difficulty of evidence composition in the underlying domain.

“This suggests that its effectiveness depends not only on the retrieval paradigm itself, but also on the difficulty of evidence composition in the underlying domain.”

• Also stated in: § ### Key · hedged

  “This suggests that its effectiveness depends not only on the retrieval paradigm itself, but also on the difficulty of evidence composition in the underlying domain.”

Our core novelty is the explicit evidence-to-claim-to-conclusion graph for process-level grounding evaluation.

“Table 5 shows that prior benchmarks cover browsing, long-form reporting, citation evaluation, interaction, or multimodal reasoning, while our core novelty is the explicit evidence-to-claim-toconclusion graph for process-level grounding evaluation.”

The difficulty partition corresponds to increasing evidence aggregation complexity.

“indicating that the difficulty partition also corresponds to increasing evidence aggregation complexity.”

The benchmark construction applies rule-based filtering, Judge-Model checking, and human validation to ensure structural validity and data quality.

“Based on the graph, we generate question-answer pairs and apply rule-based filtering, Judge-Model checking, and human validating to ensure structural validity and data quality.”