CoChat
Sign Up
PreprintCited by 5Information Retrieval and Search BehaviorMultimodal Machine Learning ApplicationsTopic Modeling

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Open access
Ye, Fangda, Hu, Yuxin, Zhu, Pengxiang, Li, Yibo, Jin, Ziqi, Xiao, Yao, Wang, Yibo, Wang, Lei +14 more·2026·arXiv
DOI 10.48550/arxiv.2603.28407arXiv 2603.28407
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.

AI snapshot

Written from the paper’s full text.

TL;DR

MiroEval introduces a 100-task benchmark (70 text-only, 30 multimodal) and a three-layer evaluation framework for deep research agents, covering adaptive synthesis quality, agentic factuality, and process-centric evaluation. Across 13 systems, the three dimensions are complementary, process quality predicts overall outcome, and multimodal tasks reduce most systems’ scores by 3–10 points, with MiroThinker-H1 highest overall (77.5 text-only, 74.5 multimodal).

Key findings

  • The benchmark contains 100 tasks (70 text-only, 30 multimodal) across 12 domains and 10 task types; human verification by three expert annotators reports 92.0% overall precision, with Fleiss’ κ of 0.81 for validity and 0.76 for non-triviality.
  • Evaluation across 13 systems shows the three dimensions capture non-redundant capability: rankings shift across synthesis quality, factual precision, and process rigor, while process quality is broadly aligned with outcome quality (top process systems: MiroThinker-H1 74.7, OpenAI Deep Research 73.1, MiroThinker-1.7 72.7).
  • Multimodal tasks are substantially harder: most systems decline by 3–10 points overall; across eight systems with both settings, average overall score drops 3.1 points, synthesis quality drops about 6 points, process about 4 points, and factuality ratio only 0.2 points. MiroThinker-H1 declines least (3.0 points), while Qwen-3.5-Plus declines most overall (8.6 points).
  • Process-report alignment reveals a traceability gap: Findings→Report scores are high (MiroThinker-H1 87.0), but Report→Process scores are much lower (MiroThinker-H1 63.3), with gaps exceeding 23 points for MiroThinker-H1, 24 points for OpenAI, and approaching 30 points for Doubao.

Methods

  • Benchmark construction uses a dual-path pipeline: 65 user-derived queries (35 text-only, 30 multimodal) from privacy-preserving rewriting and difficulty stratification, plus 35 auto-generated text-only queries from trend-grounded generation and three-stage filtering (180 candidates → 152 → 96 → 50 → 35), totaling 100 tasks.
  • Evaluation has three layers: adaptive synthesis quality with task-specific rubrics and weights; agentic factuality that decomposes reports into atomic claims and verifies them against web sources and attachments using RIGHT/WRONG/CONFLICT/UNKNOWN labels; and process-centric evaluation auditing search breadth, analytical depth, progressive refinement, critical thinking, efficiency, plus Process→Report and Report→Process alignment and contradiction detection.
  • The study evaluates 13 deep research systems using GPT-5.1 as synthesis judge, GPT-5.2 for process, and GPT-5-mini for factuality, with robustness checks (repeated runs, Gemini judge, prompt variation) and a human ranking study (5 annotators, 5 queries; Kendall’s τ = 0.91).

Limitations

  • Stated: process evaluation relies on systems exposing intermediate reasoning traces, limiting applicability to fully closed-source systems that do not provide such access.
  • Stated: factuality evaluation identifies cross-source conflicts but does not resolve them; the CONFLICT label flags disagreements without determining which source is correct.
  • Evident from the text: auto-generated queries are consistently easier than user-derived ones, with overall improvements ranging from 0.6 points for Claude-Opus-4.6 to 6.7 points for Doubao Deep Research, so benchmark difficulty depends on the source mix. Also, absolute scores vary by judge model (Gemini inflates Overall by 13–17 points), though rankings are preserved.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.28407

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 38 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Process quality serves as a reliable predictor of overall outcome.

    “process quality serves as a reliable predictor of overall outcome”
    Quote in the paperSubstantial claim
  • Findingreasoning

    All 100 tasks are grounded in real user needs.

    “all grounded in real user needs”
    • Also stated in: § 1. Introduction · plain
      “all grounded in real user needs and constructed through two complementary paths”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The internal testing phase queries reflect real user needs.
  • Findingown data

    MiroEval comprises 100 tasks, 70 text-only and 30 multimodal.

    “The benchmark comprises 100 tasks (70 text-only, 30 multimodal)”
    • Also stated in: § 2.1. Benchmark Overview · plain
      “The final benchmark comprises 100 queries: 70 text-only and 30 multimodal (Figure 3).”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Synthesis quality evaluation and factual verification do not directly evaluate the quality of the underlying research process.

    “While synthesis quality evaluation and factual verification assess the final research artifact, they do not directly evaluate the quality of the underlying research process.”
    Quote in the paperSubstantial claim
  • Findingown data

    MiroThinker-1.7-mini produces 3,397 correct and 246 wrong claims.

    “Even MiroThinker-1.7-mini maintains this discipline with 3,397 correct and 246 wrong claims.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 72
  • May combine two claims. 100
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The first path curates 65 queries (35 text-only and 30 multimodal).

    “The first path curates 65 queries (35 text-only and 30 multimodal)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The internal testing phase queries reflect real user needs.

  • May combine two claims.

    Findingown data

    The three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems.

    “the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The evaluation suite assesses systems through three complementary layers.

    “The evaluation suite assesses systems through three complementary layers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The benchmark is constructed through two complementary paths.

    “constructed through two complementary paths”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The second path generates 35 text-only queries.

    “The second path generates 35 text-only queries”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-1.7 ranks among the top three on Synthesis, Factuality, and Process with no significant weakness on any axis.

    “MiroThinker-1.7 follows closely, ranking among the top three on Synthesis, Factuality, and Process with no significant weakness on any axis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A lower tier in the Text-Only setting includes Manus-1.6-Max, Qwen-3.5-Plus, Claude-Opus-4.6, Doubao and Grok, all scoring below 65, with Grok trailing at 60.2.

    “A lower tier includes Manus-1.6-Max , Qwen-3.5-Plus,Claude-Opus-4.6, Doubao and Grok, all scoring below 65, with Grok trailing at 60.2.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    The resulting set of 65 queries covers all 8 evaluation features with balanced representation across difficulty tiers.

    “The resulting set of 65 queries covers all 8 evaluation features with balanced representation across difficulty tiers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points.

    “multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The process-centric evaluation does not directly assess the fluency, stylistic quality, or factual correctness of the final report.

    “this component does not directly assess the fluency, stylistic quality, or factual correctness of the final report”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1 ranks highest overall in both text-only and multimodal settings.

    “MiroThinker-H1 ranking the highest overall in both settings”
    • Also stated in: § 1. Introduction · plain
      “MiroThinker-H1 achieving the highest overall scores in both text-only (77.5) and multimodal (74.5) settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The agentic factuality evaluation framework is based on MiroFlow and enables an evaluation agent to retrieve and reason over evidence from multiple sources.

    “we design an agentic factuality evaluation framework based on MiroFlow [27] that enables an evaluation agent to retrieve and reason over evidence from multiple sources”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [28]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The three most common task types are Decision & Recommendation (17), Comparative Analysis (16), and Fact Enumeration & Verification (15), together accounting for nearly half the benchmark.

    “The three most common types are Decision & Recommendation (17), Comparative Analysis (16), and Fact Enumeration & Verification (15), which together account for nearly half the benchmark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A broadly similar grouping holds in the MultiModal setting, though overall scores decrease by 3 to 10 points across systems and the inter-system gaps narrow.

    “A broadly similar grouping holds in the MultiModal setting, though overall scores decrease by 3 to 10 points across systems and the inter-system gaps narrow.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The process-centric evaluation framework is organized into three components: process representation, process quality evaluation, and alignment between process-level key findings and report-level key findings.

    “Our framework is organized into three components: process representation, process quality evaluation, and alignment between process-level key findings and report-level key findings.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Process quality is an important evaluation target.
    • The authors state this as an assumption: A high-quality final report is only one facet of a successful investigation.

  • May combine two claims.

    Findingown data

    OpenAI Deep Research generates fewer correct claims at 3,335, but keeps wrong claims to just 170 and unverifiable claims to 496, achieving the highest per-task right Ratio of 83.3.

    “At the other extreme, OpenAI Deep Research generates fewer correct claims at 3,335, but keeps wrong claims to just 170 and unverifiable claims to 496, achieving the highest per-task right Ratio of 83.3.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Kimi-K2.5 achieves the highest Synthesis Avg among non-MiroThinker systems yet one of the lowest Factuality Ratios, combining the leading Insight score of 79.8 with a high wrong-claim count of 595 and the second largest pool of unverifiable claims at 1,256.

    “Kimi-K2.5, which achieves the highest Synthesis Avg among non-MiroThinker systems yet one of the lowest Factuality Ratios, 68.1 86.6 63.4 86.9 79.0 73.5 turns out to combine the leading Insight score of 79.8 with a high wrong-claim count of 595 and the second largest pool of unverifiable claims at 1,256.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGLM Agent and Gemini-3.1-Pro produce over 4,000 correct claims each, with 580 and 526 wrong claims respectively, plus over 900 unverifiable ones, pulling their Factuality Ratios down to the low 70s.

    “At one extreme, ChatGLM Agent and Gemini-3.1-Proproduce over 4,000 correct claims each, but this high volume comes with 580 and 526 wrong claims respectively, plus over 900 unverifiable ones, pulling their Factuality Ratios down to the low 70s.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1 produces the highest claim volume among top-tier systems at 3,746 correct claims while maintaining only 161 wrong ones, the lowest absolute error count of any system and a Ratio of 81.1.

    “MiroThinker-H1 produces the highest claim volume among top-tier systems at 3,746 correct claims while maintaining only 161 wrong ones, the lowest absolute error count of any system and a Ratio of 81.1.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The structured representation is used only as an auxiliary analytical interface.

    “this structured representation is used only as an auxiliary analytical interface”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1, OpenAI Deep Research, and MiroThinker-1.7 form the top tier in the Text-Only setting with overall scores of 77.5, 76.7, and 75.5 respectively.

    “MiroThinker-H1, OpenAI Deep Research, and MiroThinker-1.7 form the top tier at 77.5, 76.7, and 75.5 respectively, with MiroThinker-1.7-mini close behind at 72.9.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The final benchmark combines 65 user-derived and 35 auto-generated queries for a total of 100.

    “The final benchmark combines 65 user-derived and 35 auto-generated queries for a total of 100”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Deep-research necessity filter yields 96 queries (63.2%).

    “yielding 96 queries (63.2%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The inverse quality assessment filter retains 35 queries as the final auto-generated set.

    “35 queries are selected from the filtered pool as the final auto-generated set.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Effective benchmark queries should expose the limitations of parametric knowledge.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Each query is annotated with one of 10 task types.

    “We annotate each query with one of 10 task types”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Search validation retains 152 queries (84.4%).

    “This retains 152 queries (84.4%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1 achieves the highest MultiModal score at 74.5, followed by MiroThinker-1.7 at 71.6 and OpenAI Deep Research at 70.2.

    “MiroThinker-H1 achieves the highest MultiModal score at 74.5, followed by MiroThinker-1.7 at 71.6 and OpenAI Deep Research at 70.2”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Insight scores range from 54.8 for Manus to 80.3 for MiroThinker-H1, a 25-point spread that is substantially wider than Coverage or Instruction-following.

    “Scores range from 54.8 for Manus to 80.3 for MiroThinker-H1, a 25-point spread that is substantially wider than Coverage or Instruction-following.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Instruction-following is uniformly high among top systems and is no longer a meaningful differentiator.

    “Instruction-following, by contrast, is uniformly high among top systems and is no longer a meaningful differentiator.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Technology (20) and Finance (17) are the most represented domains, followed by Science (13).

    “Technology (20) and Finance (17) are the most represented, followed by Science (13).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MiroThinker-H1 achieves the highest overall score in both the Text-Only (77.5) and MultiModal (74.5) settings, ranking first or second on every individual dimension.

    “MiroThinker-H1 achieves the highest overall score in both the Text-Only (77.5) and MultiModal (74.5) settings, ranking first or second on every individual dimension.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Claude-Opus-4.6 has the highest Breadth of 79.1 among all systems, but its Depth of 58.8 trails behind by around 8 points.

    “Its Breadth of 79.1 is the highest among all the systems, but its Depth of 58.8 trails behind by around 8 points”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • Classified as probably not a substantial claim.

    Findingreasoning

    The tool adopts a hybrid processing strategy to accommodate diverse formats.

    “The tool adopts a hybrid processing strategy to accommodate the diverse formats encountered in realistic research scenarios.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    These mechanisms allow the evaluation agent to access and reason over information contained in diverse attachments.

    “Together, these mechanisms allow the evaluation agent to access and reason over information contained in diverse attachments”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the Text-Only setting, Breadth scores cluster between 71 and 77 for most competitive systems, whereas Depth scores spread from 41.6 for Doubao to 67.3 for OpenAI Deep Research.

    “In the Text-Only setting, Breadth scores cluster between 71 and 77 for most competitive systems, whereas Depth scores spread far more widely, from 41.6 for Doubao to 67.3 for OpenAI Deep Research.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • May combine two claims.

    Findingown data

    Specificity marks the shared weakness among systems, while Insight is what most separates systems from one another.

    “While Specificity marks the shared weakness, Insight is what most separates systems from one another.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1 proves the most resilient with a decline of only 3.0 points, while Qwen-3.5-Plus suffers the largest drop at 8.6 points.

    “MiroThinker-H1 proves the most resilient with a decline of only 3.0 points, while Qwen-3.5-Plussuffers the largest drop at 8.6 points.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Both construction paths support periodic re-execution.

    “Both construction paths support periodic re-execution”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    MiroThinker-H1 provides a per-company table classifying all 20 vendors by business model, customer segment, and technology route, scoring 9.4.

    “MiroThinker-H1 provides a per-company table classifying all 20 vendors by business model, customer segment, and technology route.The evaluator notes:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Auto-generated queries are consistently easier: nearly all systems with complete data score higher on the auto-generated subset, with overall improvements ranging from 0.6 points for Claude-Opus-4.6 to 6.7 points for Doubao Deep Research.

    “Auto-generated queries are consistently easier: nearly all systems with complete data score higher on the auto-generated subset, with overall improvements ranging from 0.6 points for Claude-Opus-4.6 to 6.7 points for Doubao Deep Research.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGLM Agent fabricates growth rates from the graphic (e.g., Cursor 1000%, Kling AI 1900%), when the attachment contains no growth data at all, scoring 0.5.

    “ChatGLM Agent fabricates growth rates from the graphic (e.g.,”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    OpenAI Deep Research and the MiroThinker series occupy the top positions in both query sources, and the lower tier (Doubao, Qwen-3.5-Plus) is also consistent.

    “OpenAI Deep Research and the MiroThinker series occupy the top positions in both cases, and the lower tier (Doubao, Qwen-3.5-Plus) is also consistent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-H1 explicitly lists all 20 companies with correct names and ranks matching the attachment, scoring 9.8.

    “MiroThinker-H1 explicitly lists all 20 companies with correct names and ranks matching the attachment.The evaluator notes:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Findings→Report (F→R) scores are generally high: MiroThinker-H1 leads at 87.0, with OpenAI Deep Research and MiroThinker-1.7 both exceeding 83, and even mid-tier systems such as MiniMax-M2.5 and ChatGLM Agent remaining above 70.

    “Findings→Report (F→R) scores are generally high: MiroThinker-H1 leads at 87.0, with OpenAI Deep Research and MiroThinker-1.7 both exceeding 83, and even mid-tier systems such as MiniMax-M2.5 and ChatGLM Agent remaining above 70.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • May combine two claims.

    Findingown data

    Contradiction detection scores: MiroThinker-H1 leads at 86.4, followed by MiroThinker-1.7 at 82.5 and OpenAI at 79.9, while Doubao and Qwen-3.5-Plus score below 57.

    “MiroThinker-H1 leads decisively at 86.4, followed by MiroThinker-1.7 at 82.5 and OpenAI at 79.9, while Doubao and Qwen-3.5-Plusscore below 57”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • May combine two claims.

    Findingown data

    Gemini-3.1-Pro partially acknowledges missing panels and marks some values as unknown, but still uses retail aggregator data without confirming regional SKU matches, scoring 3.2.

    “Gemini-3.1-Propartially acknowledges missing panels and marks some values as unknown, but still uses retail aggregator data without confirming regional SKU matches.Score: 3.2.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Synthesis quality suffers the largest average decline at approximately 6 points, with Qwen-3.5-Plus experiencing an extreme drop of 15.4 points (from 60.0 to 44.6) and MiniMax-M2.5 declining by 6.6 points.

    “By dimension, Synthesis quality suffers the largest average decline at approximately 6 points, with Qwen-3.5-Plusexperiencing an extreme drop of 15.4 points (from 60.0 to 44.6) and MiniMax-M2.5 declining by 6.6 points.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Modifying the judge prompt shifts scores by less than 2 points with no rank changes.

    “modifying the judge prompt shifts scores by less than 2 points with no rank changes;”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The relative ranking of all six systems is perfectly preserved (∆Rank = 0 for every system), yielding a Kendall's τ of 1.0.

    “Crucially, despite these large absolute shifts, the relative ranking of all six systems is perfectly preserved (∆Rank = 0 for every system), yielding a Kendall's τ of 1.0”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The systematic inflation is most pronounced on Process (deltas +16.6 to +21.3) and least on Factuality (+6.0 to +11.7).

    “This systematic inflation is most pronounced on Process (deltas of +16.6 to +21.3) and least on Factuality (+6.0 to +11.7), suggesting that Gemini applies more lenient criteria for process evaluation than for factual verification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Case 2, Gemini-3.1-Pro received an overall score of 7.6 and MiniMax-M2.5 received an overall score of 4.0.

    “We compare Gemini-3.1-Pro(overall: 7.6) against MiniMax-M2.5 (overall: 4.0).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MiroEval comprises 100 tasks (70 text-only and 30 multimodal) assessed through three complementary layers: adaptive synthesis quality, agentic factuality, and process-centric evaluation.

    “We introduced MiroEval, a benchmark and evaluation framework for deep research systems, comprising 100 tasks (70 text-only and 30 multimodal) assessed through three complementary layers: adaptive synthesis quality, agentic factuality, and process-centric evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    MiniMax-M2.5 assigns subjective star ratings without explicit trade-off analysis, scoring 4.6.

    “MiniMax-M2.5 assigns subjective star ratings without explicit trade-off analysis.The evaluator notes:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Case 1, MiroThinker-H1 received an overall score of 8.5 and ChatGLM Agent received an overall score of 4.5.

    “We compare MiroThinker-H1 (overall: 8.5) against ChatGLM Agent (overall: 4.5).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The gap between F→R and R→P exceeds 23 points for MiroThinker-H1, 24 points for OpenAI, and approaches 30 points for Doubao.

    “The gap between F→R and R→P exceeds 23 points for MiroThinker-H1, 24 points for OpenAI, and approaches 30 points for Doubao”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • May combine two claims.

    Findingown data

    ChatGLM Agent introduces companies outside the top-20 and conflates them into its analysis, scoring 2.5.

    “ChatGLM Agent introduces companies outside the top-20 and conflates them into its analysis.The evaluator notes:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Text-only reports are generally longer than their multimodal counterparts.

    “A consistent pattern is that text-only reports are generally longer than their multimodal counterparts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Efficiency is a universal weakness: even the best system on this metric, MiroThinker-H1 at 68.1, scores well below its performance on other Intrinsic dimensions, and most systems fall in the 53 to 64 range.

    “Beyond Depth, Efficiency is a universal weakness: even the best system on this metric, MiroThinker-H1 at 68.1, scores well below its performance on other Intrinsic dimensions, and most systems fall in the 53 to 64 range.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Case 1, the attachment graphic contains only vendor names and ranks; no growth rates, categories, business models, or financial metrics are shown.

    “The graphic contains only vendor names and ranks; no growth rates, categories, business models, or financial metrics are shown.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Report→Process (R→P) scores are dramatically lower across the board: even the best system, MiroThinker-H1, achieves only 63.3, and most others fall below 55, with Doubao at 36.8 and Qwen-3.5-Plus at 39.7.

    “Scores are dramatically lower across the board: even the best system, MiroThinker-H1, achieves only 63.3, and most others fall below 55, with Doubao at 36.8 and Qwen-3.5-Plus at 39.7.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Gemini-3.1-Pro constructs a two-dimensional fat-phosphorus framework and proposes mixed-feeding strategies, scoring 8.9.

    “Gemini-3.1-Proconstructs a two-dimensional fat-phosphorus framework and proposes mixed-feeding strategies.The evaluator notes:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Process scores decrease by an average of roughly 4 points, with ChatGLM showing the sharpest decline of 7.9 points (from 65.6 to 57.7).

    “Process scores decrease by an average of roughly 4 points, with ChatGLM showing the sharpest decline of 7.9 points (from 65.6 to 57.7).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    EIA monthly data show Brent crude averaging approximately $106.77 in July 2014 and $62.34 in December 2014.

    “EIA monthly data show Brent crude averaging approximately $106.77 in July 2014 and $62.34 in December 2014.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites EIA

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The agentic verdict is incorrect for the Star Raiders case.

    “The agentic verdict is incorrect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    H1 explicitly states that the available evidence supports only a candidate set of major employers, not a reliable Top-10 ranking.

    “In particular, it explicitly states that the available evidence supports only a candidate set of major employers, not a reliable Top-10 ranking.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The model judgment is correct for the LiuGong case.

    “The model judgment is correct.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The model judgment is incorrect for the Alphabet generative-AI revenue case.

    “The model judgment is incorrect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    v17 introduces some unsupported extrapolations beyond the trace, while MiroThinker includes multiple quantitative ranges that are not directly grounded in the retrieved evidence.

    “By contrast, v17 introduces some unsupported extrapolations beyond the trace, while MiroThinker includes multiple quantitative ranges that are not directly grounded in the retrieved evidence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    H1 achieves the best process-report consistency because most major report claims can be linked back to process-level findings, and the report contains few substantive contradictions.

    “H1 achieves the best process-report consistency because most major report claims can be linked back to process-level findings, and the report contains few substantive contradictions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    In Correct Case 2, the system extracted all three figures from the five-year financial highlights table in the user-uploaded screenshot and independently retrieved the same table from Huawei's 2019 Annual Report.

    “The system extracted all three figures from the five-year financial highlights table in the user-uploaded screenshot, and then independently retrieved the same table from Huawei's 2019 Annual Report.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huawei 2019 Annual Report

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    MiroThinker shows the weakest structured process: it relies more heavily on broad analogy search and limited reading, with several final numerical ranges functioning as inferential placeholders rather than findings directly supported by the process.

    “MiroThinker shows the weakest structured process: it relies more heavily on broad analogy search and limited reading, with several final numerical ranges functioning as inferential placeholders rather than findings directly supported by the process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    MiroThinker-H1 reads the attachment at the very beginning and extracts a key physical constraint: the new waterway crosses the Pamir/Karakoram region.

    “MiroThinker-H1 reads the attachment at the very beginning and extracts a key physical constraint: the new waterway crosses the Pamir/Karakoram region.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    H1 yields the best alignment profile in the case: the report remains largely traceable to the process, and unsupported extrapolations are limited.

    “This yields the best alignment profile in the case: the report remains largely traceable to the process, and unsupported extrapolations are limited.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MiroThinker-v17 retrieves a larger number of quantitative fragments and surfaces several useful numerical observations, but its trajectory is less tightly organized around a single evidential hierarchy.

    “MiroThinker-v17 retrieves a larger number of quantitative fragments and surfaces several useful numerical observations, but its trajectory is less tightly organized around a single evidential hierarchy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    An internal arithmetic cross-check confirms the margin: 77,835 ÷ 858,833 ≈ 9.07%, which rounds to the reported 9.1%.

    “An internal arithmetic cross-check further confirms the margin: 77,835 ÷ 858,833 ≈ 9.07%, which rounds to the reported 9.1%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Prices did not fall below $40 per barrel until early 2016.

    “Prices did not fall below $40 per barrel until early 2016, more than a year after the period referenced in the statement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites EIA

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingreasoning

    Both the model and the human annotator correctly judge the Huawei statement as accurate.

    “Both the model and the human annotator correctly judge the statement as accurate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    WTI ended December 2014 at approximately $59.29.

    “WTI ended December 2014 at approximately $59.29.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites EIA

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    In Correct Case 1, the specific numeric claim that oil prices fell below $40 during H2 2014 is incorrect.

    “the specific numeric claim that prices fell”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites EIA

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    System rankings are fully preserved (∆Rank = 0) under prompt modification, confirming that evaluation outcomes are robust to reasonable prompt reformulations.

    “As with the cross-judge experiment, system rankings are fully preserved (∆Rank = 0), confirming that the evaluation outcomes are robust to reasonable prompt reformulations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In the first process case, H1's process is anchored by an early scope decision: only enterprise employment is counted, while government agencies, public institutions, further study, and entrepreneurship are explicitly excluded.

    “Its process is anchored by an early scope decision: only enterprise employment is counted, while government agencies, public institutions, further study, and entrepreneurship are explicitly excluded.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Its extracted findings form an 'evidence pyramid': modern inland-sea analogues, paleoclimate analogies, reverse evidence from sea retreat and aridification, and circulation-level explanations from climate-model studies.

    “Its extracted findings form an”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Gemini-3.1-Pro sources from retail aggregators rather than manufacturer datasheets, and sometimes conflates Japan-market and US-market variants, scoring 3.4.

    “Gemini-3.1-Prosources from retail aggregators rather than manufacturer datasheets, and sometimes conflates Japan-market and US-market variants.Score: 3.4.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingcitation

    The attached technical review explicitly reports a battery capacity of 350 kWh and a rated voltage of 580 V, with a charging time under 75 minutes.

    “The attached technical review explicitly reports a battery capacity of 350 kWh and a rated voltage of 580 V, with a charging time under 75 minutes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The process-level analysis identifies two systemic weaknesses shared by current deep research systems: Analytical Depth and Efficiency are the primary Intrinsic bottlenecks; and the F→R versus R→P asymmetry reveals a fundamental traceability gap.

    “The process-level analysis identifies two systemic weaknesses shared by current deep research systems.First, Analytical Depth and Efficiency are the primary Intrinsic bottlenecks: systems retrieve broadly but rarely investigate deeply, and much of the retrieval effort is wasted.Second, the F→R versus R→P asymmetry reveals a fundamental traceability gap: reports consistently contain more than what”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected case studies are representative of typical system behavior.

  • May combine two claims.

    Findingown data

    MiniMax-M2.5 provides no manufacturer URLs or datasheets for any value, scoring 0.5.

    “MiniMax-M2.5 provides no manufacturer URLs or datasheets for any value.Score: 0.5.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    The three Huawei figures are directly verifiable from Huawei's publicly available 2019 Annual Report, and the margin is arithmetically consistent with the revenue and profit values: 77,835 ÷ 858,833 = 9.07% ≈ 9.1%.

    “The three figures are directly verifiable from Huawei's publicly available 2019 Annual Report, and the margin is arithmetically consistent with the revenue and profit values: 77, 835 ÷ 858, 833 = 9.07%≈ 9.1%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    LiuGong's official product page lists the 856HE with an LFP battery rated at 432 kWh, resulting in an 82 kWh (23%) discrepancy.

    “However, LiuGong's official product page lists the 856HE with an LFP battery rated at 432 kWh, resulting in an 82 kWh (23%) discrepancy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The only dimension showing slightly larger variation is Factuality (up to -2.7 for Gemini-3.1-Pro), consistent with the higher sensitivity of claim-level verification to prompt phrasing.

    “The only dimension showing slightly larger variation is Factuality (up to -2.7 for Gemini-3.1-Pro),consistent with the higher sensitivity of claim-level verification to prompt phrasing.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Since 400% exceeds 200%, the system interpreted the statement 'over 200%' as a conservative but numerically valid description and returned a Right verdict.

    “Since 400% exceeds 200%, the system interpreted the statement”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    All three values-revenue, operating profit, and the stated margin-are consistent across both sources.

    “All three values-revenue, operating profit, and the stated margin-are consistent across both sources.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huawei 2019 Annual Report

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The system retrieved the CEO's remarks from both the uploaded earnings document and the company's official blog, which report the figure as 'nearly 400% year-over-year.'

    “The system retrieved the CEO's remarks from both the uploaded earnings document and the company's official blog, which report the figure as”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning

    Fixed evaluation dimension and criteria cannot adequately capture synthesis quality.

    “fixed evaluation dimension and criteria cannot adequately capture synthesis quality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These profiles reflect fundamentally different generation strategies: broad claim coverage at the cost of precision versus selective generation with strict factual discipline.

    “These profiles reflect fundamentally different generation strategies: broad claim coverage at the cost of precision versus selective generation with strict factual discipline.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The MiroThinker series achieves a distinctive balance between high claim volume and low error rates.

    “The MiroThinker series achieves a distinctive balance between these extremes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Raw process logs are often noisy, verbose, and heterogeneous in form.

    “raw process logs are often noisy, verbose, and heterogeneous in form”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All named entities are replaced with realistic substitutes of the same type and scale through an automated anonymization pipeline.

    “all named entities (institutions, organizations, and individuals) are replaced with realistic substitutes of the same type and scale through an automated anonymization pipeline”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data

    Manus appears to prioritize factual caution over analytical depth, a defensible strategy for high-stakes tasks but one that limits report usability.

    “Manus appears to prioritize factual caution over analytical depth, a defensible strategy for high-stakes tasks but one that limits report usability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The design prevents the benchmark from becoming stale and reduces the risk of overfitting to known tasks.

    “This design prevents the benchmark from becoming stale and reduces the risk of overfitting to known tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Kimi's reports are analytically rich but insufficiently grounded: it generates insightful interpretations that are not always backed by verifiable evidence.

    “In other words, Kimi's reports are analytically rich but insufficiently grounded: it generates insightful interpretations that are not always backed by verifiable evidence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Real-world research tasks may involve heterogeneous evidence originating from both external web resources and task-provided attachments.

    “real-world research tasks may involve heterogeneous evidence originating from both external web resources and task-provided attachments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    An automated filter removes all queries containing privacy-sensitive content.

    “an automated filter removes all queries containing privacy-sensitive content”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    All data handling follows strict confidentiality protocols.

    “Throughout the entire pipeline, all data handling follows strict confidentiality protocols”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Robustness experiments and a human ranking study (Kendall's τ = 0.91) confirm the reliability of the evaluation framework.

    “Extensive robustness experiments and a human ranking study (Kendall's τ = 0.91) further confirm the reliability of the evaluation framework.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisenone given

    A system may produce a superficially strong report through redundant exploration or brittle reasoning, while another system may follow a more disciplined and informative process whose intermediate results are only partially reflected in the final write-up.

    “A system may produce a superficially strong report through redundant exploration or brittle reasoning, while another system may follow a more disciplined and informative process whose intermediate results are only partially reflected in the final write-up.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The joint condition on all three signals provides robustness against boundary cases where any single indicator may be unreliable.

    “The joint condition on all three signals provides robustness against boundary cases where any single indicator may be unreliable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Direct evaluation on the original process text is unstable and difficult to interpret.

    “direct evaluation on the original text is unstable and difficult to interpret.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Both sources achieve substantial inter-annotator agreement (κ > 0.74) and precision above 90%.

    “both sources achieve substantial inter-annotator agreement (κ > 0.74) and precision above 90%.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All reports were collected in March 2026 within a controlled time window to ensure fair comparison across systems.

    “All reports were collected in March 2026 within a controlled time window to ensure fair comparison across systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Extensive robustness experiments together with a human ranking study (Kendall's τ = 0.91) validate the reliability of the evaluation framework.

    “and extensive robustness experiments together with a human ranking study (Kendall's τ = 0.91) validate the reliability of the evaluation framework.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Re-running the primary GPT judge three times on the MultiModal setting yields Overall standard deviations of only 0.3 to 0.6 across systems, with identical rankings in every run.

    “First, re-running the primary GPT judge three times on the MultiModal setting yields Overall standard deviations of only 0.3 to 0.6 across systems, with identical rankings in every run.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    A human study with 5 expert annotators ranking 10 systems on 5 sampled queries found that the top three systems (MiroThinker-H1, OpenAI Deep Research, MiroThinker-1.7) match exactly, and the largest rank shift is only 2 positions.

    “We further validate against human judgment through a study with 5 expert annotators ranking 10 systems on 5 sampled queries: the top three systems (MiroThinker-H1, OpenAI Deep Research, MiroThinker-1.7) match exactly, and the largest rank shift is only 2 positions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human annotators' rankings are a valid gold standard for evaluating system performance.

  • May combine two claims.

    Interpretationown data

    MiroThinker-H1 shows the smallest overall decline at 3.0 points, suggesting stronger multimodal integration in its research process.

    “Notably, MiroThinker-H1 shows the smallest overall decline at 3.0 points, suggesting stronger multimodal integration in its research process”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Modifying the judge prompt produces Overall shifts of less than 2 points with no rank changes.

    “Third, modifying the judge prompt produces Overall shifts of less than 2 points with no rank changes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The ranking stability validates that auto-generated queries provide a reasonable proxy for real-world difficulty, supporting the scalability of automated benchmark construction.

    “First, the ranking stability validates that autogenerated queries provide a reasonable proxy for real-world difficulty, supporting the scalability of automated benchmark construction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The factuality evaluation currently identifies cross-source conflicts but does not yet resolve them: the CONFLICT label flags disagreements without determining which source is correct.

    “Additionally, the factuality evaluation currently identifies cross-source conflicts (e.g., between web evidence and user-provided attachments) but does not yet resolve them: the CONFLICT label flags disagreements without determining which source is correct”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Current research processes contain substantial redundancy, including repeated queries, circular exploration paths, and retrieved information that is never utilized.

    “This indicates that current research processes contain substantial redundancy, including repeated queries, circular exploration paths, and retrieved information that is never utilized”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    MiroEval addresses these limitations along four axes: adaptive synthesis quality assessment with dynamically generated task-specific rubrics, agentic factuality verification against both web and attachment evidence, process-centric evaluation, and grounding all tasks in real user needs through a dual-path pipeline that supports continuous refresh.

    “MiroEval addresses these limitations along four axes.For evaluation, it introduces adaptive synthesis quality assessment with dynamically generated task-specific rubrics, agentic factuality verification against both web and attachment evidence, and process-centric evaluation that audits how the system searches, reasons, and refines throughout its investigation.All three layers natively support”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Substituting Gemini as an alternative judge on the Text-Only setting inflates absolute scores by 13 to 17 points on Overall, yet the system ranking is perfectly preserved (Kendall's τ = 1.0).

    “Second, substituting Gemini as an alternative judge on the Text-Only setting inflates absolute scores by 13 to 17 points on Overall, yet the system ranking is perfectly preserved (Kendall's τ = 1.0).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The routing mechanism ensures balanced coverage across the final benchmark.

    “During query curation ( §2.2), an LLM assigns a subset of these features to each query, and the routing mechanism ensures balanced coverage across the final benchmark.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 8 evaluation features and 6 rewrite strategies adequately cover the space of deep research capabilities and query types.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Across existing lines of work, several common limitations persist: evaluation criteria tend to be fixed and task-agnostic, factual verification is often restricted to cited statements or limited evidence scopes, assessment focuses exclusively on the final output without examining the underlying research process, multimodal evaluation rarely goes beyond short-form QA, and benchmark tasks are rarely

    “Across all of these lines of work, several common limitations persist: evaluation criteria tend to be fixed and task-agnostic, factual verification is often restricted to cited statements or limited evidence scopes, assessment focuses exclusively on the final output without examining the underlying research process, multimodal evaluation rarely goes beyond short-form QA, and benchmark tasks are”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The multimodal bottleneck lies in report generation (particularly specificity and coverage of visual content) and research process quality (particularly analytical depth), not in factual verification.

    “the multimodal bottleneck lies in report generation (particularly specificity and coverage of visual content) and research process quality (particularly analytical depth), not in factual verification.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The key-facts extraction step transforms the abstract instruction into precise checkpoints, enabling fine-grained discrimination.

    “The key-facts extraction step transforms the abstract instruction into precise checkpoints (e.g.,”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Process quality reliably predicts overall outcome while revealing weaknesses invisible to output-level metrics.

    “that process quality reliably predicts overall outcome while revealing weaknesses invisible to output-level metrics”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

  • Classified as probably not a substantial claim.

    Premisecitation

    Deep research systems have emerged as a distinct paradigm in which language model agents autonomously plan multi-step web investigations, synthesize evidence across heterogeneous sources, and generate structured, citation-grounded reports.

    “Deep research systems have emerged as a distinct paradigm in which language model agents autonomously plan multi-step web investigations, synthesize evidence across heterogeneous sources, and generate structured, citation-grounded reports [8-10, 12, 32].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [8]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Absolute score differences between judge models reflect systematic calibration offsets rather than meaningful disagreements about relative system quality.

    “These results confirm that the comparative conclusions drawn in the main text are robust to the evaluation configuration, and that absolute score differences between judge models reflect systematic calibration offsets rather than meaningful disagreements about relative system quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Together, these two cases illustrate the value of process-centric evaluation beyond final-report scoring.

    “Together, these two cases illustrate the value of process-centric evaluation beyond final-report scoring.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The core distinction is therefore not simply whether a model can retrieve climate analogies, but whether it uses the attachment to define the scenario constraints early and preserves that grounding throughout the report.

    “The core distinction is therefore not simply whether a model can retrieve climate analogies, but whether it uses the attachment to define the scenario constraints early and preserves that grounding throughout the report.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Although 'nearly 400%' logically satisfies the lower-bound claim 'over 200%,' the statement materially understates the reported magnitude and thus misrepresents the underlying fact.

    “Although”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Cross-judge differences are systematic rather than selective, affecting all systems similarly and leaving comparative conclusions intact.

    “This demonstrates that cross-judge differences are systematic rather than selective, affecting all systems similarly and leaving comparative conclusions intact.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Since both sources are credible yet inconsistent, and the difference cannot be explained by rounding, the claim cannot be definitively verified without further clarification (e.g., model variants or specification updates).

    “Since both sources are credible yet inconsistent, and the difference cannot be explained by rounding, the claim cannot be definitively verified without further clarification (e.g., model variants or specification updates).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Both the attached technical review and LiuGong's official product page are credible sources.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This case shows why multimodal process evaluation cannot be reduced to output inspection alone.

    “This case shows why multimodal process evaluation cannot be reduced to output inspection alone.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The discrepancy between 350 kWh and 432 kWh indicates conflicting specifications across credible sources, likely due to different model variants or updates.

    “The discrepancy between 350 kWh and 432 kWh indicates conflicting specifications across credible sources, likely due to different model variants or updates.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Both the attached technical review and LiuGong's official product page are credible sources.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The model judgment in Correct Case 1 is correct.

    “The model judgment is correct.Although the qualitative explanation is accurate, the claim that prices fell”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The multimodal case reveals a different kind of process difference.

    “The multimodal case reveals a different kind of process difference.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This attachment-grounded constraint becomes the starting point for the later investigation.

    “This attachment-grounded constraint becomes the starting point for the later investigation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    This case highlights a common multimodal conflict scenario: consistent evidence within each source but irreconcilable differences across sources.

    “This case highlights a common multimodal conflict scenario: consistent evidence within each source but irreconcilable differences across sources.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    H1 does not achieve the widest search breadth, but it performs better on efficiency and alignment because it converts retrieved evidence into a bounded conclusion.

    “H1 does not achieve the widest search breadth, but it performs better on efficiency and alignment because it converts retrieved evidence into a bounded conclusion.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    This case illustrates a distinctive multimodal verification workflow: the system first extracts numeric data from an embedded image, then independently confirms the same figures via an authoritative external document, and finally validates internal arithmetic consistency-a three-layer cross-check that is not possible in text-only settings.

    “This case illustrates a distinctive multimodal verification workflow: the system first extracts numeric data from an embedded image (a screenshot of the financial highlights table), then independently confirms the same figures via an authoritative external document, and finally validates internal arithmetic consistency-a three-layer cross-check that is not possible in text-only settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Process quality remains a reliable predictor of overall outcome, validating process-centric evaluation as a meaningful complement to output-level assessment.

    “Despite these weaknesses, process quality remains a reliable predictor of overall outcome, validating process-centric evaluation as a meaningful complement to output-level assessment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected case studies are representative of typical system behavior.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The statement that Star Raiders was released in 1979 is factually valid under a broader definition of release.

    “Therefore, the statement is factually valid under a broader definition of release.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Star Raiders was available in 1979 as a pack-in title for the Atari 400/800 launch.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This case illustrates a common hallucination pattern: correct directional reasoning but incorrect magnitude or timing.

    “This case illustrates a common hallucination pattern: correct directional reasoning but incorrect magnitude or timing.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The selected case studies are representative of typical system behavior.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This case highlights a temporal ambiguity: different sources refer to different release events, and the agent fails to reconcile these interpretations.

    “This case highlights a temporal ambiguity: different sources refer to different release events, and the agent fails to reconcile these interpretations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    In both cases, the dynamically generated criteria provide task-specific discrimination, while the shared fixed dimensions maintain cross-task comparability.

    “In both cases, the dynamically generated criteria provide task-specific discrimination, while the shared fixed dimensions maintain cross-task comparability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected case studies are representative of typical system behavior.

  • May combine two claims.

    Interpretationreasoning

    Strong multimodal research processes are characterized by early attachment integration, evidence digestion over repeated search, and tighter traceability between intermediate findings and final conclusions.

    “Strong multimodal research processes are characterized by early attachment integration, evidence digestion over repeated search, and tighter traceability between intermediate findings and final conclusions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    In the text-only case, the decisive factor is disciplined scope control under fragmented evidence; in the multimodal case, it is whether the attachment becomes a first-class constraint in the research trajectory.

    “In the text-only case, the decisive factor is disciplined scope control under fragmented evidence; in the multimodal case, it is whether the attachment becomes a first-class constraint in the research trajectory.”
    Quote in the paperSubstantial claim
146 of 146

  • The paper relies on this without support.

    The internal testing phase queries reflect real user needs.

    “inspired by real user query patterns obtained during a closed internal testing phase”

  • The authors state this as an assumption.

    Effective benchmark queries should expose the limitations of parametric knowledge.

    “effective benchmark queries should expose the limitations of parametric knowledge”

  • The authors state this as an assumption.

    Process quality is an important evaluation target.

    “process quality is itself an important evaluation target”

  • The authors state this as an assumption.

    A high-quality final report is only one facet of a successful investigation.

    “a high-quality final report is only one facet of a successful investigation”

  • The paper relies on this without support.

    The intermediate reasoning traces exposed by systems faithfully and completely represent their research process.

    “Our process evaluation relies on systems exposing their intermediate reasoning traces, which limits applicability to fully closed-source systems that do not provide such access.”
    “Whether this reflects implicit reasoning, hallucination, or unlogged intermediate steps, the practical implication is the same: current deep research systems exhibit a significant traceability gap that undermines the auditability of their outputs.”

  • The paper relies on this without support.

    The 8 evaluation features and 6 rewrite strategies adequately cover the space of deep research capabilities and query types.

    “Table 8 defines the 8 evaluation features used to classify and balance the benchmark queries.Each feature corresponds to a core capability of deep research systems.”
    “Table 9 describes the 6 rewrite strategies (A through F) spanning three difficulty tiers.”

  • The paper relies on this without support.

    The 12 topics and 36 subtopics are sufficient for trend-grounded automated query generation.

    “Table 10 lists the 12 topics and 36 subtopics used for trend-grounded automated query generation ( §2.3).”

  • The paper relies on this without support.

    The 11 canonical domain categories are appropriate for labeling benchmark queries.

    “Each query in the benchmark is assigned a domain label from the 11 canonical categories listed in Table 11.”

  • The paper relies on this without support.

    The human annotators' rankings are a valid gold standard for evaluating system performance.

    “We further validate against human judgment through a study with 5 expert annotators ranking 10 systems on 5 sampled queries”

  • The paper relies on this without support.

    LLM-based evaluation can exhibit non-trivial variance across runs due to sampling randomness.

    “LLM-based evaluation can exhibit non-trivial variance across runs due to sampling randomness.”

  • The paper relies on this without support.

    The selected case studies are representative of typical system behavior.

    “We present two representative case studies to illustrate how the adaptive synthesis quality evaluation operates on concrete tasks”

  • The paper relies on this without support.

    Star Raiders was available in 1979 as a pack-in title for the Atari 400/800 launch.

    “While the retrieved evidence emphasizes the 1980 publication date, Star Raiders was already available in 1979 as a pack-in title for the Atari 400/800 launch.”

  • The paper relies on this without support.

    Both the attached technical review and LiuGong's official product page are credible sources.

    “Since both sources are credible yet inconsistent, and the difference cannot be explained by rounding, the claim cannot be definitively verified without further clarification (e.g., model variants or specification updates).”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MiroThinker-H1 efficiency score differs from Table 5

    Major

    The process analysis gives H1 an Efficiency score of 68.1, while the Text-Only row of Table 5 gives it 71.0.

  • One synthesis score varies by more than one point across runs

    Minor

    The claimed under-one-point variation does not hold for MiroThinker-H1: its synthesis scores range from 70.8 to 72.0, a 1.2-point span.

  • Benchmark domain count changes from 12 to 11

    Minor

    The benchmark overview says queries span 12 domains, but the appendix says they are assigned labels from 11 canonical categories.

  • Fully automated path includes manual selection

    Minor

    The auto-generated query path is described as fully automated, but its construction table lists a manual selection stage.

  • Reported agreement does not exceed 0.74 for both sources

    Minor

    The text says agreement exceeds 0.74 for both sources, whereas Table 2 reports κ = 0.74 for non-triviality in the auto-generated source.

  • Grok's multimodal overall score rises rather than falls

    Minor

    The cross-setting summary says overall scores decrease by 3 to 10 points across systems, but Table 3 shows Grok rising from 60.2 to 60.5.

6 of 6

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. Yuwen Du, Rui Ye, Shuo Tang, Xinyu Zhu, Yijun Lu, Yuzhu Cai, and Siheng Chen. OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data, 2026. URL https: //arxiv.org/abs/2603.15594.

Cited by

Sorted by citations

Authors
Ye, F., Hu, Y., Zhu, P., Li, Y., Jin, Z., Xiao, Y., Wang, Y., Wang, L., Zhang, Z., Wang, L., Deng, Y., Wang, B., Zhang, Y., Su, L., Wang, X., Zhao, H., Wei, C., Ren, Q., Hooi, B., Bo, A., Yan, S., & Bing, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.28407
arXiv
2603.28407
OpenAlex
W7147119765
Semantic Scholar
26818f2cfdfd076c373bee20a13d44d56d524f0b
Cited by
0
References
0
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
deep research agents, benchmarking, multimodal assessment, evolving knowledge, large language models, human evaluation of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review