ICBCBench: An Industry Consortium Benchmark for Financial Deep Research

Classified as probably not a substantial claim. May combine two claims.

Objective tasks assess retrieval and reasoning accuracy through verifiable answers, while subjective tasks evaluate analytical depth, logical coherence, citation grounding, and presentation quality.

“Objective tasks assess retrieval and reasoning accuracy through verifiable answers, while subjective tasks evaluate analytical depth, logical coherence, citation grounding, and presentation quality.”

Classified as probably not a substantial claim. May combine two claims.

Objective questions with verifiable answers are evaluated for factual correctness and confidence calibration, while subjective reports are assessed through an expert-aligned framework combining task-specific rubrics, citation consistency checking, and source quality verification.

“Objective questions with verifiable answers are evaluated for factual correctness and confidence calibration, while subjective reports are assessed through an expert-aligned framework combining task-specific rubrics, citation consistency checking, and source quality verification.”

• Also stated in: § Solver System Prompt (Objective Tasks) · plain

  “Objective questions with verifiable answers are evaluated for factual correctness and confidence calibration, while subjective reports are assessed through an expert-aligned framework combining task-specific rubrics, citation consistency checking, and source quality verification.”

• AssumptionRests on an assumption: Rubric-grounded prompting reduces bias, inconsistency, and hallucination in LLM-based evaluation.

May combine two claims.

The public set includes 60 English tasks targeting global markets and 60 Chinese tasks targeting domestic scenarios.

“The public set includes 60 English tasks targeting global markets and 60 Chinese tasks targeting domestic scenarios, supporting reproducible evaluation and leaderboard construction.”

May combine two claims.

The raw enterprise queries have highly uneven quality and many are unsuitable for direct use as benchmark tasks.

“Although these queries reflect real-world user needs, their quality is highly uneven and many are unsuitable for direct use as benchmark tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ICBCBench involves over 50 analysts and researchers across more than 40 organizations, spanning banks, securities firms, and asset managers.

“involving over 50 analysts and researchers across more than 40 organizations, spanning banks, securities firms, and asset managers.”

May combine two claims.

The benchmark comprises 120 tasks, including 60 in English targeting global markets and 60 in Chinese targeting domestic scenarios.

“The benchmark comprises 120 tasks, including 60 in English targeting global markets and 60 in Chinese targeting domestic scenarios.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Objective questions were designed with support from 15 well-trained students in economics and finance.

“During task construction, subjective report-based tasks were developed by over 50 researchers and analysts from financial institutions worldwide, most with over three years of domain experience, while objective questions were designed with support from 15 well-trained students in economics and finance.”

May combine two claims.

ICBCBench is a consortium-driven benchmark for financial deep research, developed with over 50 experts across more than 40 organizations.

“We introduce ICBCBench, a consortium-driven benchmark for financial deep research, developed in collaboration with domain experts from a broad range of financial institutions and academia, involving over 50 experts across more than 40 organizations.”

May combine two claims.

Enterprise dialogue tasks collect raw user queries from internal DeepResearch applications within financial institutions, with N = 6,376, spanning domains such as capital markets, banking, and insurance.

“We collect raw user queries Q = {q i } N i=1 from internal DeepResearch applications within financial institutions, where N = 6,376, spanning domains such as capital markets, banking, and insurance.”

May combine two claims.

ICBCBench contains 180 tasks in total, including 120 publicly released tasks and a private hold-out set of 60 tasks for hidden evaluation.

“ICBCBench contains 180 tasks in total, including 120 publicly released tasks and a private hold-out set of 60 tasks for hidden evaluation.”

May combine two claims.

ICBCBench is built through industry-academia collaboration, integrating enterprise dialogues, financial reports, and expert-designed tasks into a dual-track benchmark.

“ICBCBench is built through industry-academia collaboration, integrating enterprise dialogues, financial reports, and expert-designed tasks into a dual-track benchmark.”

• Also stated in: § Solver System Prompt (Objective Tasks) · plain

  “ICBCBench is built through industry-academia collaboration, integrating enterprise dialogues, financial reports, and expert-designed tasks into a dual-track benchmark.”

Classified as probably not a substantial claim. May combine two claims.

Subjective tasks are designed and validated by experienced industry professionals to reflect real-world research practice, while objective tasks are constructed with support from well-trained students in economics and finance.

“Subjective tasks are designed and validated by experienced industry professionals to reflect real-world research practice, while objective tasks are constructed with support from well-trained students in economics and finance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Most participants have at least 10 years of combined academic and professional experience in finance-related fields.

“Most participants have at least 10 years of combined academic and professional experience in finance-related fields.”

May combine two claims.

The complete benchmark spans 4 primary financial domains and 34 secondary sub-domains, while the public subset covers 27 sub-domains.

“The complete benchmark spans 4 primary financial domains and 34 secondary sub-domains, while the public subset covers 27 sub-domains.”

May combine two claims.

OpenAI-o3-deep-research achieves the highest EN Subjective score (71.84), but its ZH performance is limited by a much lower Objective score (32.50).

“For example, OpenAI-o3-deep-research achieves the highest EN Subjective score (71.84), but its ZH performance is limited by a much lower Objective score (32.50).”

Classified as probably not a substantial claim.

The difficulty of each task is determined by a combination of the number of information sources required, the number of tools involved, and the complexity of the reasoning process.

“The difficulty of each task is determined by a combination of the number of information sources required, the number of tools involved, and the complexity of the reasoning process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Figure 6 reveals a systemic difficulty gap in precise financial data extraction (Objective) compared to narrative report synthesis (Subjective).

“The stark color contrast reveals a systemic difficulty gap in precise financial data extraction (Objective) compared to narrative report synthesis (Subjective).”

May combine two claims.

Objective task construction collects over 20,000 financial research reports as primary source materials and provides them to all task designers.

“To ensure that objective questions are grounded in real-world financial research needs and remain verifiable, we first collect over 20,000 financial research reports as the primary source materials and provide them to all task designers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MiroThinker generated outputs occasionally used inconsistent formatting, alternating between Markdown and LaTeX-style structures without a unified output standard.

“MiroThinker.Generated outputs occasionally used inconsistent formatting, alternating between Markdown and LaTeX-style structures without a unified output standard.”

Classified as probably not a substantial claim.

ICBCBench adopts a dual-track evaluation methodology tailored to financial deep research.

“ICBCBench adopts a dual-track evaluation methodology tailored to financial deep research.”

May combine two claims.

DeerFlow showed several execution instability issues, including occasional ordering inconsistencies when merging multiple Markdown files, frequent tool invocations that could exceed preset limits, and report-style outputs for objective question-answering tasks.

“DeerFlow showed several execution instability issues, including occasional ordering inconsistencies when merging multiple Markdown files, frequent tool invocations that could exceed preset limits, and report-style outputs for objective question-answering tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

High-concurrency API calls occasionally led to failed or incomplete responses, requiring retries or manual filtering during data collection.

“High-concurrency API calls occasionally led to failed or incomplete responses, requiring retries or manual filtering during data collection.”

May combine two claims.

Level 3 tasks account for over 70% of the tasks, while Levels 1 and 2 together comprise no more than 30%.

“Based on these capabilities, we categorize task difficulty into three levels: Level 1 (Easy), Level 2 (Medium), and Level 3 (Hard), with Level 3 accounting for over 70% of the tasks, while Levels 1 and 2 together comprise no more than 30%.”

May combine two claims.

While many models achieve Subjective scores above 50.00, only a few systems, mostly open agentic frameworks, exceed 50.00 on Objective tasks.

“While many models achieve Subjective scores above 50.00,only a few systems, mostly open agentic frameworks, exceed 50.00 on Objective tasks.”

May combine two claims.

In binary decision-making, LLM judges agree with human experts in 72.9% of pairwise comparisons (PAR=0.729), outperforming the natural agreement rate among humans themselves.

“Furthermore, in binary decision-making, the LLM judges agree with human experts in 72.9% of pairwise comparisons (PAR=0.729),outperforming the natural agreement rate among humans themselves, suggesting that our Expert Rubrics effectively distill complex financial reasoning into reproducible machine directives, enabling LLMs to serve as highly reliable comparative judges.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

This yielded a high-quality dataset of 36 evaluation samples from 25 experts across 15 questions (5 English, 10 Chinese), each validated by at least two experts with strong consensus.

“This yielded a high-quality dataset of 36 evaluation samples from 25 experts across 15 questions (5 English, 10 Chinese), each validated by at least two experts with strong consensus.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

OpenClaw was generally more stable in tool usage, with around 10 tool invocations per task, but occasionally suffered from long response times.

“OpenClaw was generally more stable in tool usage, with around 10 tool invocations per task, but occasionally suffered from long response times.”

May combine two claims.

Expert-LLM alignment matches and even slightly exceeds this human baseline.

“Remarkably, the Expert-LLM alignment matches and even slightly exceeds this human baseline.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

Figure 4a illustrates robust alignment across diverse tasks, yielding an overarching Expert-LLM ρ of 0.643.

“Figure 4a illustrates this robust alignment across diverse tasks, yielding an overarching Expert-LLM ρ of 0.643.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

Classified as probably not a substantial claim. May combine two claims.

Financial deep research tasks are highly heterogeneous, and different analytical tasks often require different combinations of tools.

“As shown in Figure 2, financial deep research tasks are highly heterogeneous, and different analytical tasks often require different combinations of tools.”

May combine two claims.

Open-agentic frameworks can bring substantial performance gains and, in several cases, outperform monolithic closed-source systems.

“Table 1 shows that open-agentic frameworks (e.g., OpenClaw [27], DeerFlow [4]) can bring substantial performance gains and, in several cases, outperform monolithic closed-source systems.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Subjective reports are assessed through an expert-aligned framework combining rubric-based evaluation, citation consistency, and source quality verification.

“Subjective reports are assessed through an expert-aligned framework combining rubric-based evaluation, citation consistency, and source quality verification.”

Classified as probably not a substantial claim.

ICBCBench assesses source quality along two dimensions: authority and timeliness.

“We assess source quality along two dimensions: authority and timeliness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For each subjective task, a task-specific rubric is designed following a 100-point scale, typically consisting of 4-6 high-level dimensions and 12-16 fine-grained sub-dimensions.

“For each subjective task, we design a task-specific rubric tailored to the corresponding financial report type.Each rubric follows a 100-point scale and typically consists of 4-6 high-level dimensions and 12-16 fine-grained sub-dimensions, yielding an expert score S expert ∈ [0, 100] by aggregating scores across all dimensions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A dedicated platform manages the entire construction process, including task submission, review, revision, and acceptance tracking.

“To improve coordination and efficiency, we develop a dedicated platform to manage the entire construction process, including task submission, review, revision, and acceptance tracking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each expert was asked to select and score up to 5 questions strictly within their domain of expertise.

“Each expert was asked to select and score up to 5 questions strictly within their domain of expertise.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

Table 1 shows that open agentic frameworks are highly competitive with, and in several cases outperform, closed-source Deep Research systems.

“Table 1 shows that open agentic frameworks are highly competitive with, and in several cases outperform, closed-source Deep Research systems.”

Classified as probably not a substantial claim.

During evaluation, Gemini-3.1-Pro-Preview is used as the LLM judge and instructed to score each report according to the task-specific rubric.

“During evaluation, Gemini-3.1-Pro-Preview[12] is used as the LLM judge and instructed to score each report according to the task-specific rubric, with justifications grounded in the corresponding sub-dimensions.”

• cites [12]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The final subjective score assigns 80% weight to expert-aligned rubric evaluation and 10% each to citation consistency and source quality.

“The final subjective score assigns 80% weight to expert-aligned rubric evaluation and 10% each to citation consistency and source quality:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some queries failed to produce coherent final reports; in a few cases, models such as Kimi-k2.5-thinking returned intermediate tool-calling traces or planning prompts rather than final answers.

“Some queries failed to produce coherent final reports.In a few cases, models such as Kimi-k2.5-thinkingreturned intermediate tool-calling traces or planning prompts rather than final answers.”

May combine two claims.

Among closed-source systems, Gemini-deep-research is the most robust, achieving the second-highest overall score in both EN and ZH tracks.

“Among closed-source systems, Gemini-deep-research is the most robust, achieving the second-highest overall score in both EN and ZH tracks.”

May combine two claims.

Human experts establish an empirical consensus ceiling with an inter-expert Spearman's ρ of 0.638 and a Pairwise Agreement Rate (PAR) of 0.711.

“As detailed in Table 2, human experts establish an empirical consensus ceiling with an inter-expert ρ of 0.638 and a PAR of 0.711.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

Classified as probably not a substantial claim.

ICBCBench evaluates calibration using Root Mean Square Calibration Error (RMSCE).

“We evaluate calibration using Root Mean Square Calibration Error (RMSCE), which measures the discrepancy between predicted confidence and empirical accuracy:”

• AssumptionRests on an assumption: Reliable uncertainty estimation is critical in financial research because overconfident incorrect predictions may lead to high-risk decisions.

Classified as probably not a substantial claim. May combine two claims.

Subjective reports are evaluated through three components: task-specific expert rubrics as the primary measure of analytical quality, and citation consistency and source quality as auxiliary signals for factual grounding, traceability, and reliability.

“Subjective reports are evaluated through three components: task-specific expert rubrics as the primary measure of analytical quality, and citation consistency and source quality as auxiliary signals for factual grounding, traceability, and reliability.”

Classified as probably not a substantial claim. May combine two claims.

Objective tasks are scored by answer correctness against verifiable ground-truth answers, with confidence calibration reported as an auxiliary metric.

“Objective tasks are scored by answer correctness against verifiable ground-truth answers, with confidence calibration reported as an auxiliary metric.”

May combine two claims.

MiroThinker ties for the highest EN objective score (52.50), yet its subjective score (53.15) significantly trails peers like OpenClaw and DeerFlow.

“MiroThinker ties for the highest EN objective score (52.50), yet its subjective score (53.15) significantly trails peers like OpenClaw and DeerFlow.”

May combine two claims.

In the Global (EN) scenario, DeerFlow(+GPT-5.5) achieves the highest overall score (58.67), followed by Gemini-deep-research (57.38).

“In the Global (EN) scenario, DeerFlow(+GPT-5.5)achieves the highest overall score (58.67), followed by Gemini-deep-research (57.38).”

May combine two claims.

Reports generated by five representative DeepResearch Agents across 60 subjective questions were sampled and distributed to more than 30 financial experts from various institutions, the majority of whom were analysts or researchers with over three years of industry experience.

“To address this, we sampled reports generated by five representative DeepResearch Agents across our 60 subjective questions.These were distributed to more than 30 financial experts from various institutions, the majority of whom were analysts or researchers with over three years of industry experience.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Enterprise dialogue queries often exhibit three common deficiencies: colloquial expression, overly broad scope, and lack of constraints.

“Enterprise dialogue queries often exhibit three common deficiencies: (1) colloquial expression, (2) overly broad scope, and (3) lack of constraints.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Figure 3 suggests that frameworks may alter the cross-lingual behavior of their backbone models.

“Notably, Figure 3 suggests that frameworks may alter the cross-lingual behavior of their backbone models.”

May combine two claims.

Open-agentic frameworks remain highly competitive on objective tasks, with OpenClaw(+DeepSeek-V4-Pro) achieving the highest Overall score in the Global (EN) scenario and OpenClaw(+GPT-5.5) leading in the Chinese (ZH) scenario.

“Open-agentic frameworks remain highly competitive, particularly on objective tasks, with OpenClaw(+DeepSeek-V4-Pro) achieving the highest Overall score in the Global (EN) scenario and OpenClaw(+GPT-5.5)leading in the Chinese (ZH) scenario.”

May combine two claims.

Evaluation on the unreleased private hold-out set broadly corroborates the main findings while revealing non-trivial differences from the public benchmark.

“Evaluation on the unreleased private hold-out set (Table 12) broadly corroborates the main findings while revealing non-trivial differences from the public benchmark.”

May combine two claims.

Deep Research products such as Grok-3-deepsearch, Qwen-deep-research, and Tongyi-deepresearch-30b-a3b suffer from extreme calibration errors exceeding 80%, indicating that their confidence scores are essentially uncorrelated with correctness.

“Deep Research products such as Grok-3-deepsearch, Qwen-deep-research, and Tongyi-deepresearch-30b-a3b suffer from extreme calibration errors exceeding 80%, indicating that their confidence scores are essentially uncorrelated with correctness.”

May combine two claims.

Only six systems fall within the ideal zone, demonstrating strong factual reasoning and well-calibrated confidence.

“Only six systems fall within the ideal zone, demonstrating not only strong factual reasoning but also well-calibrated confidence”

May combine two claims.

OpenClaw(+GPT-5.5) presents the clearest Chinese-oriented pattern, improving from 54.80 in EN to 63.38 in ZH.

“OpenClaw(+GPT-5.5)presents the clearest Chinese-oriented pattern, improving from 54.80 in EN to 63.38 in ZH”

May combine two claims.

Many models exhibit positive localization gaps (∆), indicating stronger performance on the Global (EN) track, whereas several open agentic frameworks show negative gaps and stronger adaptation to Chinese financial scenarios.

“Many models exhibit positive localization gaps (∆), indicating stronger performance on the Global (EN) track, whereas several open agentic frameworks show negative gaps and stronger adaptation to Chinese financial scenarios.”

May combine two claims.

Gemini-deep-research drops from 61.76 to 52.50 in the ZH track, and DeerFlow(+GPT-5.5) declines from 57.14 to 52.50 in the EN track.

“Gemini-deep-research drops from 61.76 to 52.50 in the ZH track, and DeerFlow(+GPT-5.5)declines from 57.14 to 52.50 in the EN track.”

Probably not a substantial claim: may not pass S4 (based).

Existing benchmarks focus on either closed-ended question answering or open-ended report evaluation, failing to jointly capture retrieval-reasoning accuracy and end-to-end research quality required in real-world workflows.

“Existing benchmarks focus on either closedended question answering or open-ended report evaluation, failing to jointly capture retrieval-reasoning accuracy and end-to-end research quality required in real-world workflows.”

• AssumptionRests on an assumption: Financial research is time-sensitive, knowledge-intensive, and risk-sensitive, requiring reliable sources, rigorous analysis, factual accuracy, and deep reasoning to support high-stakes decision-making.

May combine two claims.

Each subjective task is defined as a concrete research assignment with 2-5 explicit analytical objectives and validated by at least 3 domain experts.

“Each task is defined as a concrete research assignment with 2-5 explicit analytical objectives and validated by at least 3 domain experts, significantly improving clarity, comparability, and alignment with real-world financial analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DeepResearch Bench, ResearchRubrics, LiveResearchBench, DRACO, and related benchmarks evaluate long-form reports through citation accuracy and checklist-based expert assessment.

“Another line of work, represented by DeepResearch Bench, ResearchRubrics, LiveResearchBench, DRACO, and related benchmarks [9,40,37,31,44,36,1], evaluates long-form reports through citation accuracy and checklist-based expert assessment.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [31]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Many prompts in existing financial report benchmarks are overly broad, lack comparability across samples, and weakly reflect practical financial research needs.

“Many prompts are overly broad, lack comparability across samples, and weakly reflect practical financial research needs [9,40,37,36,31] (e.g.,”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [31]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Existing approaches remain insufficient for the financial domain because most benchmarks rely on a single evaluation paradigm.

“Nevertheless, existing approaches remain insufficient for the financial domain.Most benchmarks rely on a single evaluation paradigm, focusing either on objective question answering or subjective report assessment, which captures only a partial view of financial research capability.”

• AssumptionRests on an assumption: Financial research is time-sensitive, knowledge-intensive, and risk-sensitive, requiring reliable sources, rigorous analysis, factual accuracy, and deep reasoning to support high-stakes decision-making.

May combine two claims.

Objective evaluation offers scalable and reproducible measurement of factual correctness and reasoning accuracy, but is limited in assessing open-ended analytical quality.

“Objective evaluation offers scalable and reproducible measurement of factual correctness and reasoning accuracy, but is limited in assessing open-ended analytical quality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Benchmarks such as HLE, GAIA, BrowseComp, DeepSearchQA, and FinSearchComp use short, verifiable closed-ended questions to indirectly assess retrieval and reasoning.

“Benchmarks such as HLE, GAIA, BrowseComp, DeepSearchQA, and FinSearchComp [29,21,38,13,15] use short, verifiable closedended questions to indirectly assess retrieval and reasoning.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

From the three sources, 218 candidate subjective tasks were constructed and each assigned to at least three domain-specific financial experts for review, scoring, and feedback.

“From the three sources above, we construct 218 candidate subjective tasks and assign each task to at least three domain-specific financial experts for review, scoring, and feedback.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ICBCBench establishes finance-specific authoring principles covering accuracy and compliance, domain relevance, task depth and complexity, and scope diversity.

“To further standardize task design, we establish finance-specific authoring principles covering accuracy and compliance, domain relevance, task depth and complexity, and scope diversity, as detailed in Appendix Table 5.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Subjective evaluation better reflects real-world analytical tasks, yet it is harder to standardize and reproduce across evaluators.

“Subjective evaluation better reflects real-world analytical tasks, yet it is harder to standardize and reproduce across evaluators.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

A key advantage of open-agentic workflows lies in customizable skill design, which contrasts with opaque proprietary pipelines.

“A key advantage of open-agentic workflows lies in customizable skill design, which contrasts with opaque proprietary pipelines.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The refined rubrics are reviewed by at least three additional domain experts, and the final versions are consolidated by the organizers.

“The refined rubrics are reviewed by at least three additional domain experts, and the final versions are consolidated by the organizers.”

May combine two claims.

This divergence exposes a systemic 'illusion of competence' within proprietary models: they excel at generating highly structured, authoritative narratives while simultaneously failing at rigorous factual extraction.

“This divergence exposes a systemic”

May combine two claims.

These observations highlight practical challenges in evaluating Deep Research systems, where performance depends not only on model capability but also on orchestration stability, tool-use efficiency, and output-format consistency.

“These observations highlight practical challenges in evaluating Deep Research systems, where performance depends not only on model capability but also on orchestration stability, tool-use efficiency, and output-format consistency.”

Classified as probably not a substantial claim.

Following HLE, ICBCBench adopts an LLM-as-a-judge pipeline for objective evaluation.

“Following HLE [29], we adopt an LLM-as-a-judge pipeline for objective evaluation.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

This weighting prioritizes expert-aligned analytical quality while incorporating citation and source-based checks for factual grounding and reliability.

“This weighting prioritizes expert-aligned analytical quality while incorporating citation and sourcebased checks for factual grounding and reliability.”

May combine two claims.

The Expert Rubrics effectively distill complex financial reasoning into reproducible machine directives, enabling LLMs to serve as highly reliable comparative judges.

“Furthermore, in binary decision-making, the LLM judges agree with human experts in 72.9% of pairwise comparisons (PAR=0.729),outperforming the natural agreement rate among humans themselves, suggesting that our Expert Rubrics effectively distill complex financial reasoning into reproducible machine directives, enabling LLMs to serve as highly reliable comparative judges.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

While frameworks improve tool-use efficiency and workflow orchestration, final analytical performance is still bounded by the capacity of the core model.

“Overall, while frameworks improve tool-use efficiency and workflow orchestration, final analytical performance is still bounded by the capacity of the core model.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The decoupling of these two capabilities underscores a critical cognitive bottleneck in current DR systems: a high subjective score guarantees readability but not reliability, while a high objective score ensures factual correctness but lacks communicative value.

“The decoupling of these two capabilities underscores a critical cognitive bottleneck in current DR systems: a high subjective score guarantees readability but not reliability, while a high objective score ensures factual correctness but lacks communicative value.”

Classified as probably not a substantial claim.

Following DeepResearch Bench, inter-rater reliability was measured using the Intraclass Correlation Coefficient (ICC).

“Following DeepResearch Bench [9], we measure inter-rater reliability using the Intraclass Correlation Coefficient (ICC).”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

Framework-level gains remain constrained by the underlying backbone model; modular orchestration cannot fully compensate for limitations in base model capability.

“However, such framework-level gains remain constrained by the underlying backbone model.The performance gaps between GPT-5.5 and DeepSeek-V4-Pro indicate that modular orchestration cannot fully compensate for limitations in base model capability.”

May combine two claims.

The contrast between inter-expert MAE (15.36) and inter-LLM MAE (5.83) demonstrates that LLMs, guided by structured rubrics, effectively transcend individual human subjectivity and fatigue, providing super-human scoring stability for open-ended financial tasks.

“This significant contrast (15.36 vs. 5.83 pts) convincingly demonstrates that LLMs, guided by our structured rubrics, effectively transcend individual human subjectivity and fatigue.They provide a super-human scoring stability for open-ended financial tasks, free from the scale-drifting often observed in human evaluations.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

May combine two claims.

Raw factual extraction alone is insufficient for financial research; producing an expert-level report demands narrative flow, structured argumentation, and domain-specific stylistic alignment.

“This pattern highlights that raw factual extraction alone is insufficient for financial research.Producing an expert-level report demands narrative flow, structured argumentation, and domain-specific stylistic alignment, demonstrating that objective data extraction and subjective text synthesis represent fundamentally orthogonal dimensions of deep research intelligence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each task receives at least three independent answers, recommendation scores, and feedback comments.

“each task receives at least three independent answers, recommendation scores, and feedback comments.”

Classified as probably not a substantial claim.

Each task is independently solved and reviewed by at least three student annotators.

“each task is independently solved and reviewed by at least three student annotators”

Classified as probably not a substantial claim.

Given the highly subjective nature of financial report evaluation, consistency experiments comparing human experts and LLM judges were designed to validate the effectiveness of the proposed Expert Rubrics.

“Given the highly subjective nature of financial report evaluation, we designed a series of consistency experiments comparing human experts and LLM judges to validate the effectiveness of our proposed Expert Rubrics.”

• AssumptionRests on an assumption: Human expert scores are a valid reference standard for financial report quality.

Classified as probably not a substantial claim. May combine two claims.

To ensure expert alignment, rubrics are drafted by financial experts, refined with GPT-5.4, reviewed by at least three additional domain experts, and finalized by the organizers.

“To ensure expert alignment, rubrics are drafted by financial experts, refined with GPT-5.4 [24], reviewed by at least three additional domain experts, and finalized by the organizers.”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Configurable skill sets allow institutions to modularly embed domain rules and business expertise into data curation, analytical processing, and standardized report generation.

“Configurable skill sets allow institutions to modularly embed domain rules and business expertise into data curation, analytical processing, and standardized report generation.”

May combine two claims.

Proprietary models excel at narrative synthesis but often suffer from an 'illusion of competence' in factual extraction, whereas open-agentic frameworks demonstrate superior objective reasoning.

“Our findings highlight a critical bifurcation in current AI systems: proprietary models excel at narrative synthesis but often suffer from an”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

This flexibility transforms general-purpose LLMs from generic conversational systems into more domain-adaptive specialized systems, making them better aligned with professional analysts' research workflows and reporting conventions.

“This flexibility transforms general-purpose LLMs from generic conversational systems into more domain-adaptive specialized systems, making them better aligned with professional analysts' research workflows and reporting conventions.”

Classified as probably not a substantial claim. May combine two claims.

Following the FACT-style evaluation in DeepResearch Bench, ICBCBench extracts and deduplicates statement-URL pairs from each report and judges whether each cited source supports the corresponding statement.

“Following the FACT-style evaluation in DeepResearch Bench [9], we extract and deduplicate statement-URL pairs from each report and judge whether each cited source supports the corresponding statement.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

ICBCBench assesses analytical depth, practical relevance, factual grounding, and structured reporting quality through task-specific criteria.

“demonstrating how ICBCBench assesses analytical depth, practical relevance, factual grounding, and structured reporting quality through task-specific criteria.”

May combine two claims.

FinRpt focuses primarily on equity research reports with relatively simple evaluation protocols.

“FinRpt focuses primarily on equity research reports with relatively simple evaluation protocols”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The private hold-out set is designed to evaluate model generalization and prevent benchmark overfitting.

“The private set is not publicly released and is designed to evaluate model generalization and prevent benchmark overfitting.”

• AssumptionRests on an assumption: The private hold-out set has not been exposed to the evaluated models.

May combine two claims.

HLE, GAIA, and BrowseComp mainly capture factual correctness and basic reasoning, and are limited in assessing complex analysis and long-form report generation required in real-world scenarios.

“However, they mainly capture factual correctness and basic reasoning, and are limited in assessing complex analysis and long-form report generation required in real-world scenarios.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

HLE, GAIA, and BrowseComp emphasize answer-based evaluation with well-defined ground truth, enabling scalable and objective measurement.

“HLE [29], GAIA [21], and BrowseComp [38] emphasize answer-based evaluation with well-defined ground truth, enabling scalable and objective measurement.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The strong positive correlation between objective and subjective scores demonstrates the systematic reliability and robust bilingual evaluation capabilities of the ICBCBench framework.

“The strong positive correlation demonstrates the systematic reliability and robust bilingual evaluation capabilities of the ICBCBench framework.”

May combine two claims.

DeepResearch Bench, DR.BENCH, LiveResearchBench, and DRACO adopt report-based evaluation with citation-aware metrics and rubric-based assessment.

“Benchmarks such as DeepResearch Bench [9], DR.BENCH [40], LiveResearchBench [37], and DRACO [44] adopt report-based evaluation with citation-aware metrics and rubric-based assessment.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

These approaches either rely on fixed evaluation dimensions or lack flexible, expert-aligned frameworks, limiting their ability to support structured, domain-specific report analysis.

“However, these approaches either rely on fixed evaluation dimensions or lack flexible, expert-aligned frameworks, limiting their ability to support structured, domain-specific report analysis.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Existing benchmarks in finance either rely on a single evaluation paradigm or fail to capture the full complexity of real-world financial research workflows.

“More generally, existing benchmarks in finance either rely on a single evaluation paradigm or fail to capture the full complexity of real-world financial research workflows.”

May combine two claims.

These discrepancies indicate that the private set provides a complementary stress test rather than a direct replication of the public benchmark, helping assess generalization and reduce the risk of benchmark overfitting.

“These discrepancies indicate that the private set provides a complementary stress test rather than a direct replication of the public benchmark, helping assess generalization and reduce the risk of benchmark overfitting.”

May combine two claims.

This degradation highlights multimodal financial reasoning as a key bottleneck, requiring models to extract precise visual evidence and integrate it with textual reasoning.

“This degradation highlights multimodal financial reasoning as a key bottleneck, requiring models to extract precise visual evidence and integrate it with textual reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This study faces limitations regarding the temporal degradation of financial data, the computational overhead of multi-agent workflows, and the inherent difficulty of computationally evaluating contrarian market insights.

“Despite its rigorous design, this study faces limitations regarding the temporal degradation of financial data, the computational overhead of multi-agent workflows, and the inherent difficulty of computationally evaluating contrarian market insights.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All student annotators receive training and are provided with five high-quality annotated examples for reference before formal human solving and review.

“Before formal human solving and review, all student annotators receive training and are provided with five high-quality annotated examples for reference.”

May combine two claims.

FinResearchBench adopts a logic tree-based agent evaluation framework that lacks broad validation from domain experts.

“while FinResearchBench adopts a logic tree-based agent evaluation framework that lacks broad validation from domain experts.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.