The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Open accessAbstract
Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
FinQA: A Dataset of Numerical Reasoning over Financial Data
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258 · cited 2×
“( ii ) Financial QA-style benchmarks (e.g., FinQA [5], FinanceQA [15]) pre-collect relevant context and bypass open-domain search and tool use, thereby under-assessing agents’ search competence and diverging from analyst workflows [22, 25] (see Table 1).”
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Zhou, P., Leon, B., Ying, X., Zhang, C., Shao, Y., Ye, Q., Chong, D., Jin, Z., Xie, C., Cao, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.19314 · cited 2×
“…proposed for evaluating browsing capabilities, while they fall short along two axes: ( i ) General-purpose browsing benchmarks, like BrowseComp [23], BrowseComp-ZH [27], and BrowseComp-Plus [7], are intentionally domain-agnostic and center on lookup-oriented tasks with short, verifiable targets.”
Cited by
Sorted by citations
StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?
Chen, Y., Yao, Z., Liu, Y., Xin, A., Ye, J., Yu, J., Hou, L., & Li, J. (2025) · arXiv · Cited by 62
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 770
Self-citationICBCBench: An Industry Consortium Benchmark for Financial Deep Research
Li, W., Tang, Z., He, Y., Wang, C., Feng, L., Sun, X., Liu, D., Wen, Z., Wei, H., Wang, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.17458
“This market-aware design reflects real-world financial applications [15].”
Self-citation
Metadata
- Authors
- Hu, L., Jiao, J., Liu, J., Ren, Y., Wen, Z., Zhang, K., Zhang, X., Gao, X., He, T., Hu, F., Liao, Y., Wang, Z., Yang, C., Yang, Q., Yin, M., Zeng, Z., Zhang, G., Zhang, X., Zhao, X., Zhu, Z., Namkoong, H., Huang, W., & Tang, Y.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.13160
- OpenAlex
- W4415317693
- Cited by
- 2
- References
- 2
- Citations by year
- 2026: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- LLM agents, benchmark, expert annotation, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
