CoChat
Sign Up
PreprintCited by 4Stock Market Forecasting MethodsFinancial Markets and Investment StrategiesFinTech, Crowdfunding, Digital Finance

When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents

Open access
Qian, Lingfei, Peng, Xueqing, Wang, Yan, Zhang, Vincent Jim, He, Huan, Smith, Hanley, Han, Yi, He, Yueru +9 more·2025·arXiv
DOI 10.48550/arxiv.2510.11695arXiv 2510.11695
Open this paper in CoChat — free accountOpen PDF

Abstract

Although Large Language Model (LLM)-based agents are increasingly used in financial trading, it remains unclear whether they can reason and adapt in live markets, as most studies test models instead of agents, cover limited periods and assets, and rely on unverified data. To address these gaps, we introduce Agent Market Arena (AMA), the first lifelong, real-time benchmark for evaluating LLM-based trading agents across multiple markets. AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework, enabling fair and continuous comparison under real conditions. It implements four agents, including InvestorAgent as a single-agent baseline, TradeAgent and HedgeFundAgent with different risk styles, and DeepFundAgent with memory-based reasoning, and evaluates them across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash. Live experiments on both cryptocurrency and stock markets demonstrate that agent frameworks display markedly distinct behavioral patterns, spanning from aggressive risk-taking to conservative decision-making, whereas model backbones contribute less to outcome variation. AMA thus establishes a foundation for rigorous, reproducible, and continuously evolving evaluation of financial reasoning and trading intelligence in LLM-based agents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.11695

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • None of the cited works has access to the user-to-agent-to-execution trace at population scale.

    “None of these has access to the user-to-agent-to-execution trace at population scale, which is the gap this record fills.”

    From What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets (2026) · cites this paper as [5]

    The Introduction describes logging agents’ trading decisions and outcomes, and §3.2 describes how trade proposals proceed toward execution. Neither passage addresses user-to-agent-to-execution traces at population scale or establishes that none of the cited works has access to them. I searched for the trace and for user interactions and population-scale execution logs. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • FinQA: A Dataset of Numerical Reasoning over Financial Data

    Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258 · cited 1× · see citing works

    “Early financial NLP benchmarks [5, 6, 22, 33, 38, 42] such as Fin-Ben [41] expanded evaluation beyond sentiment and QA to include retrieval augmented tasks and multi-step decision-making problems, yet they remained largely static and English-centric.”

Flagged first, then by citations

Research works printed but not linked: 37

  1. Eugene Msizi Buthelezi. 2025. Cryptocurrency responses to us monetary policy shocks: a data-driven exploration of price and volatility patterns. The American Economist 70, 1 (2025), 94-119.
  2. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv:2210.03849 (2022). · arXiv
  3. Casey Dougal, Joseph Engelberg, Diego Garcia, and Christopher A Parsons. 2012. Journalists and the stock market. The Review of Financial Studies 25, 3 (2012), 639-679.
  4. Matthew Gentzkow and Jesse M Shapiro. 2006. Media bias and reputation. Journal of political Economy 114, 2 (2006), 280-316.
  5. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024). · arXiv
  6. Cuiyun Han, Jinchuan Zhang, Xinyu Li, Guojin Xu, Weihua Peng, and Zengfeng Zeng. 2022. Duee-fin: A large-scale dataset for document-level event extraction. In CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 172-183.
  7. John Hull. 2012. Risk management and financial institutions,+ Web Site. Vol. 733. John Wiley & Sons.
  8. John C. Hull. 2018. Options, Futures, and Other Derivatives (10 ed.). Pearson Education.
  9. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). · arXiv
  10. Gilles Jacobs and Veronique Hoste. 2020. Extracting fine-grained economic events from business news. In Proceedings of the 1st joint workshop on financial narrative processing and multiling financial summarisation. 235-245.
  11. Changlun Li, Yao Shi, Chen Wang, Qiqi Duan, Runke Ruan, Weijie Huang, Haonan Long, Lijun Huang, Yuyu Luo, and Nan Tang. 2025. Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking. arXiv preprint arXiv:2505.11065 (2025). · arXiv
  12. Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, et al. 2024. Investorbench: A benchmark for financial decision-making tasks with llm-based agent. arXiv preprint arXiv:2412.18174 (2024). · arXiv
  13. Zhenglong Li, Vincent Tam, and Kwan L Yeung. 2024. Developing a multi-agent and self-adaptive framework with deep reinforcement learning for dynamic portfolio risk management. arXiv preprint arXiv:2402.00515 (2024). · arXiv
  14. Chin Yang Lin and João Alexandre Lobo Marques. 2024. Stock market prediction using artificial intelligence: A systematic review of systematic reviews. Social Sciences & Humanities Open 9 (2024), 100864.
  15. Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Pro- dromos Malakasiotis, Ion Androutsopoulos, and Georgios Paliouras. 2022. FiNER: Financial numeric entity recognition for XBRL tagging. arXiv preprint arXiv:2203.06482 (2022). · arXiv
  16. Malik Magdon-Ismail and Amir F Atiya. 2004. Maximum drawdown. Risk Magazine 17, 10 (2004), 99-102.
  17. Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782-796.
  18. Dimitrios K Nasiopoulos, Konstantinos I Roumeliotis, Damianos P Sakas, Kanellos Toudas, and Panagiotis Reklitis. 2025. Financial Sentiment Analysis and Classifi- cation: A Comparative Study of Fine-Tuned Deep Learning Models. International Journal of Financial Studies 13, 2 (2025), 75.
  19. Antonio Pagliaro. 2025. Artificial intelligence vs. efficient markets: A critical reassessment of predictive models in the big data era. Electronics 14, 9 (2025), 1721.
  20. Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Gian- nouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, and Sophia Ananiadou. 2025. Plutus: Benchmarking large language models in low- resource greek finance. arXiv preprint arXiv:2502.18772 (2025). · arXiv
  21. Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Jeff Zhao, Huan He, Yi Han, et al. 2025. MultiFinBen: A Multilin- gual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation. arXiv preprint arXiv:2506.14028 (2025). · arXiv
  22. Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Jimin Huang, and Qian- qian Xie. 2025. Fino1: On the transferability of reasoning enhanced llms to finance. arXiv e-prints (2025), arXiv-2502.
  23. Scott Richardson, Richard Sloan, and Haifeng You. 2012. What makes stock prices move? Fundamentals vs. investor recognition. Financial Analysts Journal 68, 2 (2012), 30-50.
  24. Soumya Sharma, Tapas Nayak, Arusarka Bose, Ajay Kumar Meena, Koustuv Dasgupta, Niloy Ganguly, and Pawan Goyal. 2022. FinRED: A dataset for relation extraction in financial domain. In Companion Proceedings of the Web Conference 2022. 595-597.
  25. William F. Sharpe. 1966. Mutual Fund Performance. Journal of Business 39, 1 (1966), 119-138. doi:10.1086/294846 · DOI · Paper page
  26. William F Sharpe. 1998. The sharpe ratio. Streetwise-the Best of the Journal of Portfolio Management 3, 3 (1998), 169-85.
  27. Paul C Tetlock. 2007. Giving content to investor sentiment: The role of media in the stock market. The Journal of finance 62, 3 (2007), 1139-1168.
  28. Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, and Qianqian Xie. 2025. FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information. arXiv preprint arXiv:2505.20650 (2025). · arXiv
  29. Mengxi Xiao, Zihao Jiang, Lingfei Qian, Zhengyu Chen, Yueru He, Yijing Xu, Yuecheng Jiang, Dong Li, Ruey-Ling Weng, Min Peng, et al. 2025. Retrieval- augmented large language models for financial time series forecasting. arXiv preprint arXiv:2502.05878 (2025). · arXiv
  30. Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. 2024. TradingAgents: Multi- agents LLM financial trading framework. arXiv preprint arXiv:2412.20138 (2024). · arXiv
  31. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems 37 (2024), 95716-95743.
  32. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A comprehensive benchmark, instruc- tion dataset and large language model for finance. Advances in Neural Information Processing Systems 36 (2023), 33469-33484.
  33. Guojun Xiong, Zhiyang Deng, Keyi Wang, Yupeng Cao, Haohang Li, Yangyang Yu, Xueqing Peng, Mingquan Lin, Kaleb E Smith, Xiao-Yang Liu, et al. 2025. FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading. arXiv preprint arXiv:2502.11433 (2025). · arXiv
  34. Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Jordan W Suchow, Denghui Zhang, and Khaldoun Khashanah. 2025. Finmem: A performance- enhanced llm trading agent with layered memory and character design. IEEE Transactions on Big Data (2025).
  35. Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. 2024. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making. Advances in Neural Information Processing Systems 37 (2024), 137010-137045.
  36. Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. 2023. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint arXiv:2308.09975 (2023). · arXiv
  37. Xiao Zhang, Ruoyu Xiang, Chenhan Yuan, Duanyu Feng, Weiguang Han, Ale- jandro Lopez-Lira, Xiao-Yang Liu, Meikang Qiu, Sophia Ananiadou, Min Peng, et al. 2024. Dólares or dollars? unraveling the bilingual prowess of financial llms between spanish and english. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6236-6246.

Cited by

Sorted by citations

Authors
Qian, L., Peng, X., Wang, Y., Zhang, V. J., He, H., Smith, H., Han, Y., He, Y., Li, H., Cao, Y., Yu, Y., Lopez-Lira, A., Lu, P., Nie, J.-Y., Xiong, G., Huang, J., & Ananiadou, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2510.11695
arXiv
2510.11695
OpenAlex
W7090795100
Cited by
4
References
1
Citations by year
2026: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
  • Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
  • Social Sciences › Business, Management and Accounting › Management Information Systems › FinTech, Crowdfunding, Digital Finance
Keywords
risk-taking, memory-based reasoning, cryptocurrency trading, stock trading, financial reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review