CoChat
Sign Up
PreprintCited by 4Game Theory and ApplicationsArtificial Intelligence in GamesMulti-Agent Systems and Negotiation

ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agents

Open access
Mao, Shaoguang, Cai, Yuzhe, Xia, Yan, Wu, Wenshan, Wang, Xun, Wang, Fengyi, Ge, Tao, Wei, Furu·2023·arXiv
DOI 10.48550/arxiv.2311.03220arXiv 2311.03220
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper introduces Alympics (Olympics for Agents), a systematic simulation framework utilizing Large Language Model (LLM) agents for game theory research. Alympics creates a versatile platform for studying complex game theory problems, bridging the gap between theoretical game theory and empirical investigations by providing a controlled environment for simulating human-like strategic interactions with LLM agents. In our pilot case study, the "Water Allocation Challenge," we explore Alympics through a challenging strategic game focused on the multi-round auction on scarce survival resources. This study demonstrates the framework's ability to qualitatively and quantitatively analyze game determinants, strategies, and outcomes. Additionally, we conduct a comprehensive human assessment and an in-depth evaluation of LLM agents in strategic decision-making scenarios. Our findings not only expand the understanding of LLM agents' proficiency in emulating human strategic behavior but also highlight their potential in advancing game theory knowledge, thereby enriching our understanding of both game theory and empowering further research into strategic decision-making domains with LLM agents. Codes, prompts, and all related resources are available at https://github.com/microsoft/Alympics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.03220

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Sandbox interface is widely used in software development and gaming.

    “This kind of interface is widely used in software development (code interpreter as simulated environment) [Hong et al., 2023], gaming (using game rules as simulated environment) [Mao et al., 2023], etc.”

    From Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024) · cites this paper as Mao et al., 2023

    Section 2.1 (Sandbox Playground; page unavailable) describes a sandbox environment for conducting games, with code defining the game rules. This supports the gaming example attributed to Mao et al., but the paper does not establish that this kind of interface is widely used. The software-development example is attributed to a different citation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • LLM-MA is well-suited for creating simulated gaming environments, enabling controlled, scalable, and dynamic settings that closely mimic human interactions, making it ideal for testing game theory hypotheses.

    “LLM-MA is well-suited for creating simulated gaming environments, allowing agents to assume various roles within games.This technology enables the development of controlled, scalable, and dynamic settings that closely mimic human interactions, making it ideal for testing a range of game theory hypotheses [Mao et al., 2023;Xu et al., 2023b;Gong et al., 2023].”

    From Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024) · cites this paper as Mao et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Cognitive Architectures for Language Agents

    Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · cited 2× · see citing works

    “…et al. , 2023; Hu et al. , 2023]; Reasoning Plugin offers specialized logic or algorithms for complex decision-making processes [Wei et al. , 2022; Yao et al. , 2023]; Persona Setting defines the agent’s behavioral profile and strategic inclinations [Wang et al. , 2023c; Xu et al. , 2023a]; Other…”
    Background

Flagged first, then by citations

Research works printed but not linked: 29

  1. Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234, 2023. · arXiv
  2. Marco Archetti and Kenneth J Pienta. Cooperation among cancer cells: applying game theory to cancer. Nature Re- views Cancer, 19(2):110-117, 2019.
  3. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
  4. Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Ma- jumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execu- tion of llm agents in an auction arena. arXiv preprint arXiv:2310.05746, 2023. · arXiv
  5. I de Zarzà, J de Curtò, Gemma Roig, Pietro Manzoni, and Carlos T Calafate. Emergent cooperation and strategy adaptation in multi-agent systems: An extended coevolu- tionary theory with llms. Electronics, 12(12):2722, 2023.
  6. Fulin Guo. Gpt agents in game theory experiments. arXiv preprint arXiv:2305.05516, 2023. · arXiv
  7. Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901, 2023. · arXiv
  8. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023. · arXiv
  9. Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434, 2023. · arXiv
  10. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Am- manabrolu, Faeze Brahman, Shiyu Huang, Chandra Bha- gavatula, Yejin Choi, and Xiang Ren. Swiftsage: A gener- ative agent with fast and slow thinking for complex inter- active tasks. arXiv preprint arXiv:2305.17390, 2023. · arXiv
  11. Nunzio Lorè and Babak Heydari. Strategic behavior of large language models: Game structure vs. contextual framing. arXiv preprint arXiv:2309.05898, 2023. · arXiv
  12. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Halli- nan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. · arXiv
  13. Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. arXiv preprint arXiv:2304.03442, 2023. · arXiv
  14. Matti Pohjola. Applications of dynamic game theory to macroeconomics. Dynamic games and applications in eco- nomics, pages 103-133, 1986.
  15. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. · arXiv
  16. Alan G Sanfey. Social decision-making: insights from game theory and neuroscience. Science, 318(5850):598-602, 2007.
  17. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weim- ing Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023. · arXiv
  18. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflex- ion: an autonomous agent with dynamic memory and self- reflection. arXiv preprint arXiv:2303.11366, 2023. · arXiv
  19. Yashar Talebirad and Amirhossein Nadiri. Multi-agent col- laboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023. · arXiv
  20. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  21. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023. · arXiv
  22. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive syn- ergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023. · arXiv
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
  24. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expert- prompting: Instructing large language models to be distin- guished experts. arXiv preprint arXiv:2305.14688, 2023. · arXiv
  25. Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020. · arXiv
  26. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. · arXiv
  27. Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collab- oration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124, 2023. · arXiv
  28. Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition behaviors in large language model-based agents. arXiv preprint arXiv:2310.17512, 2023. · arXiv
  29. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhe- hao Zhang, and Diyi Yang. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514, 2023. · arXiv

Cited by

Sorted by citations

Authors
Mao, S., Cai, Y., Xia, Y., Wu, W., Wang, X., Wang, F., Ge, T., & Wei, F.
Affiliations
Microsoft Research Asia (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.03220
arXiv
2311.03220
OpenAlex
W4388482223
Cited by
3
References
1
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
LLM agents, game theory, strategic decision-making, simulation framework, multi-round auctions, water allocation, resource constraints, strategic interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review