CoChat
Sign Up
Cited by 851Multi-Agent Systems and NegotiationAI-based Problem Solving and PlanningNatural Language Processing Techniques

Large Language Model Based Multi-agents: A Survey of Progress and Challenges

Open access
Guo, Taicheng, Chen, Xiuying, Wang, Yaqi, Chang, Ruidi, Pei, Shichao, Chawla, Nitesh V., Wiest, Olaf, Zhang, Xiangliang·2024·Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2024/890
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide array of tasks. Due to their notable capabilities in planning and reasoning, LLMs have been utilized as autonomous agents for the automatic execution of various tasks. Recently, LLM-based agent systems have rapidly evolved from single-agent planning or decision-making to operating as multi-agent systems, enhancing their ability in complex problem-solving and world simulation. To offer an overview of this dynamic field, we present this survey to offer an in-depth discussion on the essential aspects and challenges of LLM-based multi-agent (LLM-MA) systems. Our objective is to provide readers with an in-depth understanding of these key points: the domains and settings where LLM-MA systems operate or simulate; the profiling and communication methods of these agents; and the means by which these agents develop their skills. For those interested in delving into this field, we also summarize the commonly used datasets or benchmarks. To keep researchers updated on the latest studies, we maintain an open-source GitHub repository (github.com/taichengguo/LLM_MultiAgents_Survey_Papers), dedicated to outlining the research of LLM-MA research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24963/ijcai.2024/890

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Machine psychology can reveal patterns of social behavior and interaction among LLMs, individually and collectively, for problem solving or world simulation.

    “In sum, machine psychology can reveal patterns of social behavior and interaction among LLMs, individually and collectively, be it for problem solving or world simulation (Guo et al., 2024).”

    From Machine Psychology (2023) · cites this paper as Guo et al., 2024

    The Psychology passage (Sandbox section; page unavailable) describes using psychological theories and experiments to analyze individual agents’ behavior and emergent collective patterns in simulations. Section 4.1 describes agents interacting to solve problems, but does not say that psychological analysis reveals their social behavior in that setting. The claim extends the psychology point from world simulation to problem solving. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 61

  1. Agashe et al., 2023] Saaket Agashe, Yue Fan, and Xin Eric Wang. Evaluating multi-agent coordination abilities in large language models. arXiv preprint arXiv:2310.03903, 2023. · arXiv
  2. Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371. PMLR, 2023.
  3. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXiv:2305.16867, 2023. · arXiv
  4. Chan et al., 2023] Chi-Min Chan, Weize Chen, Yusheng Su, Jianx- uan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi- agent debate. arXiv preprint arXiv:2308.07201, 2023. · arXiv
  5. Chen et al., 2023a] Guangyao Chen, Siwei Dong, et al. Autoa- gents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288, 2023. · arXiv
  6. Chen et al., 2023b] Huaben Chen, Wenkang Ji, et al. Multi-agent consensus seeking via large language models. arXiv preprint arXiv:2310.20151, 2023. · arXiv
  7. Chen et al., 2023c] Weize Chen, Yusheng Su, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent be- haviors in agents. arXiv preprint arXiv:2308.10848, 2023. · arXiv
  8. Chen et al., 2023d] Yongchao Chen, Arkin, et al. Scalable multi- robot collaboration with large language models: Centralized or decentralized systems? arXiv preprint arXiv:2309.15943, 2023. · arXiv
  9. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhi- fang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. · arXiv
  10. Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590, 2023. · arXiv
  11. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and rea- soning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. · arXiv
  12. Fan et al., 2023] Caoyun Fan, Jindou Chen, et al. Can large lan- guage models serve as rational players in game theory? a sys- tematic analysis. arXiv preprint arXiv:2312.05488, 2023. · arXiv
  13. J. Doyne Farmer and Robert L. Axtell. Agent-Based Modeling in Economics and Finance: Past, Present, and Future. INET Oxford Working Papers 2022-10, June 2022.
  14. Gao et al., 2023a] Chen Gao, Xiaochong Lan, et al. S 3 : Social-network simulation system with large language model- empowered agents. arXiv preprint arXiv:2307.14984, 2023. · arXiv
  15. Gao et al., 2023b] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangx- iang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language mod- els: A survey. arXiv preprint arXiv:2312.10997, 2023. · arXiv
  16. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strate- gies. Transactions of the Association for Computational Linguis- tics, 9:346-361, 2021.
  17. Dan Hendrycks, Collin Burns, et al. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. · arXiv
  18. Sirui Hong, Xiawu Zheng, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. · arXiv
  19. Hua et al., 2023] Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model- based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227, 2023. · arXiv
  20. Huang et al., 2023a] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023. · arXiv
  21. Huang et al., 2023b] Lei Huang, Weijiang Yu, et al. A sur- vey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. · arXiv
  22. Zhao Kaiya, Michelangelo Naim, et al. Lyfe agents: Generative agents for low-cost real-time social interac- tions. arXiv preprint arXiv:2310.02172, 2023. · arXiv
  23. Tushar Khot, Harsh Trivedi, Matthew Fin- layson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. Decomposed prompting: A modular approach for solv- ing complex tasks. arXiv preprint arXiv:2210.02406, 2022. · arXiv
  24. Kovač et al., 2023] Kovač, Portelas, et al. The socialai school: In- sights from developmental psychology towards artificial socio- cultural agents. arXiv preprint arXiv:2307.07871, 2023. · arXiv
  25. Patrick Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  26. Lex and Schedl, 2022] Elisabeth Lex and Markus Schedl. Psychology-informed recommender systems: A human-centric perspective on recommender systems. CHIIR '22. ACM, 2022.
  27. Li et al., 2023a] Chao Li, Xing Su, et al. Quantifying the impact of large language models on collective opinion dynamics. arXiv preprint arXiv:2308.03313, 2023. · arXiv
  28. Li et al., 2023b] Guohao Li, Hammoud, et al. Camel: Commu- nicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023. · arXiv
  29. Li et al., 2023c] Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310.10701, 2023. · arXiv
  30. Li et al., 2023d] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool- augmented llms. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  31. Li et al., 2023e] Nian Li, Chen Gao, Yong Li, and Qingmin Liao. Large language model-empowered agents for simulating macroe- conomic activities. arXiv preprint arXiv:2310.10436, 2023. · arXiv
  32. Li et al., 2023f] Siyu Li, Jin Yang, et al. Are you in a masquer- ade? exploring the behavior and impact of large language model driven social bots in online social networks. arXiv preprint arXiv:2307.10337, 2023. · arXiv
  33. Li et al., 2023g] Yang Li, Yangyang Yu, Haohang Li, Zhi Chen, and Khaldoun Khashanah. Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance. arXiv preprint arXiv:2309.03736, 2023. · arXiv
  34. Li et al., 2023h] Yuan Li, Yixuan Zhang, et al. Metaagents: Simulating interactions of human behaviors for llm-based task- oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500, 2023. · arXiv
  35. Light et al., 2023a] Jonathan Light, Min Cai, et al. From text to tactic: Evaluating llms playing the game of avalon. arXiv preprint arXiv:2310.05036, 2023. · arXiv
  36. Light et al., 2023b] Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon. In NeurIPS 2023 Foundation Models for Decision Mak- ing Workshop, 2023.
  37. Zijun Liu, Yanzhe Zhang, et al. Dynamic llm- agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023. · arXiv
  38. Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. arXiv preprint arXiv:2307.04738, 2023. · arXiv
  39. Mao et al., 2023] Shaoguang Mao, Yuzhe Cai, et al. Alympics: Language agents meet game theory. arXiv preprint arXiv:2311.03220, 2023. · arXiv
  40. Gabriel Mukobi, Hannah Erlebach, et al. Welfare diplomacy: Benchmarking language model cooperation. arXiv preprint arXiv:2310.08901, 2023. · arXiv
  41. Nathalia Nascimento, Paulo Alencar, and Donald Cowan. Self-adaptive large language model (llm)-based multiagent systems. In ACSOS-C, pages 104-109. IEEE, 2023.
  42. Navid, Majumdar, et al. Generative agent- based modeling: Unveiling social system dynamics through cou- pling mechanistic models with generative artificial intelligence. arXiv preprint arXiv:2309.11456, 2023. · arXiv
  43. Joon Sung Park, Lindsay Popowski, et al. Social simulacra: Creating populated prototypes for social computing systems. In UIST, pages 1-18, 2022.
  44. Joon Sung Park, Joseph C O'Brien, et al. Gen- erative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. · arXiv
  45. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023. · arXiv
  46. Ruan et al., 2023] Jingqing Ruan, Yihong Chen, Bin Zhang, Zhi- wei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, et al. Tptu: large language model- based ai agents for task planning and tool usage. arXiv preprint arXiv:2308.03427, 2023. · arXiv
  47. Theodore R Sumers, Shunyu Yao, et al. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. · arXiv
  48. Wang et al., 2023c] Shenzhi Wang, Chang Liu, et al. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320, 2023. · arXiv
  49. Ross Williams, Niyousha Hosseinichimeh, et al. Epidemic modeling with generative agents. arXiv preprint arXiv:2307.04986, 2023. · arXiv
  50. Qingyun Wu, Gagan Bansal, et al. Autogen: En- abling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. · arXiv
  51. Bushi Xiao, Ziyuan Yin, et al. Simulating public administration crisis: A novel generative agent-based simulation system to lower technology barriers in social science research. arXiv preprint arXiv:2311.06957, 2023. · arXiv
  52. Tianbao Xie, Fan Zhou, et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310.10634, 2023. · arXiv
  53. Xu et al., 2023b] Yuzhuang Xu, Shuo Wang, et al. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023. · arXiv
  54. Xu et al., 2023c] Zelai Xu, Chao Yu, et al. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023. · arXiv
  55. Zhang et al., 2023a] An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. arXiv preprint arXiv:2310.10108, 2023. · arXiv
  56. Zhang et al., 2023b] Ceyao Zhang, Kaijie Yang, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint arXiv:2308.11339, 2023. · arXiv
  57. Zhang et al., 2023c] Hongxin Zhang, Weihua Du, et al. Build- ing cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485, 2023. · arXiv
  58. Zhiling Zheng, Oufan Zhang, et al. Chatgpt research group for optimizing the crystallinity of mofs and cofs. ACS Central Science, 9(11):2161-2170, 2023.
  59. Zhou et al., 2023a] Wangchunshu Zhou, Yuchen Eleanor Jiang, et al. Agents: An open-source framework for autonomous lan- guage agents. arXiv preprint arXiv:2309.07870, 2023. · arXiv
  60. Zhou et al., 2023b] Xuhui Zhou, Hao Zhu, Leena Mathur, Ruo- hong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667, 2023. · arXiv
  61. Caleb Ziems, Omar Shaikh, et al. Can large language models transform computational social science? Com- putational Linguistics, pages 1-53, 2023.

Cited by

Sorted by citations

Authors
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., & Zhang, X.
Affiliations
  • University of Notre Dame (US)
  • King Abdullah University of Science and Technology (SA)
  • Southern University of Science and Technology (CN)
  • Independent Researcher
  • University of Massachusetts Boston (US)
ORCID
Venue
Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2024-08
Pages
8048-8057
DOI
10.24963/ijcai.2024/890
OpenAlex
W4401023556
Cited by
851
References
0
Citations by year
2026: 442 · 2025: 336 · 2024: 71 · 2023: 1 · 2019: 1
FWCI
204.57
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2202693)
  • Division of Chemistry (CHE- 2202693, 2202693)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, multi-agent systems, LLM agents, autonomous agents, agent communication, world simulation, multi-agent planning, reasoning, benchmark dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review