CoChat
Sign Up
PreprintCited by 82Multi-Agent Systems and NegotiationArtificial Intelligence in GamesChild and Animal Learning Development

The DOI matches its DataCite record.

Theory of Mind for Multi-Agent Collaboration via Large Language Models

Open access
Li, Huao, Chong, Yu Quan, Stepputtis, Simon, Campbell, Joseph, Hughes, Dana, Lewis, Michael, Sycara, Katia·2023·arXiv
DOI 10.48550/arxiv.2310.10701arXiv 2310.10701
Open this paper in CoChat — free accountOpen PDF

Abstract

While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities in multi-agent collaborations remains largely unexplored. This study evaluates LLM-based agents in a multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi-Agent Reinforcement Learning (MARL) and planning-based baselines. We observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limitations in LLM-based agents' planning optimization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit belief state representations to mitigate these issues, finding that it enhances task performance and the accuracy of ToM inferences for LLM-based agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.10701

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Embedding shared representations of the environment and others' intentions within LLM agents improves coordination and reduces miscommunication.

    “Embedding shared representations of the environment and others' intentions within LLM agents has been shown to improve coordination and reduce miscommunication [72,73].”

    From Adaptive Information Modulation: Designing Governance Mechanisms for Multi-Agent Artificial Intelligence Systems (2026) · cites this paper as [73]

    Section 6.4.2 reports that explicit belief states about the task environment reduced invalid actions and improved team efficiency; Section 7 says they helped agents track information and improve team performance. Section 6.5 examines inference about teammates’ mental states, but the paper does not show that embedding shared representations of others’ intentions reduced miscommunication. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 32

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, et al. 2022. Do as i can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691. · arXiv
  2. Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):0064.
  3. Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. · arXiv
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
  5. Eean R Crawford and Jeffery A Lepine. 2013. A con- figural theory of team processes: Accounting for the structure of taskwork and teamwork. Academy of Management Review, 38(1):32-48.
  6. Xiaocong Fan and John Yen. 2004. Modeling and simu- lating human teamwork behaviors using intelligent agents. Physics of life reviews, 1(3):173-201.
  7. Thilo Hagendorff. 2023. Machine psychology: Inves- tigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303.13988. · arXiv
  8. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 9118-9147. PMLR.
  9. Boaz Keysar, Shuhong Lin, and Dale J Barr. 2003. Limits on theory of mind use in adults. Cognition, 89(1):25-41.
  10. Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models. arXiv preprint arXiv:2302.02083. · arXiv
  11. Huao Li, Ini Oguntola, Dana Hughes, Michael Lewis, and Katia Sycara. 2022. Theory of mind modeling in search and rescue teams. In 2022 31st IEEE Interna- tional Conference on Robot and Human Interactive Communication (RO-MAN), pages 483-489. IEEE.
  12. Terence X Lim, Sidney Tio, and Desmond C Ong. 2020. Improving multi-agent cooperation using theory of mind. arXiv preprint arXiv:2007.15703. · arXiv
  13. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023. Llm+ p: Empowering large language mod- els with optimal planning proficiency. arXiv preprint arXiv:2304.11477. · arXiv
  14. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fe- dorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627. · arXiv
  15. Scott A Miller. 2009. Children's understanding of second-order mental states. Psychological bulletin, 135(5):749.
  16. Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490. · arXiv
  17. Ini Oguntola, Joseph Campbell, Simon Stepputtis, and Katia Sycara. 2023. Theory of mind as intrinsic motivation for multi-agent reinforcement learning. arXiv preprint arXiv:2307.01158. · arXiv
  18. Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. · arXiv
  19. Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, 568(7753):477-486.
  20. Christoph Riedl, Young Ji Kim, Pranav Gupta, Thomas W Malone, and Anita Williams Woolley. 2021. Quantifying collective intelligence in human groups. Proceedings of the National Academy of Sciences, 118(21):e2005737118.
  21. Mikayel Samvelyan, Tabish Rashid, Chris- tian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. 2019. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043. · arXiv
  22. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding lan- guage models'(lack of) theory of mind: A plug-and- play multi-character belief tracker. arXiv preprint arXiv:2306.00924. · arXiv
  23. Peter Sunehag, Guy Lever, Audrunas Gruslys, Woj- ciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296. · arXiv
  24. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.08239. · arXiv
  25. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399. · arXiv
  26. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. · arXiv
  27. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and se- lect: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560. · arXiv
  28. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
  29. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128. · arXiv
  30. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi- agent games. Advances in Neural Information Pro- cessing Systems, 35:24611-24624.
  31. Luyao Yuan, Zipeng Fu, Linqi Zhou, Kexin Yang, and Song-Chun Zhu. 2021. Emergence of theory of mind collaboration in multiagent systems. arXiv preprint arXiv:2110.00121. · arXiv
  32. Jun Zhang, Trey Hedden, and Adrian Chia. 2012. Perspective-taking and depth of theory-of-mind rea- soning in sequential-move games. Cognitive science, 36(3):560-573.

Cited by

Sorted by citations

Metadata

Authors
Li, H., Chong, Y. Q., Stepputtis, S., Campbell, J., Hughes, D., Lewis, M., & Sycara, K.
Affiliations
  • University of Pittsburgh (US)
  • Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
180-192
DOI
10.48550/arxiv.2310.10701
arXiv
2310.10701
OpenAlex
W4389523767
Cited by
82
References
0
Citations by year
2026: 28 · 2025: 41 · 2024: 12 · 2023: 1
FWCI
14.53
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Defense Advanced Research Projects Agency (HR001120C0036)
  • Air Force Office of Scientific Research (FA9550-, FA9550-18-1-0097)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
Keywords
theory of mind, agent cooperation, large language models, LLM agents, long-horizon planning, hallucinations, emergence of cooperation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review