Human-level play in the game of Diplomacy by combining language models with strategic reasoning
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy , a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite Meta’s efforts, CICERO turned out to be an expert liar.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [4]
In the citing paper
“CICERO engages in premeditated deception, breaks the deals to which it had agreed, and tells outright falsehoods.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [4]
In the citing paper
“We found numerous examples of deception that were not reported in the published paper.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [4]
In the citing paper
“It not only betrayed other players but also engaged in premeditated deception, planning in advance to build a fake alliance with a human player in order to trick that player into leaving themselves undefended for an attack.”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [4]
In the citing paper
“The CICERO project picked Diplomacy in order to evaluate the abilities of AI systems to learn how to compete in games that involve human cooperation, where the AI cannot simply master the game through running simulations against itself.4”
From AI deception: A survey of examples, risks, and potential solutions (2024) · cites this paper as [4]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Self-citationLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Self-citationNash, J. (1951) · The Annals of Mathematics · doi:10.2307/1969529 · Cited by 7,002
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Zur Theorie der Gesellschaftsspiele
v. Neumann, J. (1928) · Mathematische Annalen · doi:10.1007/bf01448847 · Cited by 2,572
The Weighted Majority Algorithm
Littlestone, N., & Warmuth, M. K. (1994) · Information and Computation · doi:10.1006/inco.1994.1009 · Cited by 1,975
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Self-citationHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Self-citationCampbell, M., Hoane, A. J., & Hsu, F.-H. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00129-1 · Cited by 1,226
A Simple Adaptive Procedure Leading to Correlated Equilibrium
Hart, S., & Mas-Colell, A. (2000) · Econometrica · doi:10.1111/1468-0262.00153 · Cited by 1,088
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896
POMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882
Generating Natural Language Adversarial Examples
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1316 · Cited by 787
DeepStack: Expert-level artificial intelligence in heads-up no-limit poker
Moravčík, M., Schmid, M., Burch, N., Lisý, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., & Bowling, M. (2017) · Science · doi:10.1126/science.aam6960 · Cited by 786
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Superhuman AI for multiplayer poker
Brown, N., & Sandholm, T. (2019) · Science · doi:10.1126/science.aay2400 · Cited by 592
Self-citationSuperhuman AI for heads-up no-limit poker: Libratus beats top professionals
Brown, N., & Sandholm, T. (2018) · Science · doi:10.1126/science.aao1733 · Cited by 540
Self-citationA stochastic model of human-machine interaction for learning dialog strategies
Levin, E., Pieraccini, R., & Eckert, W. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.817450 · Cited by 516
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
What Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., Spillane, B., Gilmartin, E., Murad, C., Munteanu, C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300705 · Cited by 461
Deal or No Deal? End-to-End Learning of Negotiation Dialogues
Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., & Batra, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1259 · Cited by 347
Self-citationSCHATZMANN, J., WEILHAMMER, K., STUTTLE, M., & YOUNG, S. (2006) · The Knowledge Engineering Review · doi:10.1017/s0269888906000944 · Cited by 336
Williams, J. D., Asadi, K., & Zweig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1062 · Cited by 324
ParlAI: A Dialog Research Software Platform
Miller, A., Feng, W., Batra, D., Bordes, A., Fisch, A., Lu, J., Parikh, D., & Weston, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d17-2014 · Cited by 315
Self-citationThe Hanabi challenge: A new frontier for AI research
Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., Parisotto, E., Dumoulin, V., Moitra, S., Hughes, E., et al. (2020) · Artificial Intelligence · doi:10.1016/j.artint.2019.103216 · Cited by 235
Cooperative AI: machines must learn to find common ground
Dafoe, A., Bachrach, Y., Hadfield, G., Horvitz, E., Larson, K., & Graepel, T. (2021) · Nature · doi:10.1038/d41586-021-01170-0 · Cited by 232
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222
Self-citationCombining Deep Reinforcement Learning and Search for Imperfect-Information GamesPreprint
Brown, N., Bakhtin, A., Lerer, A. K., & Gong, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13544 · Cited by 221
Self-citationCollaborating with Humans without Human DataPreprint
Strouse, D., McKee, K. R., Botvinick, M., Hughes, E., & Everett, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08176 · Cited by 196
On the Utility of Learning about Humans for Human-AI Coordination
Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., & Dragan, A. (2019) · arXiv · doi:10.48550/arxiv.1910.05789 · Cited by 168
Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents
Traum, D. R., Marsella, S., Gratch, J. S., Lee, J., & Hartholt, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85483-8_12 · Cited by 162
Decoupling Strategy and Generation in Negotiation Dialogues
He, H., Chen, D., Balakrishnan, A., & Liang, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1256 · Cited by 125
Reinforcement Learning for Adaptive Dialogue Systems
Rieser, V., & Lemon, O. (2011) · Springer eBooks · doi:10.1007/978-3-642-24942-6 · Cited by 117
DESIGNING AND BUILDING A NEGOTIATING AUTOMATED AGENT
Kraus, S., & Lehmann, D. J. (1995) · Computational Intelligence · doi:10.1111/j.1467-8640.1995.tb00026.x · Cited by 116
Reasoning about Pragmatics with Neural Listeners and Speakers
Andreas, J., & Klein, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1125 · Cited by 115
Issues in Multiparty Dialogues
Traum, D. R. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24608-4_12 · Cited by 115
CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems
Chawla, K., Ramirez, J., Clever, R., Lucas, G., May, J., & Gratch, J. S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.254 · Cited by 108
Addressee and Response Selection for Multi-Party Conversation
Ouchi, H., & Tsuboi, Y. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1231 · Cited by 103
Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
Monroe, W., Hawkins, R. X. D., Goodman, N. D., & Potts, C. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00064 · Cited by 94
Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
Shi, W., Wang, X., Oh, Y. J., Zhang, J., Sahay, S., & Yu, Z. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376843 · Cited by 93
Self-citationReal or Fake? Learning to Discriminate Machine from Human Generated Text
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., & Szlam, A. (2019) · arXiv · doi:10.48550/arxiv.1906.03351 · Cited by 77
Self-citationLearning Existing Social Conventions via Observationally Augmented Self-Play
Lerer, A. K., & Peysakhovich, A. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314268 · Cited by 72
Self-citationShuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.27 · Cited by 72
Self-citationLearning to Play No-Press Diplomacy with Best Response Policy IterationPreprint
Anthony, T., Eccles, T., Tacchetti, A., Kramár, J., Gemp, I., Hudson, T. C., Porcel, N., Lanctot, M., Pérolat, J., Everett, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04635 · Cited by 57
No-Press Diplomacy from ScratchPreprint
Bakhtin, A., Wu, D., Lerer, A. K., & Brown, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02924 · Cited by 49
Self-citationEnd-to-End Trainable Non-Collaborative Dialog System
Li, Y., Qian, K., Shi, W., & Yu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6345 · Cited by 48
Self-citationShi, W., Li, Y., Sahay, S., & Yu, Z. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.295 · Cited by 47
Self-citationDynamic Thresholding and Pruning for Regret Minimization
Brown, N., Kroer, C., & Sandholm, T. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10603 · Cited by 47
Self-citationAnticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44
Self-citationKeizer, S., Guhe, M., Cuayáhuitl, H., Efstathiou, I., Engelbrecht, K.-P., Dobre, M., Lascarides, A., & Lemon, O. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2077 · Cited by 40
Negotiation in a non-cooperative environment
Kraus, S., Ephrati, E., & Lehmann, D. J. (1991) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/09528139108915294 · Cited by 37
Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning
Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., et al. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7236699 · Cited by 35
Self-citationNo Press Diplomacy: Modeling Multi-Agent GameplayPreprint
Paquette, P., Lu, Y., Bocco, S., Smith, M., Ortiz-Gagné, S., Kummerfeld, J. K., Singh, S. P., Pineau, J., & Courville, A. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02128 · Cited by 33
It Takes Two to Lie: One to Lie, and One to Listen
Peskov, D., Cheng, B., Elgohary, A., Barrow, J., Danescu-Niculescu-Mizil, C., & Boyd-Graber, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.353 · Cited by 30
The Challenge of Negotiation in the Game of Diplomacy
de Jonge, D., Baarslag, T., Aydoğan, R., Jonker, C. M., Fujita, K., & Itō, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17294-7_8 · Cited by 25
Learning non-cooperative dialogue behaviours
Efstathiou, I., & Lemon, O. (2014) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w14-4308 · Cited by 24
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation
Gupta, P., Tsvetkov, Y., & Bigham, J. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.338 · Cited by 22
DipBlue: A Diplomacy Agent with Strategic and Trust Reasoning
Ferreira, A., Cardoso, H. L., & Reis, L. P. (2015) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0005205400540065 · Cited by 17
Tactical coordination in no-press diplomacy
Johansson, S. J., & Håård, F. (2005) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1082473.1082538 · Cited by 17
Diplomat, an agent in a multi agent environment: An overview
Kraus, S., & Lehmann, D. J. (2003) · International Phoenix Conference on Computers and Communications (IPCCC) · doi:10.1109/pccc.1988.10117 · Cited by 16
Construction and Analysis of a Persuasive Dialogue Corpus
Hiraoka, T., Neubig, G., Sakti, S., Toda, T., & Nakamura, S. (2016) · Signals and communication technology · doi:10.1007/978-3-319-21834-2_12 · Cited by 14
Multitasking Inhibits Semantic Drift
Jacob, A. P., Lewis, M., & Andreas, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.421 · Cited by 9
Self-citationTowards General Cooperative Game Playing
Marinheiro, J., & Cardoso, H. L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-78301-7_8 · Cited by 8
Understanding User Resistance Strategies in Persuasive Conversations
Tian, Y., Shi, W., Li, C., & Yu, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.431 · Cited by 4
Self-citationCovert Implementations of the Turing Test: A More Level Playing Field?
Burden, D. J. H., Savin‐Baden, M., & Bhakta, R. (2016) · Springer eBooks · doi:10.1007/978-3-319-47175-4_13 · Cited by 2
Research works printed but not linked: 16
- A. Bakhtin. Mastering the game of no-press Diplomacy via human-regularized reinforcement learning and planning. arXiv:2210.05492 [cs.GT] (2022). · arXiv
- A. Holtzman J. Buys L. Du M. Forbes Y. Choi The curious case of neural text degeneration 8th International Conference on Learning Representations ICLR 2020 Addis Ababa Ethiopia April 26-30 2020 (OpenReview.net 2020).
- A. Vaswani et al . Attention is all you need in Advances in Neural Information Processing Systems I. Guyon et al. eds. (Curran Associates Inc. 2017) vol. 30.
- A. P. Jacob et al . Modeling strong and human-like gameplay with KL-regularized search in International Conference on Machine Learning (PMLR 2022) pp. 9695–9728.
- Z. Ji. Survey of hallucination in natural language generation. arXiv:2202.03629 [cs.CL] (2022). · arXiv
- C. Berner. Dota 2 with large scale deep reinforcement learning. arXiv:1912.06680 (2019). · arXiv
- N. Carlini et al . Extracting training data from large language models in 30th USENIX Security Symposium (USENIX Security 21) (2021) pp. 2633–2650.
- R. Zellers et al . Defending against neural fake news in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 NeurIPS 2019 December 8–14 2019 Vancouver BC Canada H. M. Wallach et al. eds. (2019) pp. 9051–9062.
- J. Gray A. Lerer A. Bakhtin N. Brown Human-level performance in no-press Diplomacy via equilibrium search in International Conference on Learning Representations (2020).
- D. Yarats M. Lewis Hierarchical text generation and planning for strategic dialogue in International Conference on Machine Learning (PMLR 2018) pp. 5591–5599.
- S. Afantenos et al . Developing a corpus of strategic conversation in The Settlers of Catan Workshop on Games and NLP (GAMNLP-12) (2012).
- H. Cuayáhuitl S. Keizer O. Lemon Strategic dialogue management via deep reinforcement learning in NeurIPS Workshop on Deep Reinforcement Learning (2015).
- A. Radford, Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
- D. P. Kingma J. Ba Adam: A method for stochastic optimization in 3rd International Conference on Learning Representations ICLR 2015 San Diego CA USA May 7-9 2015 Conference Track Proceedings Y. Bengio Y. LeCun eds. (2015).
- E. M. Smith D. Gonzalez-Rico E. Dinan Y. Boureau Controlling style in generated dialogue. arXiv:2009.10855 [cs.CL] (2020). · arXiv
- J. Xu Recipes for safety in open-domain chatbots. arXiv:2010.07079 [cs.CL] (2020). · arXiv
Cited by
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
AppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713600 · Cited by 652
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Metadata
- Authors
- Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., Rowe, D., Shi, W., Spisak, J., Wei, A., Wu, D., Zhang, H., & Zijlstra, M.
- Affiliations
- Meta (United States) (US)
- Harvard University (US)
- Corresponding author
- Bakhtin, Anton
- Brown, Noam
- Dinan, Emily
- Flaherty, Colin
- Gray, Jonathan
- Hu, Hengyuan
- Jacob, Athul Paul
- Lerer, Adam
- Lewis, Mike
- Miller, Alexander H.
- Renduchintala, Adithya
- Shi, Weiyan
- Wu, David
- Zhang, Hugh
- ORCID
- Venue
- Science
- Published
- 2022-12-09
- Volume
- 378
- Issue
- 6624
- Pages
- 1067-1074
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W4309663019
- Cited by
- 241
- References
- 65
- Citations by year
- 2026: 67 · 2025: 69 · 2024: 59 · 2023: 43 · 2022: 2
- FWCI
- 14.24
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Keywords
- diplomacy, AI agents, human-AI interaction, strategic reasoning, language models, belief inference, dialogue generation, planning, reinforcement learning, Cicero
- MeSH terms
- Diplomacy, Artificial Intelligence, Humans, Language, Software, Video Games
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.