CoChat
Sign Up
PreprintCited by 15Multi-Agent Systems and NegotiationSimulation Techniques and ApplicationsPsychiatry, Mental Health, Neuroscience

Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia

Open access
Vezhnevets, Alexander Sasha, Agapiou, John P., Aharon, Avia, Ziv, Ron, Matyas, Jayd, Duéñez-Guzmán, Edgar A., Cunningham, William A., Osindero, Simon +2 more·2023·arXiv
DOI 10.48550/arxiv.2312.03664arXiv 2312.03664
Open this paper in CoChat — free accountOpen PDF

Abstract

Agent-based modeling has been around for decades, and applied widely across the social and natural sciences. The scope of this research method is now poised to grow dramatically as it absorbs the new affordances provided by Large Language Models (LLM)s. Generative Agent-Based Models (GABM) are not just classic Agent-Based Models (ABM)s where the agents talk to one another. Rather, GABMs are constructed using an LLM to apply common sense to situations, act "reasonably", recall common semantic knowledge, produce API calls to control digital technologies like apps, and communicate both within the simulation and to researchers viewing it from the outside. Here we present Concordia, a library to facilitate constructing and working with GABMs. Concordia makes it easy to construct language-mediated simulations of physically- or digitally-grounded environments. Concordia agents produce their behavior using a flexible component system which mediates between two fundamental operations: LLM calls and associative memory retrieval. A special agent called the Game Master (GM), which was inspired by tabletop role-playing games, is responsible for simulating the environment where the agents interact. Agents take actions by describing what they want to do in natural language. The GM then translates their actions into appropriate implementations. In a simulated physical world, the GM checks the physical plausibility of agent actions and describes their effects. In digital environments simulating technologies such as apps and services, the GM may handle API calls to integrate with external tools such as general AI assistants (e.g., Bard, ChatGPT), and digital apps (e.g., Calendar, Email, Search, etc.). Concordia was designed to support a wide array of applications both in scientific research and for evaluating performance of real digital services by simulating users and/or generating synthetic data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.03664

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research shows exploratory empirical studies of emergent behaviour are possible in simple games and can lead to surprising outcomes.

    “Nonetheless, research such as that of Chen et al. (2024d), Park et al. (2023a), and Vezhnevets et al. (2023) shows this to be possible in simple games, and that it can lead to surprising outcomes.”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Vezhnevets et al., 2023

    Section 2.3 (page unavailable) describes experiments that vary agents or game-master settings and observe outcomes; Appendix A.8 gives small illustrative simulations. This supports the possibility of exploring behaviour in simple games, but our copy does not report that these studies produced surprising outcomes. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 15

Research works printed but not linked: 97

  1. A. Adadi and M. Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52160, 2018.
  2. B. Agüera y Arcas. Do large language models understand us? Daedalus, 151(2):183-197, 2022.
  3. G. V. Aher, R. I. Arriaga, and A. T. Kalai. Using large language models to simulate multiple hu- mans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371. PMLR, 2023.
  4. I. Ajzen. The theory of planned behavior. Organi- zational behavior and human decision processes, 50(2):179-211, 1991.
  5. A. Amirova, T. Fteropoulli, N. Ahmed, M. R. Cowie, and J. Z. Leibo. Framework-based qualitative analysis of free responses of large language models: Algorithmic fidelity. arXiv preprint arXiv:2309.06364, 2023. · arXiv
  6. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lep- ikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. PALM 2 technical report. arXiv preprint arXiv:2305.10403, 2023. · arXiv
  7. L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of one, many: Using language models to simulate human sam- ples. Political Analysis, 31(3):337-351, 2023.
  8. A. Baddeley. Working memory. Science, 255 (5044):556-559, 1992.
  9. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  10. L. F. Barrett. Are emotions natural kinds? Per- spectives on psychological science, 1(1):28-58, 2006.
  11. L. F. Barrett. The conceptual act theory: A précis. Emotion review, 6(4):292-297, 2014.
  12. A. Birhane, W. Isaac, V. Prabhakaran, M. Diaz, M. C. Elish, I. Gabriel, and S. Mohamed. Power to the people? opportunities and challenges for participatory AI. Equity and Access in Algo- rithms, Mechanisms, and Optimization, pages 1-8, 2022.
  13. N. Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111.
  14. J. Brand, A. Israeli, and D. Ngwe. Using GPT for market research. Available at SSRN 4395751, 2023.
  15. L. Brinkmann, F. Baumann, J.-F. Bonnefon, M. Derex, T. F. Müller, A.-M. Nussberger, A. Czaplicka, A. Acerbi, T. L. Griffiths, J. Hen- rich, J. Z. Leibo, R. McElreath, P.-Y. Oudeyer, J. Stray, and I. Rahwan. Machine culture. Na- ture Human Behaviour, pages 1-14, 2023.
  16. E. Brooks, L. A. Walls, R. Lewis, and S. Singh. Large language models can implement policy iteration. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023.
  17. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural in- formation processing systems, 33:1877-1901, 2020.
  18. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
  19. J. R. Carter and M. D. Irons. Are economists different, and if so, why? Journal of Economic Perspectives, 5(2):171-177, 1991.
  20. P. Dayan. Goal-directed control and its antipodes. Neural Networks, 22(3):213-219, 2009.
  21. P. Dayan and G. E. Hinton. Feudal reinforcement learning. Advances in neural information pro- cessing systems, 5, 1992.
  22. M. DeLanda. Philosophy and simulation: the emer- gence of synthetic reason. Bloomsbury Publish- ing, 2011.
  23. M. DeLanda. Assemblage theory. Edinburgh Uni- versity Press, 2016.
  24. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022. · arXiv
  25. E. A. Duéñez-Guzmán, S. Sadedin, J. X. Wang, K. R. McKee, and J. Z. Leibo. A social path to human-like artificial intelligence. Nature Machine Intelligence, pages 1-8, 2023.
  26. R. H. Fazio. Multiple processes by which attitudes guide behavior: The mode model as an integra- tive framework. In Advances in experimental social psychology, volume 23, pages 75-109. Elsevier, 1990.
  27. R. H. Frank, T. Gilovich, and D. T. Regan. Does studying economics inhibit cooperation? Jour- nal of economic perspectives, 7(2):159-171, 1993.
  28. A. Goldstein, Z. Zada, E. Buchnik, M. Schain, A. Price, B. Aubrey, S. A. Nastase, A. Feder, D. Emanuel, A. Cohen, A. Jansen, H. Gazula, G. Choe, A. Rao, C. Kim, C. Casto, L. Fanda, W. Doyle, D. Friedman, P. Dugan, L. Melloni, R. Reichart, S. Devore, A. Fliner, L. Hasenfratz, O. Levy, A. Hassidim, M. Brenner, Y. Matias, K. A. Norman, O. Devinsky, and U. Hasson. Shared computational principles for language processing in humans and deep language mod- els. Nature neuroscience, 25(3):369-380, 2022.
  29. A. G. Greenwald and M. R. Banaji. Implicit social cognition: attitudes, self-esteem, and stereo- types. Psychological review, 102(1):4, 1995.
  30. I. Grossmann, M. Feinberg, D. C. Parker, N. A. Christakis, P. E. Tetlock, and W. A. Cunning- ham. AI and the transformation of social sci- ence research. Science, 380(6650):1108-1109, 2023.
  31. G. Gygax and D. Cook. The Dungeon Master Guide, No. 2100, 2nd Edition (Advanced Dungeons and Dragons). TSR, Inc, 1989. ISBN 0880387297.
  32. G. K. Hadfield and B. R. Weingast. Law without the state: legal attributes and the coordination of decentralized collective punishment. Journal of Law and Courts, 1(1):3-34, 2013.
  33. J. A. Harris, R. Boyd, and B. M. Wood. The role of causal knowledge in the evolution of traditional technology. Current Biology, 31(8):1798-1803, 2021.
  34. J. Henrich. The secret of our success: How cul- ture is driving human evolution, domesticating our species, and making us smarter. princeton University press, 2016.
  35. S. Hong, X. Zheng, J. Chen, Y. Cheng, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, et al. MetaGPT: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. · arXiv
  36. J. J. Horton. Large language models as simu- lated economic agents: What can we learn from homo silicus? arXiv e-prints, pages arXiv-2301, 2023.
  37. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embod- ied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. · arXiv
  38. M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beat- tie, N. C. Rabinowitz, A. S. Morcos, A. Ruder- man, N. Sonnerat, T. Green, L. Deason, J. Z. Leibo, D. Silver, D. Hassabis, K. Kavukcuoglu, and T. Graepel. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science, 364(6443): 859-865, 2019.
  39. M. B. Johanson, E. Hughes, F. Timbers, and J. Z. Leibo. Emergent bartering behaviour in multi- agent reinforcement learning. arXiv preprint arXiv:2205.06760, 2022. · arXiv
  40. C. G. Jung. The archetypes and the collective un- conscious. Routledge, 1959.
  41. D. Kahneman, S. Frederick, et al. Representa- tiveness revisited: Attribute substitution in in- tuitive judgment. Heuristics and biases: The psychology of intuitive judgment, 49(49-81):74, 2002.
  42. Z. Kaiya, M. Naim, J. Kondic, M. Cortes, J. Ge, S. Luo, G. R. Yang, and A. Ahn. Lyfe agents: Generative agents for low-cost real-time social interactions. arXiv preprint arXiv:2310.02172, 2023. · arXiv
  43. I. Lakatos. History of science and its rational reconstructions. In PSA: Proceedings of the bien- nial meeting of the philosophy of science associa- tion, volume 1970, pages 91-136. Cambridge University Press, 1970.
  44. J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Pro- ceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464-473, 2017.
  45. J. Z. Leibo, E. Hughes, M. Lanctot, and T. Grae- pel. Autocurricula and the emergence of in- novation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742, 2019. · arXiv
  46. J. Z. Leibo, E. A. Dueñez-Guzman, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel. Scal- able evaluation of multi-agent reinforcement learning with Melting Pot. In International Con- ference on Machine Learning, pages 6187-6199.
  47. PMLR, 2021.
  48. C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, and J. Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 10, 2023a. · arXiv
  49. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. CAMEL: Com- municative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Sys- tems, 2023b.
  50. T. Linzen and M. Baroni. Syntactic structure from deep learning. Annual Review of Linguistics, 7: 195-212, 2021.
  51. M. W. Macy and R. Willer. From factors to ac- tors: Computational sociology and agent-based modeling. Annual review of sociology, 28(1): 143-166, 2002.
  52. J. G. March and J. P. Olsen. The Logic of Appro- priateness. In The Oxford Handbook of Political Science. Oxford University Press, 2011. doi: 10. 1093/oxfordhb/9780199604456.013.0024. · DOI
  53. J. L. McClelland, F. Hill, M. Rudolph, J. Baldridge, and H. Schütze. Placing language in an inte- grated understanding system: Next steps to- ward human-level performance in neural lan- guage models. Proceedings of the National Academy of Sciences, 117(42):25966-25974, 2020.
  54. M. McLuhan. The medium is the message. In Communication theory, pages 390-402.
  55. D. Nettle. Personality: What Makes You the Way You Are. Oxford University Press, 2007. ISBN 978-0199211432.
  56. M. A. Olson and R. H. Fazio. Implicit and ex- plicit measures of attitudes: The perspective of the mode model. In Attitudes, pages 39-84. Psychology Press, 2008.
  57. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  58. J. A. Ouellette and W. Wood. Habit and inten- tion in everyday life: The multiple processes by which past behavior predicts future behavior. Psychological bulletin, 124(1):54, 1998.
  59. J. S. Park, J. C. O'Brien, C. J. Cai, M. R. Mor- ris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behav- ior. arXiv preprint arXiv:2304.03442, 2023. · arXiv
  60. J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel. A multi-agent re- inforcement learning model of common-pool resource appropriation. Advances in neural in- formation processing systems, 30, 2017.
  61. A. R. Poteete, M. A. Janssen, and E. Ostrom. Work- ing together: collective action, the commons, and multiple methods in practice. Princeton Univer- sity Press, 2010.
  62. M. Risse. Political Theory of the Digital Age: Where Artificial Intelligence Might Take Us. Cambridge University Press, 2023.
  63. H. M. Roff. Expected utilitarianism. arXiv preprint arXiv:2008.07321, 2020. · arXiv
  64. M. Roser and M. S. Gazzaniga. Automatic brains-interpretive minds. Current Directions in Psychological Science, 13(2):56-59, 2004.
  65. M. Safdari, G. Serapio-García, C. Crepy, S. Fitz, P. Romero, L. Sun, M. Abdulhai, A. Faust, and M. Matarić. Personality traits in large language models. arXiv preprint arXiv:2307.00184, 2023. · arXiv
  66. S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548, 2023. · arXiv
  67. J. Scheurer, M. Balesni, and M. Hobbhahn. Large language models can strategically deceive their users when put under pressure. arXiv preprint arXiv:2311.07590, 2023. · arXiv
  68. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. · arXiv
  69. C. Schill, J. M. Anderies, T. Lindahl, C. Folke, S. Polasky, J. C. Cárdenas, A.-S. Crépin, M. A. Janssen, J. Norberg, and M. Schlüter. A more dynamic understanding of human behaviour for the anthropocene. Nature Sustainability, 2 (12):1075-1082, 2019.
  70. W. Schneider and R. M. Shiffrin. Controlled and automatic human information processing: I. detection, search, and attention. Psychological review, 84(1):1, 1977.
  71. M. Schrimpf, I. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. Tenenbaum, and E. Fedorenko. Artificial neural networks accu- rately predict language processing in the brain. BioRxiv, pages 2020-06, 2020.
  72. M. Shanahan, K. McDonell, and L. Reynolds. Role play with large language models. Nature, pages 1-6, 2023.
  73. E. R. Smith. Distributed connectionist models in social psychology. Social and Personality Psy- chology Compass, 3(1):64-76, 2009.
  74. C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
  75. C. R. Sunstein. How change happens. MIT Press, 2019.
  76. R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tempo- ral abstraction in reinforcement learning. Arti- ficial intelligence, 112(1-2):181-211, 1999.
  77. L. Tesfatsion. Agent-based computational eco- nomics: Overview and brief history. Artificial Intelligence, Learning and Computation in Eco- nomics and Finance, pages 41-58, 2023.
  78. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Ba- tra, P. Bhargava, S. Bhosale, et al. LLAMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  79. T. Ullman. Large language models fail on triv- ial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. · arXiv
  80. A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Ma- chine Learning, pages 3540-3549. PMLR, 2017.
  81. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jader- berg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wunsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in starcraft II us- ing multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
  82. A. Walker and M. J. Wooldridge. Understanding the emergence of conventions in multi-agent systems. In ICMAS, volume 95, pages 384-389, 1995.
  83. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Pro- cessing Systems, 35:24824-24837, 2022.
  84. K. Weick, K. Sutcliffe, and D. Obstfeld. Organiz- ing and the process of sensemaking. ORGANI- ZATION SCIENCE, 16:409-421, 07 2005. doi: 10.1287/orsc.1050.0133. · DOI · Paper page
  85. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al. Ethical and so- cial risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. · arXiv
  86. L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, and W. Isaac. Sociotechnical safety evalua- tion of generative ai systems. arXiv preprint arXiv:2310.11986, 2023. · arXiv
  87. A. Wendt. Anarchy is what states make of it: the social construction of power politics. Interna- tional organization, 46(2):391-425, 1992.
  88. A. E. Wendt. The agent-structure problem in international relations theory. International organization, 41(3):335-370, 1987.
  89. K. D. Williams, C. K. Cheung, and W. Choi. Cy- berostracism: effects of being ignored over the internet. Journal of personality and social psy- chology, 79(5):748, 2000.
  90. D. S. Wilson, E. Ostrom, and M. E. Cox. General- izing the core design principles for the efficacy of groups. Journal of economic behavior & orga- nization, 90:S21-S32, 2013.
  91. T. D. Wilson, S. Lindsey, and T. Y. Schooler. A model of dual attitudes. Psychological review, 107(1):101, 2000.
  92. P. Windrum, G. Fagiolo, and A. Moneta. Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation, 10(2):8, 2007.
  93. B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, et al. BLOOM: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. · arXiv
  94. J. Yocum, P. Christoffersen, M. Damani, J. Sveg- liato, D. Hadfield-Menell, and S. Russell. Mit- igating generative agent social dilemmas. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
  95. Z. Zada, A. Goldstein, S. Michelmann, E. Simony, A. Price, L. Hasenfratz, E. Barham, A. Zadbood, W. Doyle, D. Friedman, et al. A shared linguis- tic space for transmitting our thoughts from brain to brain in natural conversations. bioRxiv, 2023.
  96. Z. Zhao, W. S. Lee, and D. Hsu. Large lan- guage models as commonsense knowledge for large-scale task planning. arXiv preprint arXiv:2305.14078, 2023. · arXiv
  97. X. Zhou, H. Zhu, L. Mathur, R. Zhang, H. Yu, Z. Qi, L.-P. Morency, Y. Bisk, D. Fried, G. Neubig, et al. SOTOPIA: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667, 2023. · arXiv

Authors
Vezhnevets, A. S., Agapiou, J. P., Aharon, A., Ziv, R., Matyas, J., Duéñez-Guzmán, E. A., Cunningham, W. A., Osindero, S., Karmon, D., & Leibo, J. Z.
Affiliations
  • Google (United States) (US)
  • University of Toronto (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.03664
arXiv
2312.03664
OpenAlex
W4389471395
Cited by
15
References
0
Citations by year
2026: 2 · 2025: 9 · 2024: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
large language models, associative memory retrieval, tabletop role-playing games, API calls, synthetic data generation, user simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review