CoChat
Sign Up
Cited by 2,187Artificial Intelligence in GamesMulti-Agent Systems and NegotiationSpeech and dialogue systems

Generative Agents: Interactive Simulacra of Human Behavior

Open access
Park, Joon Sung, O'Brien, Joseph, Cai, Carrie Jun, Morris, Meredith Ringel, Liang, Percy, Bernstein, Michael S.·2023·Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
DOI 10.1145/3586183.3606763
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduces generative agents that store experiences in natural language, synthesize reflections, and retrieve memories dynamically for planning and social behavior.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces generative agents powered by an LLM architecture with memory, reflection, and planning, instantiated as 25 characters in a Sims-like sandbox. In controlled and end-to-end evaluations, the full architecture produced more believable behavior than ablations and supported emergent social phenomena such as information spread, relationship formation, and party coordination.

Key findings

  • In a controlled believability ranking with 100 evaluators, the full architecture scored highest (TrueSkill mean 29.89, SD 0.72), ahead of no-reflection (26.88, 0.69), no-reflection/planning (25.64, 0.68), human crowdworker (22.95, 0.69), and fully ablated (21.21, 0.70); Kruskal-Wallis χ²(4)=150.29, p<0.001; all pairwise Dunn differences were significant at p<0.001 except crowdworker vs. fully ablated; Cohen’s d=8.16 versus the prior-work baseline.
  • Over two game days with 25 agents, knowledge of Sam’s mayoral run rose from 1 agent (4%) to 8 (32%), and knowledge of Isabella’s Valentine’s party rose from 1 (4%) to 13 (52%); none who claimed knowledge of this information had hallucinated it.
  • Relationship network density increased from 0.167 to 0.74; among 453 awareness responses, 1.3% (n=6) were hallucinated; five of twelve invited agents attended the Valentine’s party.
  • Common failures included retrieving wrong or incomplete memories, embellishing memories, and inheriting overly formal or overly cooperative language from the underlying model.

Methods

  • Architecture: an LLM (gpt3.5-turbo ChatGPT) augmented with a natural-language memory stream, retrieval scoring by recency (decay 0.995), importance (LLM-rated 1–10), and relevance (embedding cosine similarity), periodic reflection, and recursive planning/reacting/dialogue.
  • Controlled evaluation: within-subjects, 100 Prolific participants ranked believability of five conditions (full architecture, three ablations, human crowdworker) across five interview question categories; analysis used TrueSkill, Kruskal-Wallis, Dunn post-hoc with Holm-Bonferroni, plus inductive qualitative coding.
  • End-to-end evaluation: 25 agents in Smallville over two game days; measured information diffusion via interviews, relationship formation via mutual-knowledge network density, and coordination via party attendance.

Limitations

  • Stated: evaluation was short (two game days) and the human baseline was crowdworker-authored, not maximal human performance; simulation cost thousands of dollars in token credits and multiple days; robustness to prompt hacking, memory hacking, and hallucination is largely untested.
  • Stated: agents inherit underlying LLM imperfections, including potential bias/stereotypes and weaker believability for marginalized subpopulations due to limited data.
  • Evident from the text: agents showed erratic location choices, misapplied physical/social norms (e.g., one-person bathroom, after-hours stores), and instruction-tuning effects toward formality and excessive agreeableness; the environment was manually authored and not the focus.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3586183.3606763

Cites

89 of 90 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 15 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    The condition lacking observations, reflections, and planning effectively represents the prior state of the art for large-language-model agents.

    “The no observation, no refection, no planning condition efectively represents the previous state of the art for agents created through large language models [12,46,80].”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [12] 12. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to under- stand GPT-3. Proceedings of the National Academy of Sciences 120, 6 (2023), e2218523120. source

      The Discussion (page not provided) describes GPT-3 as learning passively from text and recommends that future agents interact with the world. The Results (page not provided) examine observations and decision-making in psychological tasks. Neither section identifies a condition without observations, reflection, and planning as the previous state of the art for LLM agents; searches for that condition and for a prior agent baseline found no such passage. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [46] 46. John J. Horton. 2023. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? arXiv:2301.07543 [econ.GN] source

      Section 2.4 describes creating LLM agents through prompts, but does not characterize a condition without observations, reflections, and planning. Searches for that combination and for the previous state of the art found no passage in our copy addressing the comparison. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
    • cites [80] 80. Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. In In the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22) (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10. 1145/3526113.3545616 source

      Section 4.2 describes a prior GPT-3 system that generates social participants through persona-based prompt chains, so this paper is an example of an earlier language-model agent approach. But §4.2.3 says replies are prompted with the prior conversation as context; the paper does not identify its system as a no-observation, no-reflection, no-planning condition or establish that condition as the previous state of the art. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Findingown data

    The standardized effect size for the difference between the prior-work condition and the full architecture was 8.16.

    “Comparing the condition representing prior work (with no memory, planning, or refection [12,46,80]) to the full architecture produces a standardized efect size of = 8.16, or eight standard deviations.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [12] 12. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to under- stand GPT-3. Proceedings of the National Academy of Sciences 120, 6 (2023), e2218523120. source

      Two searches of Binz and Schulz’s “Using cognitive psychology to understand GPT-3” found no relevant passage; the closest snippets concern GPT-3’s exploration behavior (scores 0.179 and 0.127), not an architecture ablation or an effect size of 8.16. An unscoped search found the specific 8.16 comparison in “Generative Agents: Interactive Simulacra of Human Behavior,” but its match score was 0.428, below the rubric’s 0.70 threshold for a misattribution verdict. The cited paper does not make a contrary claim, so the prior disputed verdict is unsupported; the exact-match finding warrants a manual citation check. (LLM, conf=0.55)

      Experimental — accuracy not yet measured
    • cites [46] 46. John J. Horton. 2023. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? arXiv:2301.07543 [econ.GN] source
    • cites [80] 80. Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. In In the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22) (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10. 1145/3526113.3545616 source

      Section 5.5 reports error rates for SimReddit and its ablations, but not a standardized effect size of 8.16. Searches for that value and for a prior-work condition without memory, planning, or reflection found no matching passage in our copy; page numbers were unavailable. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • Rests on an assumption: The five conditions adequately isolate the contributions of memory, planning, and reflection to believability.
  • Interpretationcitation

    The social behaviors observed in Smallville arise emergently rather than being pre-programmed.

    “Extending prior work [80], these social behaviors are emergent rather than pre-programmed.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [80] 80. Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. In In the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22) (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10. 1145/3526113.3545616 source

      Section 3 describes SimReddit generating users and their interactions with GPT-3, supporting the prior-work premise that social behaviors can be generated rather than individually pre-programmed. The paper does not examine Smallville, so it cannot establish how Smallville’s particular behaviors arose. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Findingnone given

    Starting from just one user-provided idea that an agent wanted to hold a Valentine's Day party, the agents independently spread invitations over the next two days.

    “For example, starting with only a single user-specifed notion that one agent wants to throw a Valentine's Day party, the agents autonomously spread invitations to the party over the next two”
    Quote in the paperNot a substantial claim
  • Findingown data

    With generative agents, telling just one agent that she intends to hold a party suffices to bring the party about.

    “We demonstrate that, with generative agents, it is sufcient to simply tell one agent that she wants to throw a party.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 36
  • May combine two claims. 42
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Instruction tuning appeared to make agents more polite and cooperative overall.

    “Finally, we observed possible efects of instruction tuning [79], which seemed to guide the behavior of the agents to be more polite and cooperative overall.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [79] 79. Long Ouyang, Jef Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155 [cs.CL] source
    • Rests on an assumption: A two-day simulation with 25 agents is sufficient to observe emergent social behaviors such as information diffusion, relationship formation, and coordination.

  • May combine two claims.

    Premisereasoning

    While a large language model can generate plausible behavior in response to situational information, agents need to plan over a longer time horizon to ensure coherent and believable sequences of actions.

    “While a large language model can generate plausible behavior in response to situational information (e.g., [46,80]), agents need to plan over a longer time horizon to ensure that their sequence of actions is coherent and believable.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [46] 46. John J. Horton. 2023. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? arXiv:2301.07543 [econ.GN] source
    • cites [80] 80. Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. In In the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22) (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10. 1145/3526113.3545616 source

  • May combine two claims.

    Findingnone given

    Generative agents show context-sensitive reactions, for example extinguishing a stove when breakfast burns, waiting when a bathroom is occupied, and starting a chat with an agent they wish to address.

    “For instance, generative agents turn of the stove when they see that their breakfast is burning, wait outside the bathroom if it is occupied, and stop to chat when they meet another agent they want to talk to.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Generative agents respond to an end user's changes to their environment or to natural-language instructions.

    “they respond when the end user changes their environment or commands them in natural language.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Generative agents make day plans shaped by their traits and history, execute those plans, and respond and adjust when needed.

    “they create daily plans that refect their characteristics and experiences, act out those plans, react, and re-plan when appropriate;”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    A community of generative agents exhibits emergent social patterns: new ties form, information circulates, and agents coordinate with one another.

    “A society full of generative agents is marked by emergent social dynamics where new relationships are formed, information difuses, and coordination arises across agents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In both evaluations, the most common mistakes happened when an agent could not recall pertinent memories, added fabricated content to its memory, or took on excessively formal language or conduct from the language model.

    “Across the technical and end-to-end evaluation, the most common errors arose when the agent failed to retrieve relevant memories, fabricated embellishments to the agent's memory, or inherited overly formal speech or behavior from the language model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Even though several things could have gone wrong, the agents managed to circulate invitations and attend the party, and one agent asked another on a date, all from a single user seed.

    “Despite many potential points of failure-the party planner must remember to invite other agents to the party, attendees must remember the invitation, those who remember must decide to actually show up, and moreour agents succeed.They spread the word about the party and then”
    • Also stated in: § 1 INTRODUCTION · plain
      “show up, with one agent even asking another on a date to the party, all from a single user-generated seed suggestion.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the authors' evaluation, generative agents showed believable conduct at both the individual level and in emergent social patterns.

    “In an evaluation, these generative agents produce believable individual and emergent social behaviors.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the Valentine's Day scenario, the user provided only Isabella's party intent and Maria's crush; the architecture itself produced the social actions of spreading the word, decorating, asking others out, attending, and interacting at the party.

    “In this scenario, the end user only set Isabella's initial intent to throw a party and Maria's crush on Klaus: the social behaviors of spreading the word, decorating, asking each other out, arriving at the party, and interacting with each other at the party were initiated by the agent architecture.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Complete fabrication of knowledge was rare; agents did not affirmatively claim to have experienced events they had not.

    “It was rare for the agents to completely fabricate their knowledge: they may fail to recall certain events having taken place and respond by acknowledging their lack of memory.However, they did not afrmatively claim to have experienced something they had not.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The agents' memory streams accurately record the events and dialogues that ground their interview responses.

  • May combine two claims.

    Findingnone given

    Retrieved memories are used to form longer-term plans and create higher-level reflections, which are entered into the memory stream for future use.

    “These retrieved memories are also used to form longer-term plans and create higher-level refections, both of which are entered into the memory stream for future use.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With reflection memories, Maria confidently suggested a birthday gift based on Wolfgang's interest in mathematical music composition, whereas without reflection she expressed uncertainty.

    “For instance, when asked what she might get Wolfgang Schulz for his birthday, Maria Lopez, with no access to refection, responded by acknowledging her uncertainty, stating that she did not know what Wolfgang likes, despite having had many interactions with him.However, with access to refection memories, Maria answered confdently,”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The agents' memory streams accurately record the events and dialogues that ground their interview responses.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Records are recursively synthesized into higher-level reflections that guide behavior.

    “Records are recursively synthesized into higher-and higher-level refections that guide behavior.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    As agents move through the world, they construct personal environment trees; these trees can become outdated when the agent leaves an area and are refreshed upon re-entry.

    “Agents build individual tree representations of the environment as they navigate it -subgraphs of the overall sandbox environment tree.”
    • Also stated in: § 5.1 From Structured World Environments to Natural Language, and Back Again · hedged
      “Agents are not omniscient: their tree may get out of date as they leave an area, and is updated when they re-enter the area.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The two evaluations demonstrate that the architecture's parts matter causally and reveal failures like those caused by faulty memory retrieval.

    “Two evaluations, a controlled evaluation and an end-to-end evaluation, that establish causal efects of the importance of components of the architecture, as well as identify breakdowns arising from, e.g., improper memory retrieval.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Natural language interviews can validly assess an agent's knowledge and behavior, including its ability to stay in character, remember, plan, react, and reflect accurately.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The sandbox server places nearby agents and objects, within a set visual range, into each agent's memory so the agent can respond appropriately.

    “The sandbox server is also responsible for sending all agents and objects that are within a preset visual range for each agent to that agent's memory, so the agent can react appropriately.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The study required substantial time and resources, costing thousands of dollars in token credits and taking multiple days to simulate 25 agents for two days.

    “The present study required substantial time and resources to simulate 25 agents for two days, costing thousands of dollars in token credits and taking multiple days to complete.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The initial plan generation yields a day-level outline split into five to eight blocks.

    “This generates a rough sketch of the agent's plan for a day, divided into fve to eight chunks:”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    For an action on an object, the language model is queried to determine the resulting change in the object's state.

    “When an agent executes an action on an object, we prompt the language model to ask what happens to the state of the object.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The importance prompt returns an integer value of 2 for 'cleaning up the room' and 8 for 'asking your crush out on a date.'

    “This prompt returns an integer value of 2 for”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Agents who notice each other may talk, and through those talks information can pass from one agent to another.

    “As agents notice each other, they may engage in dialogue-as they do so, information can spread from agent to agent.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Later, when Sam met Latoya again, his conduct indicated that he recalled their first meeting.

    “In a later interaction, Sam's interactions with Latoya indicate a memory of that interaction, as he asks”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Records from the memory stream are retrieved as relevant to plan the agent's actions and react appropriately to the environment.

    “From the memory stream, records are retrieved as relevant to plan the agent's actions and react appropriately to the environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The architecture retrieves relevant memories based on perceptions and uses those retrieved actions to determine an action.

    “Based on their perceptions, the architecture retrieves relevant memories and uses those retrieved actions to determine an action.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Agents notice other agents in their vicinity, and the architecture determines whether they continue walking or begin talking.

    “They are aware of other agents in their local area, and the generative agent architecture determines whether they walk by or engage in conversation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The memory stream is a database that maintains a comprehensive record of an agent's experience.

    “At the center of our architecture is the memory stream, a database that maintains a comprehensive record of an agent's experience.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Instruction tuning appeared to make agents overly cooperative, with Isabella rarely refusing others' suggestions and her interests being shaped by them.

    “Moreover, we observed that the instruction tuning also seemed to make the agents overly cooperative with one another.For example, Isabella received a wide range of suggestions and ideas from other agents for the Valentine's Day party from other agents, such as hosting a Shakespearean reading session or a professional networking event.Despite these ideas not aligning with her own interests and”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A two-day simulation with 25 agents is sufficient to observe emergent social behaviors such as information diffusion, relationship formation, and coordination.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Findingnone given

    The output suggests that John could consider asking Eddy about his music composition project.

    “The output suggests that John could consider asking Eddy about his music composition project.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In practice, the agents reflected roughly two or three times a day.

    “In practice, our agents refected roughly two or three times a day.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Gradually, Sam's campaign became widely discussed in town, with some backing him and others still undecided.

    “Gradually, Sam's candidacy becomes the talk of the town, with some supporting him and others remaining undecided.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Agents may change their plans midstream if needed.

    “Agents may change their plans midstream if needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Recency, importance, and relevance together produce effective results.

    “In our context, we focus on three main components that, together, produce efective results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Directly asking the language model to output an integer score for importance is effective.

    “There are many possible implementations of an importance score; we fnd that directly asking the language model to output an integer score is efective.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Believability declined as components were removed in the ablation conditions, with the no-reflection condition next best, then no-reflection-or-planning, then the crowdworker condition.

    “Performance degraded with the removal of each component in the ablation conditions: the ablated architecture with no access to refection was the next best ( = 26.88;= 0.69), followed by no access to refection or planning ( = 25.64;= 0.68), and then the crowdworker condition ( = 22.95; = 0.69).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The five conditions adequately isolate the contributions of memory, planning, and reflection to believability.

  • May combine two claims.

    Findingown data

    Without observational memory, Abigail denied knowing Rajiv Patel, but with memory she recalled and described him.

    “Without access to her observational memory, Abigail denied awareness of Rajiv Patel, an acquaintance in the sandbox world.But with the aid of memory, she could successfully recall Rajiv and describe him:”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The agents' memory streams accurately record the events and dialogues that ground their interview responses.

  • May combine two claims.

    Findingnone given

    By interacting, generative agents in Smallville swap information, create new ties, and coordinate joint activities.

    “By interacting with each other, generative agents in Smallville exchange information, form new relationships, and coordinate joint activities.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The retrieval function takes the agent's current situation as input and returns a subset of the memory stream to pass to the language model.

    “Our architecture implements a retrieval function that takes the agent's current situation as input and returns a subset of the memory stream to pass on to the language model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    As time passes, Smallville agents develop new ties and retain recollections of their dealings with other agents.

    “Agents in Smallville form new relationships over time and remember their interactions with other agents.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Agents affect the world via their actions and use natural language to communicate with each other.

    “The agents interact with the world by their actions, and with each other through natural language.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The dialogue generation produces Eddy's reply that the project is going well and that he has been walking in the garden for inspiration.

    “This generates Eddy's response:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Using the planning approach, Klaus's afternoon plan avoids repeated meals: he eats lunch at Hobbs Cafe while reading at noon, works on his paper at the library at 1pm, and walks in the park at 3pm.

    “With the approach described below, Klaus's afternoon plan is less gluttonous: he has lunch at Hobbs Cafe while reading at 12pm, works on his research paper at the school library at 1pm, and takes a break for a walk in the park at 3pm.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    Of the seven invited agents who did not attend, three cited conflicts and four expressed interest but did not plan to come.

    “We further inspected the seven agents who were invited to the party but did not attend by engaging them in an interview.Three cited conficts that prevented them from joining the party.For example, Rajiv, a painter, explained that he was too busy:”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A two-day simulation with 25 agents is sufficient to observe emergent social behaviors such as information diffusion, relationship formation, and coordination.

  • May combine two claims.

    Findingown data

    The Valentine's Day party did occur, with multiple agents gathering and interacting, despite several potential failure points in the event chain.

    “Despite many possible points of failure in the ensuing chain of events-agents might not act on that intent, might forget to tell others, might not remember to show up-the Valentine's Day party does, in fact, occur, with a number of agents gathering and interacting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The reflection tree for Klaus Mueller recursively synthesized his observations, represented as leaf nodes, to produce his self-concept of being highly dedicated to research.

    “A refection tree for Klaus Mueller.The agent's observations of the world, represented in the leaf nodes, are recursively synthesized to derive Klaus's self-notion that he is highly dedicated to his research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the simulation, the generative agent John Lin woke around 6 am, completed morning routines including brushing teeth, showering, and eating breakfast, and briefly spoke with his wife Mei and son Eddy before leaving for work.

    “John wakes up around 6 am and completes his morning routine, which includes brushing his teeth, taking a shower, and eating breakfast.He briefy catches up with his wife, Mei, and son, Eddy, before heading out to begin his workday.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The complete generative agent architecture yielded more believable behavior than both the ablated architectures and the human crowdworkers.

    “The full generative agent architecture produces more believable behavior than the ablated architectures and the human crowdworkers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Generative agents are believable human-behavior simulacra whose conduct is dynamically shaped by their evolving experiences and surroundings.

    “Generative agents, believable simulacra of human behavior that are dynamically conditioned on agents' changing experiences and environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The new architecture gives generative agents the capacity to remember, retrieve, reflect, interact with others, and plan while circumstances evolve.

    “A novel architecture that makes it possible for generative agents to remember, retrieve, refect, interact with other agents, and plan through dynamically evolving circumstances.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Large language models by themselves are not enough, but with a suitable architecture they offer a fresh path to believable agents.

    “We synthesize this research to suggest that large language models, though not sufcient by themselves, open up a new angle for creating believable agents when leveraged using the appropriate architecture.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Large language models give a chance to revisit believable-agent research, provided an effective architecture can turn memories into believable conduct.

    “Our argument is that large language models ofer an opportunity to re-examine these questions, provided that we can craft an efective architecture to synthesize memories into believable behavior.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Reflection allows agents to reflect not only on their observations but also on other reflections.

    “Refection explicitly allows the agents to refect not only on their observations but also on other refections: for example, the second statement about Klaus Mueller above is a refection that Klaus previously had, not an observation from his environment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The architecture builds on large language models' prompting strengths and adds components for sustained coherence, managing memory as it evolves, and recursively forming higher-level reflections.

    “The architecture leverages the powerful prompting capabilities of large language models and supplements those capabilities to support longer-term agent coherence, the ability to manage dynamically evolving memory, and recursively produce higher-level refections.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The robustness of generative agents remains largely unknown.

    “Lastly, the robustness of generative agents is still largely unknown.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In a within-subjects design, 100 participants compared responses to interview questions from four agent architectures and a human-authored condition for the same agent.

    “The study followed a within-subjects design, where 100 participants compared interview responses generated by four diferent agent architectures and a human-authored condition for the same agent.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Generative agents may struggle to produce believable behavior for certain subpopulations, especially marginalized populations, because of limited data availability.

    “Moreover, like many large language models, generative agents may struggle to generate believable behavior for certain subpopulations, particularly marginalized populations, due to limited data availability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Any imperfections in the underlying large language models will be inherited by generative agents.

    “In general, any imperfections in the underlying large language models will be inherited by generative agents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The evaluation of agent behavior in this study was limited to a relatively short timescale and a baseline human crowdworker condition.

    “In terms of evaluation, the assessment of generative agents' behavior in this study was limited to a relatively short timescale and a baseline human crowdworker condition.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The first author checked the workers' answers to the question about a typical weekday schedule to ensure they were coherent and in the agent's voice.

    “To ensure that the crowdworker-authored responses met at least a baseline expectation of quality, the frst author manually inspected the workers' responses to the question”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For every affirmative response about knowledge, the authors verified non-hallucination by locating the specific dialogue in the memory stream.

    “Additionally, for every response that confrmed the agents' knowledge of the information, we verifed that the agents did not hallucinate their responses by locating the specifc dialogue in their memory stream that provided them with the information.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The agents' memory streams accurately record the events and dialogues that ground their interview responses.

  • May combine two claims.

    Interpretationreasoning

    Without these mechanisms, large language models can output behavior, but the resulting agents may not react based on past experiences, may not make important inferences, and may not maintain long-term coherence.

    “Without these mechanisms, large language models can output behavior, but the resulting agents may not react based on the agent's past experiences, may not make important inferences, and may not maintain long-term coherence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Four sets of worker-authored responses failed the criteria and were regenerated by other workers.

    “Four sets of crowdworker-authored responses did not meet these criteria and were re-generated by other workers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    All architectures had the same access to the agent's accumulated memories up to the interview, so the observed differences likely underestimate the true differences.

    “Architectures were given equivalent access to all memories accrued by the agent up until the moment of the interview, so the diferences observed here likely represent a conservative estimate of the true diferences: in reality, the ablated architectures would not have followed the same path as the full architecture through the two-day simulation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Generative agents may be vulnerable to prompt hacking, memory hacking, and hallucination.

    “They may be vulnerable to prompt hacking, memory hacking-where a carefully crafted conversation could convince an agent of the existence of a past event that never occurred-and hallucination, among other issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Given known biases in language models, generative agents may exhibit biased behavior or stereotypes.

    “Given the known biases of language models, generative agents may potentially exhibit biased behavior or stereotypes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The design gave all architectures the same memory access because re-simulating each would lead to divergent states and make comparison difficult.

    “We chose to design the experiment this way as re-simulating for each architecture would cause the simulations to diverge into diferent states, making comparison challenging.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The starting memories shape the agent's behavior, and as the agent accumulates experience, its summary and behavior change.

    “These serve as the initial memories that determine the agent's behavior.These memories are initial starting points: as the agents gain more experience in the sandbox world, and as more records saturate the memory stream, the agent's summary and behavior will evolve.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The first author conducted an inductive analysis with two phases of qualitative open coding to compare response types across conditions.

    “Furthermore, the frst author conducted an inductive analysis [95] to study the qualitative distinctions between the responses produced in each condition.We employed qualitative open coding [33] in two phases.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [95] 95. David R. Thomas. 2006. A General Inductive Approach for Analyzing Qualitative Evaluation Data. American Journal of Evaluation 27, 2 (2006), 237-246. https: //doi.org/10.1177/1098214005283748 source
    • cites [33] 33. Uwe Flick. 2009. An Introduction to Qualitative Research. SAGE.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

68 of 68

  • The paper relies on this without support.

    Natural language interviews can validly assess an agent's knowledge and behavior, including its ability to stay in character, remember, plan, react, and reflect accurately.

    “In the technical evaluation, we leverage a methodological opportunity to evaluate an agent's knowledge and behavior by”

  • The paper relies on this without support.

    The language model can reliably produce coherent and contextually appropriate natural language for agent behavior.

    “Underlying this behavior is a novel agent architecture that combines a large language model with mechanisms for synthesizing and retrieving relevant information to condition the language model's output.”

  • The paper relies on this without support.

    The retrieval and synthesis mechanisms are sufficient to maintain long-term coherence.

    “Because generative agents produce large streams of events and memories that must be retained, a core challenge of our architecture is to ensure that the most relevant pieces of the agent's memory are retrieved and synthesized when needed.”

  • The paper relies on this without support.

    Human evaluators' believability rankings are a valid measure of behavioral believability.

    “Our dependent variable is the believability of the behavior, a central dependent variable in prior work on agents (e.g., [10]).”

  • The paper relies on this without support.

    The crowdworker-authored responses provide a meaningful human baseline for behavioral competency.

    “In addition to the ablation conditions, we added a condition with human crowdworker-authored behavior intended to provide a human baseline.”

  • The paper relies on this without support.

    Human evaluators' rankings of believability provide a valid measure of the believability of agent behavior.

    “Our experiment produced 100 sets of rank data, where each participant ranked the fve conditions by believability.”
    “Our fndings suggest that the full architecture of generative agents generates the most believable behavior among all the conditions.”

  • The paper relies on this without support.

    The five conditions adequately isolate the contributions of memory, planning, and reflection to believability.

    “Performance degraded with the removal of each component in the ablation conditions: the ablated architecture with no access to refection was the next best ( = 26.88;= 0.69), followed by no access to refection or planning ( = 25.64;= 0.68), and then the crowdworker condition ( = 22.95; = 0.69).”

  • The paper relies on this without support.

    The agents' memory streams accurately record the events and dialogues that ground their interview responses.

    “we verifed that the agents did not hallucinate their responses by locating the specifc dialogue in their memory stream that provided them with the information.”

  • The paper relies on this without support.

    A two-day simulation with 25 agents is sufficient to observe emergent social behaviors such as information diffusion, relationship formation, and coordination.

    “we allowed 25 agents to interact with each other continuously over two full game days in Smallville.”
    “We observed evidence of emergent outcomes across all three cases.”

  • The paper relies on this without support.

    The crowdworker condition is a meaningful baseline for human believability.

    “the crowdworker condition ( = 22.95; = 0.69).”
    “While the crowdworker condition provided a helpful comparison point, it did not represent the maximal human performance that could serve as the gold standard in terms of believability.”

  • The paper relies on this without support.

    The sandbox game world is a valid testbed for simulating human behavior.

    “We then demonstrate the potential of generative agents by manifesting them as non-player characters in a Sims-style game world and simulating their lives within it.”

  • The paper relies on this without support.

    For agents to coordinate, they must hear about an event and decide to attend by planning to arrive at the correct time and place.

    “To coordinate their behavior, agents need to hear about the event and choose to act on it by planning to show up at the right time and location.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Agent memory systems treat retrieval as a pure read, leaving state unchanged.

    “Agent memory systems do the same.”

    From Is Agent Memory a Database? Rethinking Data Foundations for Long-Term AI Agent Memory (2026) · cites this paper as [32]

    Section 4.1, “Memory and Retrieval” (page not provided), says memory objects have a most recent access timestamp and that retrieval scores recency by the time since a memory was last retrieved. Retrieval therefore affects the state used for subsequent retrievals, rather than leaving it unchanged. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • There are three commonly used criteria for memory information extraction: recency, relevance, and importance.

    “Usually, there three commonly used criteria for information extraction, that is, the recency, relevance, and importance [20].”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [20]

    The Introduction says the paper’s memory-retrieval model combines recency, relevance, and importance to select records. It does not establish that these are commonly used criteria for information extraction generally; the claim extends the paper’s specific design into a broader practice. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Multi-agent simulations show that LLM agents exhibit key social traits like trust and coordination.

    “Multi-agent simulations show that LLM agents exhibit key social traits like trust and coordination54,55;”

    From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [55]

    Section 7.1.2 (page not available) reports coordination around a party in a 25-agent simulation, as well as new relationships among agents. Section 7.1.1 measures relationships through mutual awareness, not trust; the paper does not establish that the agents exhibited trust. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • During decision-making, Generative Agents retrieve relevant reflections from semantic memory, then reason to make a high-level plan of the day.

    “During decision-making, it retrieves relevant reflections from semantic memory, then reasons to make a high-level plan of the day.”

    From Cognitive Architectures for Language Agents (2023) · cites this paper as Park et al., 2023

    Sections 4.2 (Reflection) and 4.3 (Planning and Reacting; page not provided) say reflections are stored with other memories and available for retrieval, and that the agent generates a broad-strokes plan for the day. The paper calls the store a “memory stream,” not “semantic memory,” and describes the initial daily plan as prompted by a summary of the agent and its previous day rather than by a distinct step of retrieving relevant reflections. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Generative Agents' action space has all four kinds of actions: grounding, reasoning, retrieval, and learning.

    “Its action space also has all four kinds of actions: grounding, reasoning, retrieval, and learning.”

    From Cognitive Architectures for Language Agents (2023) · cites this paper as Park et al., 2023

    The Introduction describes memory retrieval, reflection that draws higher-level inferences, and planning; Section 5.1 describes grounding the agent’s reasoning in the sandbox world. But the paper identifies three main architectural components—memory, reflection, and planning—not an action space with four kinds of actions. Recording and updating memories may resemble learning, but the cited passages do not establish learning as a fourth action kind. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • John's wake-up time differs between the text and figure caption

    Minor

    The account of John Lin's morning says he wakes at 7 a.m., while the caption for the same morning says he wakes around 6 a.m.

  • The party scenario is described as having both one and two user-set premises

    Minor

    The paper attributes the party scenario to a single user-generated seed, but its detailed account says the user also set Maria's crush on Klaus. That second premise bears directly on the reported date invitation.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 20

  1. John R. Anderson. 1993. Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale, NJ.
  2. Ruth Aylett. 1999. Narrative in virtual environments—towards emergent narrative. In Narrative Intelligence: Papers from the AAAI Fall Symposium (Technical Report FS-99-01). AAAI Press 83–86.
  3. 10.5555/7900.16137 · DOI
  4. Kevin Dill and L Martin. 2011. A Game AI Approach to Autonomous Control of Virtual Characters. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC’11). Orlando, FL, USA.
  5. Arpad E Elo. 1967. The Proposed USCF Rating System, Its Development, Theory, and Applications. Chess Life XXII, 8 (August 1967), 242–247.
  6. Rebecca Fiebrink and Perry R Cook. 2010. The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht), Vol. 3. Citeseer, 2–1.
  7. Uwe Flick. 2009. An Introduction to Qualitative Research. SAGE.
  8. 10.1609/aimag.v22i2.1558 · DOI
  9. John E. Laird. 2000. It Knows What You’re Going To Do: Adding Anticipation to a QUAKEBOT. In Papers from the AAAI 2000 Spring Symposium on Artificial Intelligence and Interactive Entertainment(Technical Report SS-00-02). AAAI Press 41–50.
  10. Josh McCoy, Michael Mateas, and Noah Wardrip-Fruin. 2009. Comme il Faut: A System for Simulating Social Games Between Autonomous Characters. In Proceedings of the 7th International Conference on Digital Arts and Culture. 87–94.
  11. Marvin Minsky and Seymour Papert. 1970. Draft of a proposal to ARPA for research on artificial intelligence at MIT, 1970–71.
  12. 10.1109/SNPD.2017.8023884 · DOI
  13. Kyle Orland. 2021. So what is ’the metaverse’, exactly?Ars Technica (7 November 2021). arxiv:2111.04169https://arstechnica.com/gaming/2021/11/so-what-is-the-metaverse-exactly/
  14. Richard W. Pew and Ann S. Mavor (Eds.). 1998. Modeling Human and Organizational Behavior: Applications to Military Simulations. National Academy Press Washington D.C.
  15. Prolific. 2022. Prolific: Quickly Find Research Participants You Can Trust. https://www.prolific.co/
  16. Mark O. Riedl. 2012. Interactive narrative: A novel application of artificial intelligence for computer games. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI’12). 2160–2165.
  17. Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42–61.
  18. Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.). Vol. 34. Curran Associates, Inc., 16183–16195. https://proceedings.neurips.cc/paper_files/paper/2021/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
  19. Graham Upton and Ian Cook. 2006. A Dictionary of Statistics (2 ed.). Oxford University Press, Oxford, United Kingdom.
  20. Robert Zubek. 2002. Towards implementation of social interaction. In AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment. AAAI Press. https://www.aaai.org/Papers/Symposia/Spring/2002/SS-02-01/SS02-01-003.pdf

Cited by

Sorted by citations

Related

Authors
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S.
Affiliations
  • Stanford University (US)
  • Google (United States) (US)
ORCID
Venue
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
Publisher
ACM
Published
2023-10-29
Pages
1-22
DOI
10.1145/3586183.3606763
OpenAlex
W4387835442
Semantic Scholar
5278a8eb2ba2429d4029745caf4e661080073c81
Cited by
2,187
References
70
Citations by year
2026: 961 · 2025: 745 · 2024: 380 · 2023: 101
FWCI
386.60
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Microsoft Research
  • Center for Advanced Study in the Behavioral Sciences, Stanford University
  • Center for Advanced Study, University of Illinois at Urbana-Champaign
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
generative AI agents, large language models, interactive simulation environment, natural language interaction, agent planning, interpersonal communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review