The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systems
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 40 references it cites were found in a registry or an index; 5 of the 17 sources its claims cite were checked for support.
Abstract
A growing body of multi-agent studies with Large Language Models (LLMs) explores how norms and cooperation emerge in mixed-motive scenarios, where pursuing individual gain can undermine the collective good. While prior work has explored these dynamics in both richly contextualized simulations and simplified game-theoretic environments, most LLM systems featuring common-pool resource (CPR) games provide agents with explicit reward functions directly tied to their actions. In contrast, human cooperation often emerges without explicit knowledge of the payoff structure or how individual actions translate into long-run outcomes, relying instead on heuristics, communication, and enforcement. We introduce a CPR simulation framework that removes explicit reward signals and embeds cultural-evolutionary mechanisms: social learning (adopting strategies and beliefs from successful peers) and norm-based punishment, grounded in Ostrom's principles of resource governance. Agents also individually learn from the consequences of harvesting, monitoring, and punishing via environmental feedback, enabling norms to emerge endogenously. We establish the validity of our simulation by reproducing key findings from existing studies on human behavior. Building on this, we examine norm evolution across a 2x2 grid of environmental and social initialisations (resource-rich vs. resource-scarce; altruistic vs. selfish) and benchmark how agentic societies comprised of different LLMs perform under these conditions. Our results reveal systematic model differences in sustaining cooperation and norm formation, positioning the framework as a rigorous testbed for studying emergent norms in mixed-motive LLM societies. Such analysis can inform the design of AI systems deployed in social and organizational contexts, where alignment with cooperative norms is critical for stability, fairness, and effective governance of AI-mediated environments.
Overview
CoChat Summary
TL;DR
The authors introduce a common-pool resource (CPR) simulation framework for LLM multi-agent societies in which the action-to-payoff mapping is latent and cooperation must emerge endogenously through payoff-biased social learning, individual punishment, and a lightweight propose→vote group-norm mechanism grounded in Ostrom's design principles. Validated against agent-based-model baselines from human-behavior studies, the framework reveals systematic model-specific differences in sustaining cooperation and norm formation.
Key findings
- Removing both alignment channels (social learning and group norms) produced the lowest survival times (T_s = 16.22 vs. 20.98, t(898) = −2.78, p = 0.006), confirming some form of coordination is necessary for stability.
- Explicit alignment alone (Only Group Decision) sometimes outperformed the full system, particularly under selfish priors (T_s = 38.21 vs. 27.1, t(238) = 3.44, p < 0.001); pure social learning without a shared norm was often unstable (T_s = 17.56 vs. 20.98, t(898) = −1.96, p = 0.050).
- Two-way ANOVA: in the harsh environment only the model main effect was significant (p = 0.031), with a significant Model × Altruistic ratio interaction (p = 0.012); in the rich environment both model (p < 0.001, η² = 0.28) and society type (p = 0.030) were significant.
- Model-specific patterns: deepseek-r1 often reached the 50-round cap and survived significantly longer than all other models in the rich environment, while claude-sonnet-4 and gpt-4o plateaued near 30 rounds; norm structure showed provider-family clustering (e.g., claude-sonnet-4 top-right on alignment/homogeneity, Llama variants lower-left), with initialization effects second-order to model effects.
Methods
- A CPR simulation with N = 10 agents sharing a renewable stock (carrying capacity K = 300) under logistic (Verhulst) regeneration; agents choose effort, consume a fixed amount, may monitor and punish at personal cost, imitate higher-payoff peers via a logit rule, and set a group harvest threshold through a propose→vote procedure (two API calls per agent per round).
- Agents receive no explicit reward function or payoff table; consequences are inferred from private observations of personal and sampled peer outcomes, with LLM decisions made in-context via dedicated prompts.
- Framework validated against ABM baselines on three axes (punishment sustains cooperation and declines when removed at t = 15; outcomes vary with punishment strength β and growth rate r; altruistic vs. selfish populations differ), then LLM societies tested across a 2×2 grid (resource-rich r = 0.6 vs. resource-scarce r = 0.2; altruistic vs. selfish initialization), capped at 50 rounds with 10 trials per condition, analyzed via two-way ANOVA and Tukey's HSD post-hoc tests (α = 0.05) with Compact Letter Display.
Limitations
- Stated: computational constraints limited trials and time horizons, possibly underrepresenting long-term norm evolution; the single-resource CPR setting with a narrow set of governance mechanisms cannot capture multi-resource, cross-group, or layered-norm complexity; reliance on in-context learning introduces sensitivity to prompt design and model biases, limiting reproducibility; closed-source models hinder transparency and independent replication.
- Evident from the text: the 50-round cap was chosen partly because preliminary runs showed most populations collapse by roughly 50 rounds, so survival-time comparisons are truncated (deepseek-r1 "often reaches the 50-round cap"), which may compress differences among longer-surviving models; the ablation results are context-dependent (e.g., OSL competitive under altruistic priors), so the ranking of mechanisms is not uniform across conditions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 2 Not valid · 7 Not checked · 80 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Payoff-biased learning is a common learning strategy among humans.”
What the sources say
- Partly supportedcites [20] [20] Richard McElreath, Adrian V Bell, Charles Efferson, Mark Lubell, Peter J Richerson, and Timothy Waring. 2008. Beyond existence and aiming outside the laboratory: estimating frequency-dependent and pay-off-biased social learning strategies. Philosophical Transactions of the Royal Society B: Biological Sciences 363, 1509 (2008), 3515-3528. - sourceExperimental — accuracy not yet measured
The Discussion (p. n/a, §6) states that 'our human participants relied heavily on pay-off-biased social learning', and the Analysis summary (§2f) shows that, unless the environment is extremely stochastic or almost perfectly stable, 'some kind of pay-off-biased learning is an evolutionarily stable strategy, in our simulations' — so the source supports payoff-biased learning being used by humans and favoured under a wide range of conditions. However, the generality claimed ('a common learning strategy among humans') exceeds this evidence: the paper's own data come from a single laboratory decision-task with human subjects, not from natural human populations. Indeed the Discussion explicitly cautions that the experimental environment 'provides highly accurate (noise free) pay-off information, whereas real social environments certainly do not', and that 'the clear advantage of pay-off bias depends upon the ability to know that any cue of success arises from a particular behaviour'. The claim's unqualified frequency statement across humans is therefore only partly supported. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In all cases, the utility function is clearly defined, such as 'units harvested' or 'tokens contributed to the public good', and directly linked to actions.”
What the sources say
- Partly supportedcites [3] [3] Steffen Backmann, David Guzman Piedrahita, Emanuel Tewolde, Rada Mihalcea, Bernhard Schölkopf, and Zhijing Jin. 2025. When Ethics and Payoffs Diverge: LLMAgents in Morally Charged Social Dilemmas. arXiv 2505.19212 (2025), 1-33. - sourceExperimental — accuracy not yet measured
Our copy does define payoffs as explicit functions of the agents' actions for the games it uses — Section 3.1 gives the public goods payoff p_i = E - c_i + (a/N)Σc_j over the contribution choice c_i ∈ [0,E], and Section 4.1 writes the round payoff as p_i,t(a_i,t, a_-i,t) — so the 'clearly defined and directly linked to actions' part holds for the public-goods (contribution/token) setting. But the paper studies no common-pool resource game and never uses a 'units harvested' utility: its Limitations state it covers only the prisoner's dilemma and public goods game, and neither the Introduction nor Section D (Related work, which merely lists Piatti et al. 2024 as 'dynamic multi-agent resource environments') characterises prior CPR studies' utility functions. The sweeping claim about 'all prior CPR studies' is therefore not the source's own point. (LLM, conf=0.66)
- Partly supportedcites [26] [26] Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. 2024. Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents. In Proceedings of the 37th International Conference on Neural Information Processing Systems . Curran Associates Inc., Red Hook, US, 111715-111759. - sourceExperimental — accuracy not yet measured
Section 2.4 (p. 4) defines each agent’s total gain as the sum of resources collected, linking a measured outcome to harvesting actions. The paper does not establish that the utility function is clearly defined this way in all prior CPR studies, nor does it discuss the example of tokens contributed to a public good. (LLM, conf=0.78)
- Supportedcites [27] [27] David Guzman Piedrahita, Yongjin Yang, Mrinmaya Sachan, Giorgia Ramponi, Bernhard Schölkopf, and Zhijing Jin. 2025. Corrupted by Reasoning: Reasoning 28. Language Models Become Free-Riders in Public Goods Games. In Proceedings of the 2nd Conference on Language Modeling . OpenReview, 1-37. - source
Section 3.1 (p. 4) formally defines the payoff structure π_i in terms of the initial endowment e, agent i's contribution c_i, the multiplication factor α and group size N, i.e. utility is an explicit function of the contribution action — exactly the 'tokens contributed to the public good' example the citing sentence gives. Section 3.2 (p. 4) extends this to the sanctioning stage (reward +1, punishment −3, cost 1 token from endowment s), and the agent prompt in Appendix A.1 (p. 14) spells the same payoff arithmetic out to the model, so the utility function is clearly defined and directly linked to actions throughout the design. The only element not this paper's own is the citing authors' umbrella label 'prior CPR studies' — the source frames itself as a public goods game (citing Ostrom 1990, Ledyard 1994) — but the substance attributed to it is stated. (LLM, conf=0.85)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Family clustering: models from the same provider occupy similar regions-for example, the Llama variants lie lower-left (less homogeneous, weakly aligned), the OpenAI pair ( gpt-4o and gpt-4o-mini ) clusters mid-high with gpt-4o-mini highest on both axes, claude-sonnet-4 sits top-right (very high alignment and homogeneity), and qwen3-32b falls in the highalignment band”
- Also stated in: § C SUPPLEMENTAL FIGURES · p. 12 · boosted
“Models exhibit clear family clustering: Llama variants lie lower-left (weaker coordination), the OpenAI pair clusters mid-high with gpt-4o-mini highest on both axes, claude-sonnet-4 sits top-right, and qwen3-32b falls in the high-alignment band.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § C SUPPLEMENTAL FIGURES · p. 12 · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while the altruistic population sometimes suffered from starvation.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, OSL can outperform other ablations in some settings (e.g., altruistic initializations)”
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
52 more claims
Create a free account to see all 92 claims and to verify the 4 references still open.
- It may state two separate claims as one. 28
- Probably not a substantial claim. 12
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ B PARAMETERS FOR THE SIMULATIONS · p. 10
“selfish agents harvest more and punish less (second-order free-riding), while altruistic agents harvest less and punish more.”
§ Table 3: Parameters and initial values · p. 11
“| 𝐵 𝑎𝑙𝑡𝑟𝑢𝑖𝑠𝑡𝑖𝑐 | The probability of Altruistic agents punishing another agent if they violate the group norm | Uniform ( 0 , 0 . 1 ) | | 𝐵 𝑠𝑒𝑙 𝑓 𝑖𝑠ℎ | The probability of selfish agents punishing another agent if they violate the group norm | Uniform ( 0 . 4 , 0 . 5 ) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Framework · p. 3
“Technology and sanctions are parameterized by productivity 𝛼 > 0, penalty 𝛽 > 0, and punisher cost 𝛾 > 0.”
§ Table 3: Parameters and initial values · p. 11
“| 𝛾 (punishing cost) | The cost of punishing others | 0 |”
Citations
Cites
Hardin, G. (1968) · Science · doi:10.1126/science.162.3859.1243 · Cited by 25,167
Governing the Commons: The Evolution of Institutions for Collective Action
Brady, G., & Ostrom, E. (1993) · Southern Economic Journal · doi:10.2307/1059950 · Cited by 16,839 · cited 3×
“Game-theory frameworks such as common pool resources games (CPR) provide a useful tool to understand the different components of cooperation in complex social-ecological systems, and help practitioners develop efficient self-governance systems [23].”
InfluentialA General Framework for Analyzing Sustainability of Social-Ecological Systems
Ostrom, E. (2009) · Science · doi:10.1126/science.1172133 · Cited by 8,656 · cited 2×
“…on insights from political science and institutional economics, particularly Ostrom’s institutional design principles for governing the commons [23, 25], and from cultural evolution theory [6–8 , Figure 1 provides an overview of our framework, which comprises four modules: Harvest and Consumption…”
(1992) · Reviews in Fish Biology and Fisheries · doi:10.1007/bf00042883 · Cited by 3,500 · cited 1×
“𝑖 ( 𝑡 ) = 𝛼 𝑒 𝑖 ( 𝑡 ) 𝑅 ( 𝑡 ) , so total extraction scales linearly with current stock and individual effort [16].”
Szabó, G., & Fáth, G. (2007) · Physics Reports · doi:10.1016/j.physrep.2007.04.004 · Cited by 2,974 · cited 1×
“Agent 𝑖 meets 𝑘 at random and adopts 𝑠 𝑘 ( 𝑡 ) with the logit rule where ¯ 𝑃 𝑖 ( 𝑡 ) is a payoff (e.g., an exponential moving average) and 𝛿 > 0 controls selection strength ( [34]; Eq.”
Henrich, J., & Gil-White, F. J. (2001) · Evolution and Human Behavior · doi:10.1016/s1090-5138(00)00071-4 · Cited by 2,500 · cited 2×
“One explanation is that sanctioning practices can spread locally through conformity [15], and spread across groups through payoff-biased learning [7].”
On the Rationale of Group Decision-making
Black, D. (1948) · Journal of Political Economy · doi:10.1086/256633 · Cited by 2,439 · cited 1×
“When proposals are numeric along a single policy dimension, we update the group norm by the median-voter rule [5]: 𝐺 ( 𝑡 + 1 ) = median (cid:0) G( 𝑡 ) (cid:1) .”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 3×
“Park et al. [26] introduced one of the first large-scale simulations of an agentic society in the Smallville sandbox environment, where LLM-driven agents navigate rich daily-life contexts.”
(2001) · doi:10.1006/jtbi.2000.2202 · Cited by 803
Local Enforcement and Better Forests
Gibson, C. C., Williams, J. T., & Ostrom, E. (2005) · World Development · doi:10.1016/j.worlddev.2004.07.013 · Cited by 593 · cited 1×
“…axes: (a) punishment sustains cooperation, but if removed, cooperation declines [32, 35]; (b) cooperation outcomes vary with punishment strength [10] and environmental growth rate; and (c) populations with different levels of altruism [4], defined by their harvest thresholds, show distinct…”
Group beneficial norms can spread rapidly in a structured population.
BOYD, R., & RICHERSON, P. J. (2002) · Journal of Theoretical Biology · doi:10.1006/jtbi.2001.2515 · Cited by 455 · cited 2×
“Cultural evolution highlights payoff-biased learning as well as group-level selection as evolutionary mechanisms that can select for group-beneficial norms [7, 33].”
Trustworthiness and competitive altruism can also solve the “tragedy of the commons”
Barclay, P. (2004) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2004.04.002 · Cited by 403 · cited 1×
“…punishment sustains cooperation, but if removed, cooperation declines [32, 35]; (b) cooperation outcomes vary with punishment strength [10] and environmental growth rate; and (c) populations with different levels of altruism [4], defined by their harvest thresholds, show distinct survival patterns.”
Punitive sentiment as an anti-free rider psychological device.
Price, M. E., Cosmides, L., & Tooby, J. (2002) · Evolution and Human Behavior · doi:10.1016/s1090-5138(01)00093-9 · Cited by 360 · cited 1×
“Punishment and sanctioning are important for maintaining cooperation [14, 24, 29].”
McElreath, R., Bell, A. V., Efferson, C., Lubell, M., Richerson, P. J., & Waring, T. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0131 · Cited by 283
The spontaneous emergence of conventions: An experimental study of cultural evolution
Centola, D., & Baronchelli, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418838112 · Cited by 282 · cited 1×
“Instead, people act based on local heuristics, and cooperation emerges over time through normative values, punishment, and other social mechanisms [9].”
Cooperation, Punishment, and the Evolution of Human Institutions
Henrich, J. (2006) · Science · doi:10.1126/science.1126398 · Cited by 233 · cited 1×
“Instead, people act based on local heuristics, and cooperation emerges over time through normative values, punishment, and other social mechanisms [9].”
The Moral Economy of Communities
Bowles, S., & Gintis, H. (1998) · Evolution and Human Behavior · doi:10.1016/s1090-5138(98)00015-4 · Cited by 181 · cited 1×
“…economics, particularly Ostrom’s institutional design principles for governing the commons [23, 25], and from cultural evolution theory [6–8 , Figure 1 provides an overview of our framework, which comprises four modules: Harvest and Consumption , Individual Punishment , Social Learning…”
Székely, Á., Lipari, F., Antonioni, A., Paolucci, M., Sánchez, Á., Tummolini, L., & Andrighetto, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25734-w · Cited by 148 · cited 1×
“We validate the framework along three axes: (a) punishment sustains cooperation, but if removed, cooperation declines [32, 35]; (b) cooperation outcomes vary with punishment strength [10] and environmental growth rate; and (c) populations with different levels of altruism [4], defined by their…”
Voting with your feet: Payoff biased migration and the evolution of group beneficial behavior
Boyd, R. T., & Richerson, P. J. (2008) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2008.12.007 · Cited by 136 · cited 1×
“…economics, particularly Ostrom’s institutional design principles for governing the commons [23, 25], and from cultural evolution theory [6–8 , Figure 1 provides an overview of our framework, which comprises four modules: Harvest and Consumption , Individual Punishment , Social Learning ,…”
Emergence of Social Norms in Generative Agent Societies: Principles and Architecture
Ren, S., Cui, Z., Song, R., Wang, Z., & Hu, S. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/874 · Cited by 107 · cited 2×
“To understand cooperation in such settings, researchers have explored both complex, high-context scenarios, such as LLM agents in historical diplomacy [17, 30] or virtual societies [26, 38], and simplified, game-theoretic environments that serve as testbeds for cooperative mechanisms [27, 31, 37].”
Escalation Risks from Language Models in Military and Diplomatic Decision-Making
Rivera, J.-P., Mukobi, G., Reuel, A., Lamparth, M., Smith, C., & Schneider, J. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658942 · Cited by 77 · cited 1×
“To understand cooperation in such settings, researchers have explored both complex, high-context scenarios, such as LLM agents in historical diplomacy [17, 30] or virtual societies [26, 38], and simplified, game-theoretic environments that serve as testbeds for cooperative mechanisms [27, 31, 37].”
Cultural group selection and human cooperation: a conceptual and empirical review
Major‐Smith, D. (2020) · Evolutionary Human Sciences · doi:10.1017/ehs.2020.2 · Cited by 67 · cited 1×
“Cultural evolution highlights payoff-biased learning as well as group-level selection as evolutionary mechanisms that can select for group-beneficial norms [7, 33].”
Cultural Evolution of Cooperation among LLM Agents
Vallinder, A., & Hughes, E. (2024) · Adaptive Agents and Multi-Agent Systems · doi:10.48550/arxiv.2412.10270 · Cited by 46 · cited 2×
“To understand cooperation in such settings, researchers have explored both complex, high-context scenarios, such as LLM agents in historical diplomacy [17, 30] or virtual societies [26, 38], and simplified, game-theoretic environments that serve as testbeds for cooperative mechanisms [27, 31, 37].”
Emergent Bartering Behaviour in Multi-Agent Reinforcement LearningPreprint
Johanson, M. B., Hughes, E., Timbers, F., & Leibo, J. Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.06760 · Cited by 43
The cultural evolution of collective property rights for sustainable resource governance
Andrews, J., Clark, M., Hillis, V., & Borgerhoff Mulder, M. (2024) · Nature Sustainability · doi:10.1038/s41893-024-01290-1 · Cited by 31 · cited 1×
“There is much evidence that individuals who excel tend to be imitated excessively ([15]), which creates a selective force toward cultural strategies that yield higher payoffs [1, 21].”
Punishment Leads to Cooperative Behavior in Structured Societies
Shutters, S. T. (2011) · Evolutionary Computation · doi:10.1162/evco_a_00062 · Cited by 17 · cited 1×
“We validate the framework along three axes: (a) punishment sustains cooperation, but if removed, cooperation declines [32, 35]; (b) cooperation outcomes vary with punishment strength [10] and environmental growth rate; and (c) populations with different levels of altruism [4], defined by their…”
Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov GamesPreprint
Oldenburg, N., & Tan, Z.-X. (2024) · Adaptive Agents and Multi-Agent Systems · doi:10.48550/arxiv.2402.13399 · Cited by 16 · cited 1×
“Relatedly, Oldenburg and Zhi-Xuan [22] study norm inference via a Bayesian model over an explicit candidate norm space, whereas our agents propose and adopt free-form natural-language norms and adapt through payoff-biased social learning and enforcement.”
War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars
Hua, W., Fan, L., Li, L., Mei, K., Ji, J., Ge, Y., Hemphill, L., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2311.17227 · Cited by 15 · cited 1×
“To understand cooperation in such settings, researchers have explored both complex, high-context scenarios, such as LLM agents in historical diplomacy [17, 30] or virtual societies [26, 38], and simplified, game-theoretic environments that serve as testbeds for cooperative mechanisms [27, 31, 37].”
Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents
Piatti, G., Jin, Z., Kleiman-Weiner, M., Schölkopf, B., Sachan, M., & Mihalcea, R. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3548 · Cited by 14 · cited 5×
“The latter offer greater control and inter-pretability, yet the pathways by which LLM societies autonomously develop norms or sustain cooperation remain underexplored.”
InfluentialNorm Enforcement with a Soft Touch: Faster Emergence, Happier AgentsPreprint
Tzeng, S.-T., Ajmeri, N., & Singh, M. P. (2024) · Adaptive Agents and Multi-Agent Systems · doi:10.48550/arxiv.2401.16461 · Cited by 8 · cited 1×
“Tzeng et al. [36] investigate norm compliance using structured normative messages; in contrast, we allow open-ended norm expression and use propose → vote to approximate deliberation under limited API budgets.”
Warnakulasuriya, K., Dissanayake, P., Silva, N. D., Cranefield, S., Savarimuthu, B. T. R., Ranathunga, S., & Silva, N. D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.19487 · Cited by 6 · cited 1×
“To understand cooperation in such settings, researchers have explored both complex, high-context scenarios, such as LLM agents in historical diplomacy [17, 30] or virtual societies [26, 38], and simplified, game-theoretic environments that serve as testbeds for cooperative mechanisms [27, 31, 37].”
Agent Alignment in Evolving Social Norms
Li, S., Sun, T., Cheng, Q., & Qiu, X. (2024) · arXiv · doi:10.48550/arxiv.2401.04620 · Cited by 3 · cited 1×
“For example, Ren et al. [30] proposed CRSEC, a four-module framework for norm emergence encompassing Creation & Representation, Spreading, Evaluation, and Compliance, while [19] developed an EvolutionaryAgent that evolves cooperative norms over time.”
Warnakulasuriya, K., Dissanayake, P., Silva, N. D., Cranefield, S., Savarimuthu, B. T. R., Ranathunga, S., & de Silva, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-17542-7_7 · Cited by 2
Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games
Piedrahita, D. G., Yang, Y., Sachan, M., Ramponi, G., Schölkopf, B., & Jin, Z. (2025) · arXiv · doi:10.48550/arxiv.2506.23276 · cited 3×
“Piedrahita et al. [28] adapted CPR settings to study norm enforcement via sanctioning, allowing norms to adapt over time.”
When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas
Backmann, S., Piedrahita, D. G., Zhang, T. J., Tewolde, E., Mihalcea, R., Schölkopf, B., & Jin, Z. (2025) · arXiv · doi:10.48550/arxiv.2505.19212 · cited 2×
“Backmann et al. [3] examined CPR settings with moral imperatives in conflict with explicit incentives.”
Verhulst and the logistic equation (1838)
Bacaër, N. (2011) · A Short History of Mathematical Population Dynamics · doi:10.1007/978-0-85729-115-8_6
Research works printed but not linked: 2
- [15][15] Ray Hilborn and Carl J Walters. 2013. Quantitative fisheries stock assessment: choice, dynamics and uncertainty . Springer Science & Business Media. -
- [19][19] Chen Cecilia Liu. 2025. Cooperative Behaviour in LLMs via Cultural Evolution of Norms and Strategies. In Proceedings of the 1st CoLM Workshop on Social Simulation with LLMs . OpenReview, 1-11. -
Cited by
Chen, E., Guan, C., Elshafiey, A., Zhao, Z., Zekeri, J., Shaibu, A. E., Prince, E. O., & Wu, C. J. (2026) · arXiv · doi:10.48550/arxiv.2602.18832 · Cited by 11
Gender Dynamics and Homophily in a Social Network of LLM Agents
Fadaei, F., Moran, J., & Yasseri, T. (2026) · arXiv.org · doi:10.1098/rsta.2025.0205 · Cited by 8
“…identities and shared norms from identical initial conditions [57]; display behavioural patterns in social dilemmas and common pool resource environments [23, 39]; establish cooperative norms under punishment mechanisms [59, 62]; and may adopt harmful behaviours through repeated exposure [15].”
Chen, E., Guan, C., Elshafiey, A., Zhao, Z., Zekeri, J., Shaibu, A. E., & Prince, E. O. (2026) · arXiv · doi:10.48550/arxiv.2602.14477
“Research on multi-agent systems shows LLMs can produce outputs resembling social behaviors: cooperation [6], norm formation [14], and cultural evolution [19].”
Syrnikov, M. B., Pierucci, F., Prandi, M., Galisai, M., Bisconti, P., Giarrusso, F., & Nardi, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.22188 · Cited by 2
“Related work shows that collective outcomes also depend on norm formation, power asymmetries, and mechanisms designed to sustain cooperation (Gupta et al., 2025; Borah, 2026; Tewolde et al., 2026).”
Haghrah, A. A., & Haghrah, A. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.27039 · Cited by 1
“Furthermore, the literature on Agent-Based Modeling (ABM) has explored how local feedback loops, such as individual rewards and punishments, maintain system cooperative stability [25].”
Metadata
- Authors
- Gupta, P., Zhong, Q., Yakura, H., Eisenmann, T., & Rahwan, I.
- Affiliations
- Max Planck Institute for Human Development (DE)
- Venue
- Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems
- Published
- 2026-05-24
- ISSN
- 1548-8403 (ISSN-L)
- OpenAlex
- W7125906388
- Semantic Scholar
- ae76ecc1bd942b495eb5dffa079d9e1e75fbac41
- Cited by
- 2
- References
- 40
- Citations by year
- 2026: 2
- FWCI
- 47.57
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Adaptive Agents and Multi-Agents Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Japan Science and Technology Agency
- Precursory Research for Embryonic Science and Technology (JPMJPR246B)
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Keywords
- cooperation, social learning, norm formation, cultural evolution, emergent norms, natural resource governance, AI alignment, simulation framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.