The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index; 39 of the 65 sources its claims cite were checked for support.
Emergent social conventions and collective bias in LLM populations
Open accessPublished version of a preprint
Abstract
Social conventions are the backbone of social coordination, shaping how individuals form a group. As growing populations of artificial intelligence (AI) agents communicate through natural language, a fundamental question is whether they can bootstrap the foundations of a society. Here, we present experimental results that demonstrate the spontaneous emergence of universally adopted social conventions in decentralized populations of large language model (LLM) agents. We then show how strong collective biases can emerge during this process, even when agents exhibit no bias individually. Last, we examine how committed minority groups of adversarial LLM agents can drive social change by imposing alternative social conventions on the larger population. Our results show that AI systems can autonomously develop social conventions without explicit programming and have implications for designing AI systems that align, and remain aligned, with human values and societal goals.
Overview
CoChat Summary
TL;DR
Populations of LLM agents playing a local naming game spontaneously converge on global social conventions, and this process can generate collective biases toward particular conventions even when individual agents appear unbiased. Committed minorities can overturn established conventions once they reach model- and convention-dependent critical masses.
Key findings
- Across all four tested LLMs, group-wide linguistic conventions spontaneously emerged; a shared convention was established by population round 15 in all cases except Llama-2-70b-Chat, and convergence held for populations as large as N = 200 and name pools as large as W = 26.
- Consensus conventions were not uniformly distributed: some names were much more likely to become the convention, with preferred names varying by model. In W = 2 {Q,M} experiments, collective bias toward a “strong” convention emerged even when initial individual selections were statistically unbiased (P = 0.068, 0.116, 0.757, and 0.849 across models); by interaction 3, dominant memory configurations showed bias toward the strong convention (P < 2.2 × 10⁻¹⁶), e.g., P(M | {1:M,Q; 2:Q,M}) = 0.848 versus P(Q | {1:Q,M; 2:M,Q}) = 0.451.
- Agents continued a successful name 99.4% of the time and switched after failure 97.3% of the time.
- A committed minority could flip a consensus, with critical masses ranging from as small as 2% (Llama-3-70B-Instruct) to as large as 67% (Llama-2-70b-Chat); stronger conventions required larger committed minorities to overturn, and Llama-3.1-70B-Instruct populations spontaneously switched to the stronger convention without any committed agents.
Methods
- Naming-game simulation: N agents randomly paired each time step choose a “name” from a finite pool W, with memory of the past H interactions and payoffs of +100 for success and −50 for failure; no incentive promotes global consensus. Homogeneous populations were instantiated from Llama-2-70b-Chat, Llama-3-70B-Instruct, Llama-3.1-70B-Instruct, and Claude-3.5-Sonnet, with default N = 24, W = 10, H = 5 unless otherwise specified.
- Individual bias was measured from first-round choices with empty memory: exact binomial tests for W = 2 (null probability 0.5) and chi-square tests for W = 10 (null probability 0.1), rejecting unbiasedness at P < 0.05.
- Critical-mass experiments introduced committed minority agents using a fixed alternative convention; a consensus flip was defined as 95% of the past 3N interactions succeeding after introduction, or for other models as the minimum proportion needed to flip consensus within 30 population rounds.
Limitations
- Paper-stated: results rely on several parameters, including the LLM model, prompt, and specific conventions; future work should generalize to different controlled settings, larger populations and semantic spaces, structured social networks, interactions involving more than two agents, and more realistic human norms such as gender and race.
- Paper-stated: the study uses unstructured populations with random pairing and does not directly compare with human populations; LLM-specific collective-bias phenomena call for further human testing and caution for digital-twin modeling or social deployment.
- Evident from the text: experiments use homogeneous populations of four LLMs and alphabet-letter conventions, with key bias and critical-mass analyses often restricted to W = 2, which constrains generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
24 Valid · 1 Not valid · 3 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While most research has focused on investigating and addressing bias in one-to-one interactions between humans and LLMs (18–20), less attention has been given to how these biases evolve through repeated communications in populations of LLM agents and, ultimately, in mixed human-LLM ecosystems (15), even though the safety of a single LLM does not necessarily imply the safety of a multi-agent system”
§ Emergent social conventions and collective bias in LLM populations > INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites [18] D. Roselli, J. Matthews, N. Talagala, “Managing bias in AI,” in Companion Proceedings of the 2019 World Wide Web Conference (ACM, 2019), pp. 539–544. source
- Partly supportedcites [19] E. Ferrara, Should ChatGPT be biased? Challenges and risks of bias in large language models. First Mon. 28, doi.org/10.5210/fm.v28i11.13346 (2023). sourceExperimental — accuracy not yet measured
Section 3.3 discusses identifying and mitigating bias through human review, feedback, and user control of LLM outputs. But the paper does not establish that most research focuses on one-to-one human–LLM interactions; that comparative claim goes beyond what our copy says. (LLM, conf=0.82)
- Partly supportedcites [20] T. Hu, Y. Kyrychenko, S. Rathje, N. Collier, S. van der Linden, J. Roozenbeek, Generative language models exhibit social identity biases. arXiv:2310.15819 (2023). PMC11774750 39668254 sourceExperimental — accuracy not yet measured
The Main and Study 3 Results examine bias in individual human–LLM conversations, and the Discussion describes the paper’s bias measure as single-turn. These passages support the example of one-to-one bias research, but they do not establish that most research has focused on such interactions. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“less attention has been given to how these biases evolve through repeated communications in populations of LLM agents and, ultimately, in mixed human-LLM ecosystems (15)”
§ Emergent social conventions and collective bias in LLM populations > INTRODUCTION
What the sources say
- Partly supportedcites [15] A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson, T. Graepel, Cooperative AI: Machines must learn to find common ground. Nature 593, 33– 36 (2021). 33947992 sourceExperimental — accuracy not yet measured
The paper says that multi-agent AI research has not yet tackled hard cooperation problems and calls for more work on interactions among AI systems and humans (Preprint, p. 3). It does not address how biases evolve through repeated communication, specifically among LLM agents or in mixed human-LLM ecosystems. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“the safety of a single LLM does not necessarily imply the safety of a multi-agent system (21)”
§ Emergent social conventions and collective bias in LLM populations > INTRODUCTION
What the sources say
- Supportedcites [21] U. Anwar, A. Saparov, J. Rando, D. Paleka, M. Turpin, P. Hase, E. S. Lubana, E. Jenner, S. Casper, O. Sourbut, B. L. Edelman, Z. Zhang, M. Günther, A. Korinek, J. Hernandez-Orallo, L. Hammond, E. Bigelow, A. Pan, L. Langosco, T. Korbak, H. Zhang, R. Zhong, Seán Ó hÉigeartaigh, G. Recchia, G. Corsi, A. Chan, M. Anderljung, L. Edwards, A. Petrov, Christian Schroeder de Witt, S. R. Motwan, Y. Bengio, D. Chen, P. H. S. Torr, S. Albanie, T. Maharaj, J. Foerster, F. Tramer, H. He, A. Kasirzadeh, Y. Choi, D. Krueger, Foundational challenges in assuring alignment and safety of large language models. a source
Section 2.6, “Multi-Agent Safety Is Not Assured by Single-Agent Safety,” explicitly says that single-agent alignment and safety are insufficient to assure desirable outcomes in multi-agent settings. This directly supports the citing sentence. (LLM, conf=0.95)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“these models pose serious risks, as the vast, unfiltered internet data used to train them can reinforce and amplify harmful biases, disproportionately harming marginalized communities (73).”
§ Emergent social conventions and collective bias in LLM populations > DISCUSSION
What the sources say
- Supportedcites [73] I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, N. K. Ahmed, Bias and fairness in large language models: A survey. arXiv:2309.00770 (2024). source
The Introduction (p. 2) says LLMs are typically trained on enormous amounts of uncurated Internet data and inherit harmful stereotypes and language that disproportionately affect marginalized communities. It also says LLMs can amplify existing biases and reinforce inequity. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“efforts to measure and instill human social norms in LLMs have so far yielded mixed results (78, 79)”
§ Emergent social conventions and collective bias in LLM populations > DISCUSSION
What the sources say
- Not checked — no full text of the sourcecites [78] Y. Yuan, K. Tang, J. Shen, M. Zhang, C. Wang, Measuring social norms of large language models. arXiv:2404.02491 (2024). source
- Supportedcites [79] K. Hämmerl, B. Deiseroth, P. Schramowski, J. Libovický, A. Fraser, K. Kersting, Do multilingual language models capture differing moral norms? arXiv:2203.09904 (2022). source
The discussion on p. 1 describes efforts to measure agreement with human values and align models with human-labelled data; it says one alignment effort showed promise while general alignment remains an open question. The initial experiments on p. 2 likewise report good correlation with human judgments for one model but generally no correlation for several others, supporting the characterization of mixed results. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims · 1 with their cited sources checked
Create a free account to see all 98 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 21
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Above c ∗ , two stable states coexist, and the single-agent accuracy p selects the destination (Abramiuk-Szurlej et al., 2021; Ashery et al., 2025).”
From Multi-agent discussion gains less when dissent is withheld (2026) · cites this paper as Ashery et al., 2025
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLM-driven populations display spontaneous emergent behaviours akin to human societies, such as scale-free networks [26], information diffusion [27], and social conventions through interactions [28].”
From Selective agreement, not sycophancy: investigating opinion dynamics in LLM interactions (2025) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“| 2 | 1: Q, M | 1: Q, M | 0.049 | 0.0951 | 0.487* | | 2 | 1: M, Q | 1: M, Q | 0.995 | 0.005 | 0.487* | | 2 | 1: Q, Q | 1: Q, Q | 0.997 | 0.003 | 0.487* | | 2 | 1: M, M | 1: M, M | 0.010 | 0.990 | 0.487* |”
§ Emergent social conventions and collective bias in LLM populations > RESULTS > Collective bias in convention selection
Reported 0.487 · recomputed 0.273275
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Emergent social conventions and collective bias in LLM populations > MATERIALS AND METHODS > Measuring individual bias
“reject the hypothesis that the model is biased if P<0.05.”
§ Emergent social conventions and collective bias in LLM populations > MATERIALS AND METHODS > Measuring individual bias
“reject the null hypothesis that the model is unbiased if P<0.05.”
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Statistical physics of social dynamics
Castellano, C., Fortunato, S., & Loreto, V. (2009) · Reviews of Modern Physics · doi:10.1103/revmodphys.81.591 · Cited by 4,119 · cited 1×
“Because collective processes can, in general, both suppress and amplify individual traits ( 16 , 17 ), answering this question is also relevant for practical applications.”
Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications
Mezard, M., Parisi, G., & Virasoro, M. (1986) · World Scientific Lecture Notes in Physics · doi:10.1142/0271 · Cited by 3,653
Some Effects of Proportions on Group Life: Skewed Sex Ratios and Responses to Token Women
Kanter, R. M. (1977) · American Journal of Sociology · doi:10.1086/226425 · Cited by 3,154 · cited 1×
“However, real-world observations reveal a wider range, with some studies proposing 30 to 40 % for gender conventions in corporate leadership ( 25 , 48 ), and others indicating that minorities as small as 0.3 % can trigger substantial linguistic and social changes ( 29 , 49 – 51 ).”
InfluentialThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Young, H. P. (1993) · Econometrica · doi:10.2307/2951778 · Cited by 2,399
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“Unlike studies that use LLMs to predict human responses in social science experiments ( 37 ) or to simulate human societies ( 38 – 40 ), our work does not treat LLMs as proxies for human participants but rather investigates how conventions emerge organically within a population of communicating AI…”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Research works printed but not linked: 4
- F. Hayek The Constitution of Liberty (University of Chicago Press 1960).
- D. Hume A Treatise of Human Nature (Oxford Univ. Press 2000).
- P. Michel M. Rita K. Mathewson O. Tieleman A. Lazaridou “Revisiting populations in multi-agent communication ” in The Eleventh International Conference on Learning Representations (ICLR 2023) pp. 1–16.
- I. Gemp R. Patel Y. Bachrach M. Lanctot V. Dasagi L. Marris G. Piliouras S. Liu K. Tuyls “Steering language models with game-theoretic solvers ” in Agentic Markets Workshop at ICML 2024 (ICML 2024) pp. 1–22.
Cited by
How malicious AI swarms can threaten democracy
Schroeder, D. T., Cha, M., Baronchelli, A., Bostrom, N., Christakis, N. A., Garcia, D., Goldenberg, A., Kyrychenko, Y., Leyton-Brown, K., Lutz, N., et al. (2026) · Science · doi:10.1126/science.adz1697 · Cited by 53
“This chorus erodes the independence essential to collective intelligence and democracy, already weakened by pervasive social influence operations on contemporary platforms.”
Self-citationLarge language models for spreading dynamics in complex systems
Jiang, S., Ren, H., Gao, Y., Zhang, Y., Li, Q., Xiao, D., Fan, J., Tang, R., & Wang, W. (2026) · Physics Reports · doi:10.1016/j.physrep.2026.01.006 · Cited by 10
Selective agreement, not sycophancy: investigating opinion dynamics in LLM interactions
Cau, E., Pansanella, V., Pedreschi, D., & Rossetti, G. (2025) · EPJ Data Science · doi:10.1140/epjds/s13688-025-00579-1 · Cited by 9
“LLM-driven populations display spontaneous emergent behaviours akin to human societies, such as scale-free networks [26], information diffusion [27], and social conventions through interactions [28].”
Evolvable AI: Threats of a new major transition in evolution
Müller, V., Steels, L., & Szathmáry, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2527700123 · Cited by 13
Reinforcement learning at the interface of artificial intelligence and cognitive science
Alkam, T., Tarshizi, E., & Benschoten, A. H. V. (2025) · Neuroscience · doi:10.1016/j.neuroscience.2025.09.004 · Cited by 8
Metadata
- Authors
- Ashery, A. F., Aiello, L. M., & Baronchelli, A.
- Affiliations
- Department of Mathematics, City St George’s, University of London, Northampton Square, London EC1V 0HB, UK.
- IT University of Copenhagen (DK)
- The Alan Turing Institute (GB)
- Corresponding author
- Baronchelli, Andrea
- Venue
- Science Advances
- Published
- 2025-05-16
- Volume
- 11
- Issue
- 20
- Pages
- eadu9368-eadu9368
- ISSN
- 2375-2548 (ISSN-L)
- PMC
- PMC12077490
- OpenAlex
- W4410358178
- Cited by
- 81
- References
- 54
- Citations by year
- 2026: 55 · 2025: 26
- FWCI
- 83.36
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- Science Advances (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by-nc)
- City Research Online (City University London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Opinion Dynamics and Social Influence
- Keywords
- social conventions, LLM agents, emergent behavior, AI alignment, minority influence, adversarial agents, social coordination
- MeSH terms
- Artificial Intelligence, Humans, Language, Models, Theoretical, Social Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Emergent social conventions and collective bias in LLM populations. https://app.cochat.ai/papers/10.1126/sciadv.adu9368
- The paper
- Ashery, A. F., Aiello, L. M., & Baronchelli, A. (2025). Emergent social conventions and collective bias in LLM populations. Science Advances. https://doi.org/10.1126/sciadv.adu9368
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.