Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Significance: The fact that humans enforce and comply with norms is an important reason why humans enjoy higher levels of cooperation and welfare than other animals. Some norms are relatively easy to explain: They may prohibit obviously harmful or uncooperative actions. But many norms are not easy to explain. For example, most cultures prohibit eating certain kinds of foods, and almost all societies have rules about what constitutes appropriate clothing, language, and gestures. Using a computational model focused on learning shows that apparently pointless rules can have an indirect effect on welfare. They can help agents learn how to enforce and comply with norms in general, improving the group’s ability to enforce norms that have a direct effect on welfare.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Instead, the use of direct and third-party punishment against norm violators has been proposed as a key factor in the emergence of large-scale non-kin cooperation observed within human societies [13, 15, 27, 31, 32].”
From Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems (2025) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“for example, agents might need to learn”
From Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems (2025) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They show that the frequency of unjust punishment in a population decreases in environments with "silly rules" that do not confer any benefits to individuals or society.”
From Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems (2025) · cites this paper as [27]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Although reputation and partner selection can facilitate the emergence of cooperation within small groups, they have a negligible impact on larger populations.”
From Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems (2025) · cites this paper as [27]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614 · cited 1×
“Formally, we consider multiagent RL in partially observable general sum Markov games (59, 60).”
BackgroundShapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,382
The evolution of altruistic punishment
Boyd, R., Gintis, H., Bowles, S., & Richerson, P. J. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0630443100 · Cited by 1,748 · cited 1×
“30–33 ).”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.01561 · Cited by 1,708 · cited 1×
“The neural networks were trained byreceivingimportance-weightedpolicyupdates(62)sampledfromaqueue oftrajectories.”
Punishment allows the evolution of cooperation (or anything else) in sizable groups
Boyd, R., & Richerson, P. J. (1992) · Ethology and Sociobiology · doi:10.1016/0162-3095(92)90032-y · Cited by 1,482 · cited 3×
“Group selection in this context would operate not at the level of individual norms but rather at the level of normative infrastructure: the practice of punishing what the group designates as punishable, whatever that might be (8).”
BackgroundCulture and conformity: A meta-analysis of studies using Asch's (1952b, 1956) line judgment task.
Bond, R., & Smith, P. B. (1996) · Psychological Bulletin · doi:10.1037//0033-2909.119.1.111 · Cited by 1,480 · cited 1×
“The mechanisms behind social learning in humans may be multiple: a psychological propensity to conformity (19, 20), deliberate teaching practices (21), and individual trial-and-error learning in conjunction with third-party punishment of failures to follow norms (22).”
BackgroundThe cultural niche: Why social learning is essential for human adaptation
Boyd, R., Richerson, P. J., & Henrich, J. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1100290108 · Cited by 1,303 · cited 2×
“Another account holds that silly rules are stable because, in any society, the survival of each generation depends on the transmission from prior generations of a large amount of culturally specific and causally opaque knowledge (12).”
BackgroundThe Evolution of Conformist Transmission and the Emergence of Between-Group Differences
Henrich, J., & Boyd, R. (1998) · Evolution and Human Behavior · doi:10.1016/s1090-5138(98)00018-x · Cited by 1,230 · cited 6×
“As Boyd et al. (12) emphasize, this is a critical pathway by which culture raises human well-being: through the transmission of cultural practices, such as the avoidance of harmful foods, even when agents lack direct causal awareness of why their practices are beneficial (12, 15).”
BackgroundInfluential
Research works printed but not linked: 6
- P. J. Richerson, R. Boyd, Not by Genes Alone: How Culture Transformed Human Evolution (University of Chicago Press, 2008).
- 10.1037/0033-2909.119.1.111 · DOI · Paper page
- J. S. Coleman, Foundations of Social Theory (Harvard University Press, 1994).
- J. W. Weibull, Evolutionary Game Theory (MIT Press, 1997).
- T. Eccles E. Hughes J. Kramár S. Wheelwright J. Z. Leibo Learning reciprocity in complex sequential social dilemmas. arXiv [Preprint] (2019). https://arxiv.com/abs/1903.08082 (Accessed 25 November 2021).
- E. Vinitsky . A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings. arXiv [Preprint] (2021). https://arxiv.com/abs/2106.09012 (Accessed 25 November 2021).
Cited by
Norm Dynamics: Interdisciplinary Perspectives on Social Norm Emergence, Persistence, and Change
Gelfand, M. J., Gavrilets, S., & Nunn, N. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-033020-013319 · Cited by 178
“Copying such arbitrary or “silly” norms may have a function: People are better able to learn about important functional norms when they are paired with silly rules, as this gives them more practice in rule enforcement (Köster et al. 2022).”
Reputation-Based Interaction Promotes Cooperation With Reinforcement Learning
Ren, T., & Zeng, X. (2023) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2023.3304911 · Cited by 48
“One major weakness of EGT, however, is its proclivity to oversimplify real-life social phenomena by reducing them to abstract choices, neglecting the underlying structures and self-learning adaptive behavior involved [22].”
BackgroundThe cognitive origin and cultural evolution of taboos in human societies
Hong, Z. (2024) · Journal of the Royal Anthropological Institute · doi:10.1111/1467-9655.14098 · Cited by 51
Vinitsky, E., Koster, R., Agapiou, J., Duéñez‐Guzmán, E. A., Vezhnevets, A. S., & Leibo, J. Z. (2023) · Collective Intelligence · doi:10.1177/26339137231162025 · Cited by 88
“In the real world, for every beneficial norm enabling collective action, there are hosts of unsavory norms (but see also (Hadfield-Menell et al., 2019; Köster et al., 2022)).”
BackgroundInfluentialSelf-citationAdebiyi, O. O. (2023) · Asian Journal of Economics Business and Accounting · doi:10.9734/ajeba/2023/v23i221153 · Cited by 37
Metadata
- Authors
- Köster, R., Hadfield-Menell, D., Everett, R., Weidinger, L., Hadfield, G. K., & Leibo, J. Z.
- Affiliations
- DeepMind, London EC4A 3TW, United Kingdom
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Google DeepMind (United Kingdom) (GB)
- University of Toronto (CA)
- Corresponding author
- Leibo, Joel Z.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2022-01-12
- Volume
- 119
- Issue
- 3
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8784148
- OpenAlex
- W4205912589
- Cited by
- 37
- References
- 50
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 7 · 2023: 14 · 2022: 5
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- TSpace (University of Toronto) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Keywords
- social norms, rule compliance, rule enforcement, taboos, multi-agent reinforcement learning, credit assignment, delayed rewards, artificial agents, normative behavior
- MeSH terms
- Environment, Humans, Learning, Reinforcement, Psychology, Social Norms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. https://app.cochat.ai/papers/10.1073/pnas.2106028118
- The paper
- Köster, R., Hadfield-Menell, D., Everett, R., Weidinger, L., Hadfield, G. K., & Leibo, J. Z. (2022). Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2106028118
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.