CoChat

Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

Open access
Köster, Raphael, Hadfield-Menell, Dylan, Everett, Richard, Weidinger, Laura, Hadfield, Gillian K., Leibo, Joel Z. · 2022 · Proceedings of the National Academy of Sciences

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

Abstract

Significance: The fact that humans enforce and comply with norms is an important reason why humans enjoy higher levels of cooperation and welfare than other animals. Some norms are relatively easy to explain: They may prohibit obviously harmful or uncooperative actions. But many norms are not easy to explain. For example, most cultures prohibit eating certain kinds of foods, and almost all societies have rules about what constitutes appropriate clothing, language, and gestures. Using a computational model focused on learning shows that apparently pointless rules can have an indirect effect on welfare. They can help agents learn how to enforce and comply with norms in general, improving the group’s ability to enforce norms that have a direct effect on welfare.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2106028118

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Schematic overview of the experimental conditions. In the no rules condition, agents collect berries for reward. One berry color is poisonous and, after a time delay, reduces reward obtained from consumed berries. Being poisoned is invisible to all players and a hard credit assignment problem. In the important rule condition, eating the poisonous berry is a social taboo. When eaten, the player who ate the berry immediately gets marked, which is only visible to other agents. Other agents can collect a reward by punishing a marked agent. In the silly rule condition, the same taboo against the poisonous berry is in place, but, additionally, there is an identical taboo on a berry that is not poisonous. Therefore, there are two taboos for which agents can experience punishment by others. Our experiment sets out to study the effects of these different normative schemes.
Fig. 1. Schematic overview of the experimental conditions. In the no rules condition, agents collect berries for reward. One berry color is poisonous and, after a time delay, reduces reward obtained from consumed berries. Being poisoned is invisible to all players and a hard credit assignment problem. In the important rule condition, eating the poisonous berry is a social taboo. When eaten, the player who ate the berry immediately gets marked, which is only visible to other agents. Other agents can collect a reward by punishing a marked agent. In the silly rule condition, the same taboo against the poisonous berry is in place, but, additionally, there is an identical taboo on a berry that is not poisonous. Therefore, there are two taboos for which agents can experience punishment by others. Our experiment sets out to study the effects of these different normative schemes.CC BY-NC-ND · PubMed Central
Fig. 2. Depiction of the environment. The agents inhabit a 2D world. Agents earn a reward for eating berries, which regrow probabilistically after being harvested. One type of berry is poisonous, and, if collected by an agent, it diminishes the agent’s ability to gather rewards from other berries, after a delay period. If an agent eats one of the poisonous berries in the important rule condition, the agent immediately gets marked and appears in a different color to the other agents. In the silly rule condition, one additional, nonpoisonous, berry also triggers an agent’s marking. Agents are able to punish each other using a “punishing beam,” causing a loss to themselves and a larger loss to the punished agent. If a marked agent is punished, the punishing agent receives a significant reward.
Fig. 2. Depiction of the environment. The agents inhabit a 2D world. Agents earn a reward for eating berries, which regrow probabilistically after being harvested. One type of berry is poisonous, and, if collected by an agent, it diminishes the agent’s ability to gather rewards from other berries, after a delay period. If an agent eats one of the poisonous berries in the important rule condition, the agent immediately gets marked and appears in a different color to the other agents. In the silly rule condition, one additional, nonpoisonous, berry also triggers an agent’s marking. Agents are able to punish each other using a “punishing beam,” causing a loss to themselves and a larger loss to the punished agent. If a marked agent is punished, the punishing agent receives a significant reward.CC BY-NC-ND · PubMed Central
Fig. 3. Agent architecture and training procedure. Agents learn together in one population of 12 agents, 8 of which are selected to play in one episode in order to generate experiences (in multiple parallel environments). Each agent contains an independent neural network that receives a batch of its own experiences from these environments to update its neuronal weights. The inputs the agents receive are the raw pixels from their field of view. The neural network architecture of each agent learns abstract representations that parse the observations in a way that allows the agent to take reward-maximizing action. The output of neural network on each time step is a prediction of the value of the current state and an action (movement or zap). The expected received reward and actual received reward are compared, and the network weights are gradually adjusted to maximize long-term cumulative reward.
Fig. 3. Agent architecture and training procedure. Agents learn together in one population of 12 agents, 8 of which are selected to play in one episode in order to generate experiences (in multiple parallel environments). Each agent contains an independent neural network that receives a batch of its own experiences from these environments to update its neuronal weights. The inputs the agents receive are the raw pixels from their field of view. The neural network architecture of each agent learns abstract representations that parse the observations in a way that allows the agent to take reward-maximizing action. The output of neural network on each time step is a prediction of the value of the current state and an action (movement or zap). The expected received reward and actual received reward are compared, and the network weights are gradually adjusted to maximize long-term cumulative reward.CC BY-NC-ND · PubMed Central
Fig. 4. We are examining per episode group-level metrics about agent populations (y axis) over the trajectory of learning (x axis in time steps). We plot the average trajectory per condition (and standard error of the mean). (A) Number of times unmarked agents are punished (agents that have not broken a taboo). (B) Number of times marked agents are punished (agents that have broken a taboo). (C) Fraction of time spent marked after breaking a taboo. (D) Fraction of time agents spent poisoned. (E) The number of “taboo” berries eaten (poisonous and nonpoisonous combined, if available in the condition). (F) Total sum of reward gained by group (including costs of punishing). In total, we observe a benefit of the silly rule condition in the intermediate stages of learning, driven by an increased ability to avoid poisonous berries. We also see a temporal order to learned behaviors, for example, an increase in social punishment that then declines together with a decrease in number of taboo berries eaten.
Fig. 4. We are examining per episode group-level metrics about agent populations (y axis) over the trajectory of learning (x axis in time steps). We plot the average trajectory per condition (and standard error of the mean). (A) Number of times unmarked agents are punished (agents that have not broken a taboo). (B) Number of times marked agents are punished (agents that have broken a taboo). (C) Fraction of time spent marked after breaking a taboo. (D) Fraction of time agents spent poisoned. (E) The number of “taboo” berries eaten (poisonous and nonpoisonous combined, if available in the condition). (F) Total sum of reward gained by group (including costs of punishing). In total, we observe a benefit of the silly rule condition in the intermediate stages of learning, driven by an increased ability to avoid poisonous berries. We also see a temporal order to learned behaviors, for example, an increase in social punishment that then declines together with a decrease in number of taboo berries eaten.CC BY-NC-ND · PubMed Central
Fig. 5. Higher rates of punishment in early stages are related to less time spent poisoned later in training. Each marker is an independent population of agents. On the y axis, we plot how often players are punished in an episode early in training (between time steps 0 and 2e8). On the x axis, we plot the mean time players spend poisoned in later episodes (time steps 2e8 to 4e8). The results are consistent with the interpretation that a high peak in punishment early in training is followed by more avoidance of the poisoned berry later.
Fig. 5. Higher rates of punishment in early stages are related to less time spent poisoned later in training. Each marker is an independent population of agents. On the y axis, we plot how often players are punished in an episode early in training (between time steps 0 and 2e8). On the x axis, we plot the mean time players spend poisoned in later episodes (time steps 2e8 to 4e8). The results are consistent with the interpretation that a high peak in punishment early in training is followed by more avoidance of the poisoned berry later.CC BY-NC-ND · PubMed Central
Fig. 6. Single target probes or “zero-shot generalization.” (A) Depiction of probe. An agent is placed in an empty room with just one other object (berry or agent), and we measure how many time steps it takes before either the agent interacts with the object (eats the berry or zaps the player) or time expires (30 time steps). Note that, in all following plots, the y axis is reversed: The origin represents that the maximal amount of time steps passed (30), and the top represents that the minimal amount of time steps passed (zero) until the episode is terminated. (B–D) Berry types across the three different conditions. Agents learn to avoid berries that are taboo. Lines depict the mean across populations of how quickly the agent interacts with the object (y axis) over learning (x axis). Error bars represent SEM over different independent populations. (E and F) Difference between important rule and silly rule for approaching the poisoned berry (same as in C and D) and punishing the marked player. Agents are faster to learn to avoid the poisoned berry and punish taboos in the silly rule condition. (G) Early punishing (mean 0 to 2e8 steps) of the marked player is associated with reduced consumption of the poisoned berry (mean 2e8 to 4e8 steps) later in training.
Fig. 6. Single target probes or “zero-shot generalization.” (A) Depiction of probe. An agent is placed in an empty room with just one other object (berry or agent), and we measure how many time steps it takes before either the agent interacts with the object (eats the berry or zaps the player) or time expires (30 time steps). Note that, in all following plots, the y axis is reversed: The origin represents that the maximal amount of time steps passed (30), and the top represents that the minimal amount of time steps passed (zero) until the episode is terminated. (B–D) Berry types across the three different conditions. Agents learn to avoid berries that are taboo. Lines depict the mean across populations of how quickly the agent interacts with the object (y axis) over learning (x axis). Error bars represent SEM over different independent populations. (E and F) Difference between important rule and silly rule for approaching the poisoned berry (same as in C and D) and punishing the marked player. Agents are faster to learn to avoid the poisoned berry and punish taboos in the silly rule condition. (G) Early punishing (mean 0 to 2e8 steps) of the marked player is associated with reduced consumption of the poisoned berry (mean 2e8 to 4e8 steps) later in training.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 60 references

Research works printed but not linked: 6

  1. P. J. Richerson, R. Boyd, Not by Genes Alone: How Culture Transformed Human Evolution (University of Chicago Press, 2008).
  2. 10.1037/0033-2909.119.1.111 · DOI · Paper page
  3. J. S. Coleman, Foundations of Social Theory (Harvard University Press, 1994).
  4. J. W. Weibull, Evolutionary Game Theory (MIT Press, 1997).
  5. T. Eccles E. Hughes J. Kramár S. Wheelwright J. Z. Leibo Learning reciprocity in complex sequential social dilemmas. arXiv [Preprint] (2019). https://arxiv.com/abs/1903.08082 (Accessed 25 November 2021).
  6. E. Vinitsky . A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings. arXiv [Preprint] (2021). https://arxiv.com/abs/2106.09012 (Accessed 25 November 2021).

Cited by

See all 38

Metadata

Authors
Köster, R., Hadfield-Menell, D., Everett, R., Weidinger, L., Hadfield, G. K., & Leibo, J. Z.
Affiliations
  • DeepMind, London EC4A 3TW, United Kingdom
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Google DeepMind (United Kingdom) (GB)
  • University of Toronto (CA)
Corresponding author
Leibo, Joel Z.
Venue
Proceedings of the National Academy of Sciences
Published
2022-01-12
Volume
119
Issue
3
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4205912589
Cited by
37
References
50
Citations by year
2026: 4 · 2025: 7 · 2024: 7 · 2023: 14 · 2022: 5
FWCI
2.71
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • TSpace (University of Toronto) (submittedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
Keywords
social norms, rule compliance, rule enforcement, taboos, multi-agent reinforcement learning, credit assignment, delayed rewards, artificial agents, normative behavior
MeSH terms
Environment, Humans, Learning, Reinforcement, Psychology, Social Norms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. https://app.cochat.ai/papers/10.1073/pnas.2106028118
The paper
Köster, R., Hadfield-Menell, D., Everett, R., Weidinger, L., Hadfield, G. K., & Leibo, J. Z. (2022). Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2106028118

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.