CoChat
Sign Up
PreprintCited by 4Multi-Agent Systems and NegotiationReinforcement Learning in RoboticsEvolutionary Game Theory and Cooperation

The DOI matches its DataCite record.

A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings

Open access
Vinitsky, Eugene, Köster, Raphael, Agapiou, John P., Duéñez-Guzmán, Edgar, Vezhnevets, Alexander Sasha, Leibo, Joel Z.·2021·arXiv
DOI 10.48550/arxiv.2106.09012arXiv 2106.09012
Open this paper in CoChat — free accountOpen PDF

Abstract

Society is characterized by the presence of a variety of social norms: collective patterns of sanctioning that can prevent miscoordination and free-riding. Inspired by this, we aim to construct learning dynamics where potentially beneficial social norms can emerge. Since social norms are underpinned by sanctioning, we introduce a training regime where agents can access all sanctioning events but learning is otherwise decentralized. This setting is technologically interesting because sanctioning events may be the only available public signal in decentralized multi-agent systems where reward or policy-sharing is infeasible or undesirable. To achieve collective action in this setting we construct an agent architecture containing a classifier module that categorizes observed behaviors as approved or disapproved, and a motivation to punish in accord with the group. We show that social norms emerge in multi-agent systems containing this agent and investigate the conditions under which this helps them achieve socially beneficial outcomes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.09012

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Humans are typically unable to share their reward signals or observations with other agents.

    “as humans are typically unable to share their reward”

    From Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems (2025) · cites this paper as [55]

    Related work (p. 3) says humans cannot publicize their instantaneous reward signals, supporting the reward-signal part of the claim. It does not say humans are typically unable to share their observations with other agents; the discussion of individual observations in §3.3 (p. 5) does not establish that point. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 75

  1. G. Akerlof. The economics of caste and of the rat race and other woeful tales. The Quarterly Journal of Economics , pages 599-617, 1976.
  2. B. Baker. Emergent reciprocity and team formation from randomized uncertain social preferences. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.
  3. L. Balafoutas, N. Nikiforakis, and B. Rockenbach. Direct and indirect punishment among strangers in the field. Proceedings of the National Academy of Sciences , 111(45):15924-15927, 2014.
  4. D. A. Baldwin. The power of positive sanctions. World Politics , 24(1):19-38, 1971.
  5. C. Bicchieri. The grammar of society: The nature and dynamics of social norms . Cambridge University Press, 2006.
  6. C. Bicchieri. Norms in the wild: How to diagnose, measure, and change social norms . Oxford University Press, 2016.
  7. C. Bicchieri, R. Muldoon, and A. Sontuoso. Social Norms. In The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Winter 2018 edition, 2018.
  8. C. Boehm. Moral origins: The evolution of virtue, altruism, and shame . Soft Skull Press, 2012.
  9. R. Boyd and S. Mathew. Arbitration supports reciprocity when there are frequent perception errors. Nature Human Behaviour , pages 1-8, 2021.
  10. M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. A. Seshia, P. Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 5175-5186, 2019.
  11. S. Çolak, A. Lima, and M. C. González. Understanding congested travel in urban areas. Nature communications , 7(1):1-8, 2016.
  12. J. W. Crandall, M. Oudah, F. Ishowo-Oloko, S. Abdallah, J.-F. Bonnefon, M. Cebrian, A. Shariff, M. A. Goodrich, I. Rahwan, et al. Cooperating with machines. Nature communications , 9(1):1-12, 2018.
  13. T. Eccles, E. Hughes, J. Kramár, S. Wheelwright, and J. Z. Leibo. Learning reciprocity in complex sequential social dilemmas. arXiv preprint arXiv:1903.08082 , 2019. · arXiv
  14. L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning , pages 1407-1416. PMLR, 2018.
  15. E. Fehr and U. Fischbacher. Social norms and human cooperation. Trends in cognitive sciences , 8(4): 185-190, 2004.
  16. J. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 122-130, 2018a.
  17. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018b.
  18. I. Gemp, K. R. McKee, R. Everett, E. A. Duéñez-Guzmán, Y. Bachrach, D. Balduzzi, and A. Tacchetti. D3c: Reducing the price of anarchy in multi-agent learning. arXiv preprint arXiv:2010.00575 , 2020. · arXiv
  19. H. Gintis. Social norms as choreography. politics, philosophy & economics , 9(3):251-264, 2010.
  20. M. Granovetter. Threshold models of collective behavior. American journal of sociology , 83(6): 1420-1443, 1978.
  21. G. K. Hadfield and B. R. Weingast. What is law? a coordination model of the characteristics of legal order. Journal of Legal Analysis , 4(2):471-514, 2012.
  22. G. K. Hadfield and B. R. Weingast. Law without the state: legal attributes and the coordination of decentralized collective punishment. Journal of Law and Courts , 1(1):3-34, 2013.
  23. G. K. Hadfield and B. R. Weingast. Microfoundations of the rule of law. Annual Review of Political Science , 17:21-42, 2014.
  24. D. Hadfield-Menell, M. Andrus, and G. Hadfield. Legible normativity for ai alignment: The value of silly rules. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 115-121, 2019.
  25. D. D. Heckathorn. The dynamics and dilemmas of collective action. American sociological review , pages 250-277, 1996.
  26. J. Henrich and M. Muthukrishna. The origins and psychology of human cooperation. Annual Review of Psychology , 72:207-240, 2021.
  27. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.
  28. E. Hughes, J. Z. Leibo, M. Phillips, K. Tuyls, E. A. Duéñez-Guzmán, A. G. Castañeda, I. Dunning, T. Zhu, K. R. McKee, R. Koster, H. Roff, and T. Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 3330-3340, 2018.
  29. S. Iqbal and F. Sha. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning. arXiv preprint arXiv:1905.12127 , 2019. · arXiv
  30. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.
  31. N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning , pages 3040-3049. PMLR, 2019.
  32. P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 , 2019. · arXiv
  33. H. H. Kelley, J. G. Holmes, N. L. Kerr, H. T. Reis, C. E. Rusbult, and P. A. Van Lange. An atlas of interpersonal situations . Cambridge University Press, 2003.
  34. R. Köster, K. R. McKee, R. Everett, L. Weidinger, W. S. Isaac, E. Hughes, E. A. Duéñez-Guzmán, T. Graepel, M. Botvinick, and J. Z. Leibo. Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences. arXiv preprint arXiv:2010.09054 , 2020. · arXiv
  35. R. Köster, D. Hadfield-Menell, R. Everett, L. Weidinger, G. K. Hadfield, and J. Z. Leibo. Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences , 119(3), 2022.
  36. A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 , 2019. · arXiv
  37. M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and T. Graepel. A unified game-theoretic approach to multiagent reinforcement learning. arXiv preprint arXiv:1711.00832 , 2017. · arXiv
  38. J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS 2017) , Sao Paulo, Brazil, 2017.
  39. J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742 , 2019. · arXiv
  40. J. Z. Leibo, E. A. Dueñez-Guzman, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning , pages 6187-6199. PMLR, 2021.
  41. D. Lewis. Convention . Harvard University Press, 1969.
  42. X. Liang, Y. Liu, T. Chen, M. Liu, and Q. Yang. Federated transfer reinforcement learning for autonomous driving. arXiv preprint arXiv:1910.06001 , 2019. · arXiv
  43. M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the 11th International Conference on Machine Learning (ICML) , pages 157-163, 1994.
  44. R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 6382-6393, 2017.
  45. A. Lupu and D. Precup. Gifting in multi-agent reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 789-797, 2020.
  46. G. Mackie. Ending footbinding and infibulation: A convention account. American sociological review , pages 999-1017, 1996.
  47. G. Marwell and P. Oliver. The Critical Mass in Collective Action . Cambridge University Press, 1993.
  48. S. McAleer, J. Lanier, P. Baldi, and R. Fox. Xdo: A double oracle algorithm for extensive-form games. arXiv preprint arXiv:2103.06426 , 2021. · arXiv
  49. J. L. McClelland, B. L. McNaughton, and R. C. O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review , 102(3):419, 1995.
  50. K. R. McKee, I. Gemp, B. McWilliams, E. A. Duèñez-Guzmán, E. Hughes, and J. Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 869-877, 2020.
  51. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.
  52. K. Nyborg, J. M. Anderies, A. Dannenberg, T. Lindahl, C. Schill, M. Schlüter, W. N. Adger, K. J. Arrow, S. Barrett, S. Carpenter, et al. Social norms as solutions. Science , 354(6308):42-43, 2016.
  53. M. Olson. The Logic of Collective Action: Public Goods and the Theory of Groups, Second Printing with a New Preface and Appendix , volume 124. Harvard University Press, 1965.
  54. A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. · arXiv
  55. E. Ostrom. A behavioral approach to the rational choice theory of collective action: Presidential address, american political science association, 1997. American political science review , 92(1):1-22, 1998.
  56. E. Ostrom. Understanding institutional diversity . Princeton university press, 2009.
  57. J. Pérolat, J. Z. Leibo, V. F. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 3643-3652, 2017.
  58. A. Peysakhovich and A. Lerer. Prosocial learning agents solve generalized stag hunts better than selfish ones. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 2043-2044, 2018.
  59. E. A. Platanios, A. Saparov, and T. Mitchell. Jelly bean world: A testbed for never-ending learning. In International Conference on Learning Representations , 2019.
  60. T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4295-4304. PMLR, 2018.
  61. S. Sen and S. Airiau. Emergence of norms through social learning. In IJCAI , volume 1507, page 1512, 2007.
  62. L. S. Shapley. Stochastic Games. In Proc. of the National Academy of Sciences of the United States of America , 1953.
  63. Y. Shoham and M. Tennenholtz. On the emergence of social conventions: modeling, analysis, and simulations. Artificial Intelligence , 94(1-2):139-166, 1997.
  64. S. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In Proceedings of the 17th International Conference on Neural Information Processing Systems , pages 1281-1288, 2004.
  65. K. O. Stanley. Why open-endedness matters. Artificial life , 25(3):232-235, 2019.
  66. P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 2085-2087, 2018.
  67. M. Tomasello and A. Vaish. Origins of human cooperation and morality. Annual review of psychology , 64:231-255, 2013.
  68. E. Ullmann-Margalit. The emergence of norms . Oxford University Press, 1977/2015.
  69. P. Vanderschraaf. Convention as correlated equilibrium. Erkenntnis , 42(1):65-87, 1995.
  70. A. Vezhnevets, Y. Wu, M. Eckstein, R. Leblond, and J. Z. Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In International Conference on Machine Learning , pages 9733-9742. PMLR, 2020.
  71. J. X. Wang, E. Hughes, C. Fernando, W. M. Czarnecki, E. A. Duéñez-Guzmán, and J. Z. Leibo. Evolving intrinsic motivations for altruistic behavior. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pages 683-692, 2019.
  72. W. Z. Wang, M. Beliaev, E. Biyik, D. A. Lazar, R. Pedarsani, and D. Sadigh. Emergent prosociality in multi-agent games through gifting. In 30th International Joint Conference on Artificial Intelligence (IJCAI) , 8 2021.
  73. P. Wiessner. Norm enforcement among the ju hoansi bushmen. Human Nature , 16(2):115-145, 2005.
  74. E. Xiao and D. Houser. Emotion expression in human punishment behavior. Proceedings of the National Academy of Sciences , 102(20):7398-7401, 2005.
  75. H. P. Young. The evolution of social norms. economics , 7(1):359-387, 2015.

Cited by

Sorted by citations

Related

Metadata

Authors
Vinitsky, E., Köster, R., Agapiou, J. P., Duéñez-Guzmán, E., Vezhnevets, A. S., & Leibo, J. Z.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.09012
arXiv
2106.09012
OpenAlex
W4287117786
Cited by
2
References
0
Citations by year
2024: 1 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
Keywords
social norms, sanctioning, multi-agent systems, decentralized learning, norm formation, collective action, free-riding, punishment, public signals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review