The DOI matches its DataCite record.
Deconstructing Cooperation and Ostracism via Multi-Agent Reinforcement Learning
Open accessAbstract
Cooperation is challenging in biological systems, human societies, and multi-agent systems in general. While a group can benefit when everyone cooperates, it is tempting for each agent to act selfishly instead. Prior human studies show that people can overcome such social dilemmas while choosing interaction partners, i.e., strategic network rewiring. However, little is known about how agents, including humans, can learn about cooperation from strategic rewiring and vice versa. Here, we perform multi-agent reinforcement learning simulations in which two agents play the Prisoner's Dilemma game iteratively. Each agent has two policies: one controls whether to cooperate or defect; the other controls whether to rewire connections with another agent. This setting enables us to disentangle complex causal dynamics between cooperation and network rewiring. We find that network rewiring facilitates mutual cooperation even when one agent always offers cooperation, which is vulnerable to free-riding. We then confirm that the network-rewiring effect is exerted through agents' learning of ostracism, that is, connecting to cooperators and disconnecting from defectors. However, we also find that ostracism alone is not sufficient to make cooperation emerge. Instead, ostracism emerges from the learning of cooperation, and existing cooperation is subsequently reinforced due to the presence of ostracism. Our findings provide insights into the conditions and mechanisms necessary for the emergence of cooperation with network rewiring.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike existing RL cooperation studies, our approach does not rely on predefined social norms or external incentives [Ueshima et al. , 2023; Tennant et al. , 2023].”
From Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning (2024) · cites this paper as Ueshima et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. 2018. Emergent communication through negotiation. arXiv preprint arXiv:1804.03980 (2018). · arXiv
- Caroline Claus and Craig Boutilier. 1998. The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. In Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence (Madison, Wisconsin, USA) (AAAI '98/IAAI '98). American Association for Artificial Intelligence, USA, 746-752.
- Enrique Munoz de Cote, Alessandro Lazaric, and Marcello Restelli. 2006. Learning to cooperate in multi-agent social dilemmas. In Proceedings of the fifth interna- tional joint conference on Autonomous agents and multiagent systems (Hakodate, Japan) (AAMAS '06). Association for Computing Machinery, New York, NY, USA, 783-785.
- Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon White- son. 2016. Learning to communicate with deep multi-agent reinforcement learn- ing. Advances in neural information processing systems 29 (2016).
- Feng Fu, Christoph Hauert, Martin A Nowak, and Long Wang. 2008. Reputation- based partner choice promotes cooperation in social networks. Physical Review E 78, 2 (Aug. 2008), 026117-026118.
- Garrett Hardin. 1968. The Tragedy of the Commons. Science 162, 3859 (1968), 1243-1248.
- Christian Hilbe, Martin A Nowak, and Karl Sigmund. 2013. Evolution of extortion in Iterated Prisoner's Dilemma games. Proc. Natl. Acad. Sci. U. S. A. 110, 17 (April 2013), 6913-6918.
- Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac- Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kam- yar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Col- menarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. 2020. Acme: A Research Framework for Distributed Reinforcement Learning. arXiv preprint arXiv:2006.00979 (2020). https://arxiv.org/abs/2006.00979 · arXiv
- Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez- Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. 2018. Inequity aversion improves cooperation in intertem- poral social dilemmas. Advances in neural information processing systems 31 (2018).
- Spiros Kapetanakis and Daniel Kudenko. 2002. Reinforcement Learning of Coor- dination in Cooperative Multi-Agent Systems. In Eighteenth National Conference on Artificial Intelligence (Edmonton, Alberta, Canada). American Association for Artificial Intelligence, USA, 326-331.
- Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Gerald Tesauro, Murray Campbell, Sami Mourad, Golnaz Habibi, and Jonathan P How. 2020. Heterogeneous knowledge transfers via hierarchical teaching in cooperative multiagent reinforcement learning. In Proc of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020). 1-8.
- Raphael Köster, Dylan Hadfield-Menell, Richard Everett, Laura Weidinger, Gillian K Hadfield, and Joel Z Leibo. 2022. Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proc. Natl. Acad. Sci. U. S. A. 119, 3 (Jan. 2022).
- Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (São Paulo, Brazil) (AAMAS '17). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 464-473.
- Andrei Lupu and Doina Precup. 2020. Gifting in Multi-Agent Reinforcement Learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS '20). International Foundation for Au- tonomous Agents and Multiagent Systems, Richland, SC, 789-797.
- David Melamed, Ashley Harrell, and Brent Simpson. 2018. Cooperation, cluster- ing, and assortative mixing in dynamic networks. Proc. Natl. Acad. Sci. U. S. A. 115, 5 (Jan. 2018), 951-956.
- Martin A Nowak. 2006. Evolutionary Dynamics. Harvard University Press.
- Mancur Olson. 1965. The Logic of Collective Action: Public Goods and the Theory of Groups. Harvard University Press.
- Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. 2017. Deep decentralized multi-task multi-agent reinforcement learn- ing under partial observability. In International Conference on Machine Learning. PMLR, 2681-2690.
- Matjaž Perc and Attila Szolnoki. 2010. Coevolutionary games-A mini review. Biosystems. 99, 2 (Feb. 2010), 109-125.
- David G Rand, Samuel Arbesman, and Nicholas A Christakis. 2011. Dynamic social networks promote cooperation in experiments with humans. Proc. Natl. Acad. Sci. U. S. A. 108, 48 (Jan. 2011), 19193-19198.
- Mark Rowland, Shayegan Omidshafiei, Daniel Hennes, Will Dabney, Andrew Jaegle, Paul Muller, Julien Pérolat, and Karl Tuyls. 2021. Temporal difference and return optimism in cooperative multi-agent reinforcement learning. In AAMAS ALA Workshop.
- Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016. Prioritized Experience Replay. arXiv:1511.05952 [cs.LG] · arXiv
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). · arXiv
- Karl Tuyls and Ann Nowé. 2005. Evolutionary game theory and multi-agent reinforcement learning. The Knowledge Engineering Review 20, 1 (2005), 63-90.
- Hado van Hasselt, Arthur Guez, and David Silver. 2015. Deep Reinforcement Learning with Double Q-learning. arXiv:1509.06461 [cs.LG] · arXiv
Cited by
Sorted by citations
Ren, T., & Zeng, X.-J. (2024) · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/22 · Cited by 8
“Unlike existing RL cooperation studies, our approach does not rely on predefined social norms or external incentives [Ueshima et al. , 2023; Tennant et al. , 2023].”
Method
Metadata
- Authors
- Ueshima, A., Omidshafiei, S., & Shirado, H.
- Affiliations
- Tohoku University (JP)
- Google (United States) (US)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.04623
- OpenAlex
- W4387559255
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Keywords
- cooperation, ostracism, network rewiring, prisoner's dilemma, multi-agent reinforcement learning, social dilemmas, free-riding, strategic partner selection, emergence of cooperation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deconstructing Cooperation and Ostracism via Multi-Agent Reinforcement Learning. https://app.cochat.ai/papers/10.48550/arxiv.2310.04623
- The paper
- Ueshima, A., Omidshafiei, S., & Shirado, H. (2023). Deconstructing Cooperation and Ostracism via Multi-Agent Reinforcement Learning. arXiv. https://doi.org/10.48550/arxiv.2310.04623
