Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index; 31 of the 70 sources its claims cite were checked for support.

Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems

Open access
Dasgupta, Nayana, Musolesi, Mirco · 2025 · Autonomous Agents and Multi-Agent Systems

Abstract

Solving the problem of cooperation is fundamentally important for the creation and maintenance of functional societies. Problems of cooperation are omnipresent within human society, with examples ranging from navigating busy road junctions to negotiating treaties. As the use of AI becomes more pervasive throughout society, the need for socially intelligent agents capable of navigating these complex cooperative dilemmas is becoming increasingly evident. Direct punishment is a ubiquitous social mechanism that has been shown to foster the emergence of cooperation in both humans and non-humans. In the natural world, direct punishment is often strongly coupled with partner selection and reputation and used in conjunction with third-party punishment. The interactions between these mechanisms could potentially enhance the emergence of cooperation within populations. However, no previous work has evaluated the learning dynamics and outcomes emerging from multi-agent reinforcement learning populations that combine these mechanisms. This paper addresses this gap. It presents a comprehensive analysis and evaluation of the behaviors and learning dynamics associated with direct punishment, third-party punishment, partner selection, and reputation. Finally, we discuss the implications of using these mechanisms on the design of cooperative AI systems.

Overview

CoChat Summary

TL;DR

This paper systematically evaluates MARL populations playing the Iterated Prisoner’s Dilemma with combinations of direct punishment, third-party punishment, partner selection, and reputation. It finds that direct punishment promotes cooperation and yields the highest societal reward, while third-party punishment reaches higher cooperation and combining direct with third-party punishment achieves the highest cooperation fastest.

Key findings

  • Direct punishment is effective at promoting cooperation, but populations using direct punishment or direct punishment with partner selection converge to slightly lower cooperation than those using third-party punishment; populations combining direct and third-party punishment achieve the highest cooperation and converge fastest, applying just punishment within the first 100 episodes.
  • Populations using direct punishment with partner selection achieve the highest societal reward at convergence, followed closely by direct punishment alone, despite lower cooperation than third-party punishment populations.
  • Direct punishment populations experience early pervasive unjust punishment before the 140th episode, then learn to always perform just punishment; cooperation emerges rapidly at the 350th episode, and punishment falls to the lowest levels by the 475th episode.
  • Scheme 2, where just punishment yields a net profit of 2 reward units, lets all populations converge to cooperation; Scheme 1, where just punishment yields a net loss of 3 reward units, causes all populations to fail to converge to cooperation.

Methods

  • Simulations used N=5 agents in the main experiments, 2000 episodes with 10 rounds each, repeated 20 times; additional experiments tested population sizes 5, 10, 15, 20, 25, and 30. Agents played the Iterated Prisoner’s Dilemma with payoff matrix C,C=3,3; C,D=0,4; D,C=4,0; D,D=1,1.
  • Social mechanism combinations were TPP-S, TPP, DP-S, DP, TPPDP-S, and TPPDP. Each episode had up to three stages: partner selection or random matching, Prisoner’s Dilemma play, and punishment. Reputation was public and updated by play and punishment actions; just punishment increased reputation by 2, unjust punishment decreased it by 3, punishment cost the punisher 10 and the punished 3, and just punishment gave the punisher 7 (Scheme 1) or 12 (Scheme 2) reward.
  • Agents were independent DQN learners with up to three models for partner selection, dilemma play, and punishment; each model had one hidden layer of 128 ReLU neurons. Results report rolling means over 100 episodes with 95% confidence intervals across 20 repeats.

Limitations

  • The paper states that it assumes all agents have complete and accurate global reputations, that all reputation-contributing events maintain the same importance over an agent’s lifetime, and that the environment provides a top-down normative order defining just and unjust punishment; a more realistic model would let agents develop their own social norms.
  • The paper notes that in the combined third-party and direct punishment condition, both punishment types are controlled by a single punishment model, and future work could split this into specialized direct and third-party punishment models.
  • The text makes evident that the main experiments use a population of five agents, although Appendix C reports additional population sizes; reputation transfer is assumed complete and honest, isolating reputation from the complexity of agents learning to assign reputations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10458-025-09698-5

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

19 Valid · 5 Not valid · 9 Not checked · 76 Own results and interpretations

62 more claims · 3 with their cited sources checked

Create a free account to see all 109 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 19
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 62 references

Research works printed but not linked: 14

  1. Anastassacos, N., García, J., Hailes, S., & Musolesi, M. (2021). In Proceedings of the 20th international conference on autonomous agents and multiagent systems (AAMAS’21).
  2. Beckemeyer, N., Macke, W., & Sen, S. (2017). In G. Sukthankar, & J.A. Rodriguez-Aguilar (Eds.), Autonomous Agents and Multiagent Systems. Springer International Publishing.
  3. Beckemeyer, N., Macke, W., & Sen, S. (2017). In G. Sukthankar, & J.A. Rodriguez-Aguilar Autonomous agents and multiagent systems. Springer International Publishing
  4. Barclay, P., & Raihani, N. (2016). Partner choice versus punishment in human prisoner’s dilemmas. Evolution and Human Behavior, 37, 263–271. · DOI · Paper page
  5. Fu, F., Hauert, C., Nowak, M. A., & Wang, L. (2008). Reputation-based partner choice promotes cooperation in social networks. Physical Review E, 78(2), 026117. · DOI · Paper page
  6. Számadó, S., Balliet, D., Giardini, F., Power, E. A., & Takács, K. (2021). The language of cooperation: Reputation and honest signalling. Philosophical Transactions of the Royal Society B: Biological Sciences, 376, 20200286. · DOI
  7. Roberts, G., Raihani, N., Bshary, R., Manrique, H. M., Farina, A., Samu, F., & Barclay, P. (2021). The benefits of being seen to help others: Indirect reciprocity and reputation-based partner choice. Philosophical Transactions of the Royal Society B: Biological Sciences, 376, 20200290. · DOI
  8. Mahmoud, S., Miles, S., & Luck, M. (2016). In Proceedings of the 2016 international conference on autonomous agents & multiagent systems (International foundation for autonomous agents and multiagent systems, Richland, SC), AAMAS ’16
  9. Brooks, L. C., Iba, W., & Sen, S. (2011). In Twenty-second international joint conference on artificial intelligence
  10. Schmid, K., Belzner, L., & Linnhoff-Popien, C. (2021). In Proceedings of the 2021 conference on artificial life (ALIFE 2021) . 59.
  11. Barclay, P., & Willer, R. (2007). Partner choice creates competitive altruism in humans. Proceedings of the Royal Society B: Biological Sciences, 274, 749–753. · DOI · Paper page
  12. Sen, S., Gursel, A., & Airiau, S. (2007). In Working notes of the adaptive and learning agents workshop at AAMAS, vol. 7.
  13. Hadfield-Menell, D., Andrus, M., & Hadfield, G. In Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society (Association for Computing Machinery, New York, NY, USA, 2019), AIES ’19
  14. Leibo, J. Z., Zambaldi, V. F., Lanctot, M., Marecki, J., & Graepel, T. (2017). In K. Larson, M. Winikoff, S. Das, & E.H. Durfee Proceedings of the 16th Conference on Autonomous Agents and Multi Agent Systems (AAMAS 2017). ACM.

Cited by

See all 8

Metadata

Authors
Dasgupta, N., & Musolesi, M.
Affiliations
University College London (GB)
Corresponding author
Musolesi, Mirco
Venue
Autonomous Agents and Multi-Agent Systems
Published
2025-03-11
Volume
39
Issue
1
ISSN
  • 1387-2532 (ISSN-L)
  • 1573-7454
OpenAlex
W4408283893
Cited by
8
References
62
Citations by year
2026: 3 · 2025: 2
FWCI
8.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Autonomous Agents and Multi-Agent Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
Keywords
emergence of cooperation, multi-agent reinforcement learning, third-party punishment, mate choice, reputation, social mechanisms, learning dynamics, evolutionary game theory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems. https://app.cochat.ai/papers/10.1007/s10458-025-09698-5
The paper
Dasgupta, N., & Musolesi, M. (2025). Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems. Autonomous Agents and Multi-Agent Systems. https://doi.org/10.1007/s10458-025-09698-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.