The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index; 31 of the 70 sources its claims cite were checked for support.
Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems
Open accessAbstract
Solving the problem of cooperation is fundamentally important for the creation and maintenance of functional societies. Problems of cooperation are omnipresent within human society, with examples ranging from navigating busy road junctions to negotiating treaties. As the use of AI becomes more pervasive throughout society, the need for socially intelligent agents capable of navigating these complex cooperative dilemmas is becoming increasingly evident. Direct punishment is a ubiquitous social mechanism that has been shown to foster the emergence of cooperation in both humans and non-humans. In the natural world, direct punishment is often strongly coupled with partner selection and reputation and used in conjunction with third-party punishment. The interactions between these mechanisms could potentially enhance the emergence of cooperation within populations. However, no previous work has evaluated the learning dynamics and outcomes emerging from multi-agent reinforcement learning populations that combine these mechanisms. This paper addresses this gap. It presents a comprehensive analysis and evaluation of the behaviors and learning dynamics associated with direct punishment, third-party punishment, partner selection, and reputation. Finally, we discuss the implications of using these mechanisms on the design of cooperative AI systems.
Overview
CoChat Summary
TL;DR
This paper systematically evaluates MARL populations playing the Iterated Prisoner’s Dilemma with combinations of direct punishment, third-party punishment, partner selection, and reputation. It finds that direct punishment promotes cooperation and yields the highest societal reward, while third-party punishment reaches higher cooperation and combining direct with third-party punishment achieves the highest cooperation fastest.
Key findings
- Direct punishment is effective at promoting cooperation, but populations using direct punishment or direct punishment with partner selection converge to slightly lower cooperation than those using third-party punishment; populations combining direct and third-party punishment achieve the highest cooperation and converge fastest, applying just punishment within the first 100 episodes.
- Populations using direct punishment with partner selection achieve the highest societal reward at convergence, followed closely by direct punishment alone, despite lower cooperation than third-party punishment populations.
- Direct punishment populations experience early pervasive unjust punishment before the 140th episode, then learn to always perform just punishment; cooperation emerges rapidly at the 350th episode, and punishment falls to the lowest levels by the 475th episode.
- Scheme 2, where just punishment yields a net profit of 2 reward units, lets all populations converge to cooperation; Scheme 1, where just punishment yields a net loss of 3 reward units, causes all populations to fail to converge to cooperation.
Methods
- Simulations used N=5 agents in the main experiments, 2000 episodes with 10 rounds each, repeated 20 times; additional experiments tested population sizes 5, 10, 15, 20, 25, and 30. Agents played the Iterated Prisoner’s Dilemma with payoff matrix C,C=3,3; C,D=0,4; D,C=4,0; D,D=1,1.
- Social mechanism combinations were TPP-S, TPP, DP-S, DP, TPPDP-S, and TPPDP. Each episode had up to three stages: partner selection or random matching, Prisoner’s Dilemma play, and punishment. Reputation was public and updated by play and punishment actions; just punishment increased reputation by 2, unjust punishment decreased it by 3, punishment cost the punisher 10 and the punished 3, and just punishment gave the punisher 7 (Scheme 1) or 12 (Scheme 2) reward.
- Agents were independent DQN learners with up to three models for partner selection, dilemma play, and punishment; each model had one hidden layer of 128 ReLU neurons. Results report rolling means over 100 episodes with 95% confidence intervals across 20 repeats.
Limitations
- The paper states that it assumes all agents have complete and accurate global reputations, that all reputation-contributing events maintain the same importance over an agent’s lifetime, and that the environment provides a top-down normative order defining just and unjust punishment; a more realistic model would let agents develop their own social norms.
- The paper notes that in the combined third-party and direct punishment condition, both punishment types are controlled by a single punishment model, and future work could split this into specialized direct and third-party punishment models.
- The text makes evident that the main experiments use a population of five agents, although Appendix C reports additional population sizes; reputation transfer is assumed complete and honest, isolating reputation from the complexity of agents learning to assign reputations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Valid · 5 Not valid · 9 Not checked · 76 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“it has also been observed in non-human populations [13, 32]”
What the sources say
- Not checked — no full text of the sourcecites [13] Redhead, D., Dhaliwal, N., & Cheng, J. T. (2021). Taking charge and stepping in: Individuals who punish are rewarded with prestige and dominance. Social and Personality Psychology Compass. https://doi.org/10.1111/spc3.12581 source
- Supportedcites [32] Kenward, B., & Östh, T. (2012). Enactment of third-party punishment by 4-year-olds. Frontiers in Psychology, 3, 373. source
The Introduction explicitly says that direct punishment occurs in non-human animals, whereas third-party punishment had, at the time, only been identified in humans. This supports the distinction attributed to the source. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“populations that combine the use of direct punishment with commitment achieve higher proportions of cooperation than if commitment was used alone [35, 47].”
§ 8 Direct Punishment and Emergence of Cooperation in MARL Systems > 2.6 Combinations of Social Mechanisms
What the sources say
- Supportedcites [35] Han, T. A. (2016). Emergence of social punishment and cooperation through prior commitments. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v30i1.10120 source
The Concluding Remarks (p. 6) state that when both punishment and commitment are available, cooperation is significantly higher than when either option is absent, including commitment alone. The finding is in the paper’s one-shot prisoner’s dilemma model. (LLM, conf=0.94)
- Partly supportedcites [47] Han, T. A., Santos, F. C., Lenaerts, T., & Pereira, L. M. (2015). Synergy between intention recognition and commitments in cooperation dilemmas. Scientific Reports, 5, 9312. sourceExperimental — accuracy not yet measured
The Results (page not available) report higher cooperation when intention recognition is combined with conditional commitment than with commitment alone. They do not test direct punishment combined with commitment; the Discussion (page not available) mentions punishment only when describing a different experiment. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“unjust punishment of cooperators is per-”
What the sources say
- Not checked — no full text of the sourcecites [34] Pleasant, A., & Barclay, P. (2018). Why hate the good guy? Antisocial punishment of high cooperators is greater when people compete to be chosen. Psychological Science, 29, 868–876. source
- Could not verifycites [72] Rand, D. G., Armao IV, J. J., Nakamaru, M., & Ohtsuki, H. (2010). Anti-social punishment can prevent the co-evolution of punishment and cooperation. Journal of Theoretical Biology, 265(4), 624–632. https://doi.org/10.1016/j.jtbi.2010.06.010 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) calls punishment of cooperators ‘anti-social’ and a surprising behavior, while the Discussion describes its harmful effects on cooperation. Neither section reports whether people perceive such punishment unfavorably; searches for perceptions and negative reactions found no such passage. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, there is also evidence that unjust punishment can be restrained by commitment mechanisms [35, 47].”
§ 8 Direct Punishment and Emergence of Cooperation in MARL Systems > 2.5 Unjust Punishment
What the sources say
- Supportedcites [35] Han, T. A. (2016). Emergence of social punishment and cooperation through prior commitments. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v30i1.10120 source
The Concluding Remarks (p. 6) explicitly state that commitment-proposing agents can significantly restrain antisocial punishers. The claim paraphrases that finding in the paper’s one-shot prisoner’s-dilemma setting. (LLM, conf=0.94)
- Could not verifycites [47] Han, T. A., Santos, F. C., Lenaerts, T., & Pereira, L. M. (2015). Synergy between intention recognition and commitments in cooperation dilemmas. Scientific Reports, 5, 9312. sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) reports that intention recognition was associated with little or no punishment of unintentional defection in a separate experiment. The paper's own results concern commitments and cooperation, not whether commitment mechanisms restrain unjust punishment. Searches for unjust punishment and commitment mechanisms found no passage establishing that link. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As performing punishment is costly for the punisher in the natural world”
What the sources say
- Not checked — no full text of the sourcecites [13] Redhead, D., Dhaliwal, N., & Cheng, J. T. (2021). Taking charge and stepping in: Individuals who punish are rewarded with prestige and dominance. Social and Personality Psychology Compass. https://doi.org/10.1111/spc3.12581 source
- Partly supportedcites [14] Barclay, P., & Raihani, N. (2016). Partner choice versus punishment in human Prisoner’s Dilemmas. Evolution and Human Behavior, 37(4), 263–271. https://doi.org/10.1016/j.evolhumbehav.2015.12.004 sourceExperimental — accuracy not yet measured
The Discussion (lines 215–230; page not provided) describes partner rejection as potentially costly to the person doing the punishing, because choosing a fairer partner could mean forgoing a higher payoff. This supports the cost element, but the passage concerns an experimental game and does not establish that performing punishment is costly in the natural world. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
62 more claims · 3 with their cited sources checked
Create a free account to see all 109 claims and every source verdict.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 19
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“• Target Update ∈ {2x | x ∈ [500, 5001), x%500 = 0}”
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“Table A1 Hyperparameters used for all experiments, with a target update of 200, a discount rate of 0.9 and a batch size of 100.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“• Batch Size ∈ [2x | x ∈ [10, 19)]”
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“Table A1 Hyperparameters used for all experiments, with a target update of 200, a discount rate of 0.9 and a batch size of 100.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“• Minimum ϵ = np.linspace(1e-4, 1, num=10)”
§ 34 Direct Punishment and Emergence of Cooperation in MARL Systems
“Selection Model 131072 0.0001 0.8889 0.3 0.01 Playing Model 131072 0.01 0.8889 0.3 0.1 Punishing Model 524288 0.2 0.8889 0.5 0.001”
Citations
Cites
Axelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083 · cited 1×
“The proportion of cooperators selected per episode at convergence is the same for all the different punishment mechanisms shown in Figure 10.”
Five Rules for the Evolution of Cooperation
Nowak, M. A. (2006) · Science · doi:10.1126/science.1133755 · Cited by 5,894 · cited 1×
“However, the development of socially intelligent agents is challenging, as cooperation is a complex cognitive skill [25] and the mechanisms influencing its evolution are still not well understood [3, 4], despite being extensively studied in the past [8, 29, 30, 43].”
Connolly, T., & Axelrod, R. (1984) · Administrative Science Quarterly · doi:10.2307/2392957 · Cited by 3,890
Evolution of indirect reciprocity by image scoring
Nowak, M. A., & Sigmund, K. (1998) · Nature · doi:10.1038/31225 · Cited by 2,700 · cited 1×
“Anti-social punishment can be detrimental to the evolution of cooperation, as the punishment of cooperators further reduces the incentives for it and encourages instead pervasive defection [31, 45, 46].”
Evolutionary Dynamics: Exploring the Equations of Life
NOWAK, M. A. (2006) · doi:10.2307/j.ctvjghw98 · Cited by 2,438
An Evolutionary Approach to Norms
Axelrod, R. (1986) · American Political Science Review · doi:10.1017/s0003055400185016 · Cited by 2,205 · cited 1×
“An individual’s reputation can be seen as a record of their prior behaviours, formed from direct observations or through indirect communication mechanisms, such as gossip [22–25].”
Antisocial Punishment Across Societies
Herrmann, B., Thöni, C., & Gächter, S. (2008) · Science · doi:10.1126/science.1153808 · Cited by 2,038 · cited 1×
“Anti-social punishment can be detrimental to the evolution of cooperation, as the punishment of cooperators further reduces the incentives for cooperation and encourages pervasive defection [20, 32, 36].”
Dawes, R. M. (1980) · Annual Review of Psychology · doi:10.1146/annurev.ps.31.020180.001125 · Cited by 2,022
Social dilemmas: the anatomy of cooperation
Kollock, P. (1998) · Annual Review of Sociology · doi:10.1146/annurev.soc.24.1.183 · Cited by 1,746
Research works printed but not linked: 14
- Anastassacos, N., García, J., Hailes, S., & Musolesi, M. (2021). In Proceedings of the 20th international conference on autonomous agents and multiagent systems (AAMAS’21).
- Beckemeyer, N., Macke, W., & Sen, S. (2017). In G. Sukthankar, & J.A. Rodriguez-Aguilar (Eds.), Autonomous Agents and Multiagent Systems. Springer International Publishing.
- Beckemeyer, N., Macke, W., & Sen, S. (2017). In G. Sukthankar, & J.A. Rodriguez-Aguilar Autonomous agents and multiagent systems. Springer International Publishing
- Barclay, P., & Raihani, N. (2016). Partner choice versus punishment in human prisoner’s dilemmas. Evolution and Human Behavior, 37, 263–271. · DOI · Paper page
- Fu, F., Hauert, C., Nowak, M. A., & Wang, L. (2008). Reputation-based partner choice promotes cooperation in social networks. Physical Review E, 78(2), 026117. · DOI · Paper page
- Számadó, S., Balliet, D., Giardini, F., Power, E. A., & Takács, K. (2021). The language of cooperation: Reputation and honest signalling. Philosophical Transactions of the Royal Society B: Biological Sciences, 376, 20200286. · DOI
- Roberts, G., Raihani, N., Bshary, R., Manrique, H. M., Farina, A., Samu, F., & Barclay, P. (2021). The benefits of being seen to help others: Indirect reciprocity and reputation-based partner choice. Philosophical Transactions of the Royal Society B: Biological Sciences, 376, 20200290. · DOI
- Mahmoud, S., Miles, S., & Luck, M. (2016). In Proceedings of the 2016 international conference on autonomous agents & multiagent systems (International foundation for autonomous agents and multiagent systems, Richland, SC), AAMAS ’16
- Brooks, L. C., Iba, W., & Sen, S. (2011). In Twenty-second international joint conference on artificial intelligence
- Schmid, K., Belzner, L., & Linnhoff-Popien, C. (2021). In Proceedings of the 2021 conference on artificial life (ALIFE 2021) . 59.
- Barclay, P., & Willer, R. (2007). Partner choice creates competitive altruism in humans. Proceedings of the Royal Society B: Biological Sciences, 274, 749–753. · DOI · Paper page
- Sen, S., Gursel, A., & Airiau, S. (2007). In Working notes of the adaptive and learning agents workshop at AAMAS, vol. 7.
- Hadfield-Menell, D., Andrus, M., & Hadfield, G. In Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society (Association for Computing Machinery, New York, NY, USA, 2019), AIES ’19
- Leibo, J. Z., Zambaldi, V. F., Lanctot, M., Marecki, J., & Graepel, T. (2017). In K. Larson, M. Winikoff, S. Das, & E.H. Durfee Proceedings of the 16th Conference on Autonomous Agents and Multi Agent Systems (AAMAS 2017). ACM.
Cited by
Emergence of cooperation in the one-shot Prisoner’s dilemma through Discriminatory and Samaritan AIs
Zimmaro, F., Miranda, M., Fernández, J. M. R., López, J. A. M., Reddel, M., Widler, V., Antonioni, A., & Han, T. A. (2024) · Journal of The Royal Society Interface · doi:10.1098/rsif.2024.0212 · Cited by 30
Bidirectional Influence and Interaction for Multiagent Reinforcement Learning
Sun, S., Xu, K., Feng, D., & Ding, B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3401649 · Cited by 6
“Potential-based reward shaping provides additional rewards to guide agents toward subgoals, promoting collaboration and reducing conflicts by aligning with the overall task objective [12].”
BackgroundLeaders and Collaborators: Addressing Sparse Reward Challenges in Multi-Agent Reinforcement Learning
Sun, S., Liu, H., Xu, K., & Ding, B. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3488772 · Cited by 3
“Researchers have employed techniques such as reward shaping [9], and punishment[10]toencouragediversityandexploratorybehavior andpreventagentsfromexhibitingoverlysimilarbehavioratthe startoftraining.”
Evolution of Coordination Through Institutional Incentives: An Evolutionary Game Theory Approach
Ogbo, N. B., Song, Z., & Han, T. A. (2025) · International Conference on Distributed Artificial Intelligence (DAI) · doi:10.1145/3772429.3772434 · Cited by 6
“The strategic use of rewards and sanctions serve as key instruments in fostering and sustaining cooperation within structured interactions [7, 12, 18, 30, 35, 68].”
A Large Language Model-Enabled Framework for Simulating Multi-Agent Cooperative Game
Tao, X., Shen, Q., Pu, Q., Wang, Q., & Yang, F. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401542
“…mechanism design, role division-based systems [8], two-stage communication incentive methods [9], and punishment-reputation coordination mechanisms [10] either ignore dynamic updates of individual trust evaluation or fail to integrate payoff differences and random perturbations for strategy…”
Metadata
- Authors
- Dasgupta, N., & Musolesi, M.
- Affiliations
- University College London (GB)
- Corresponding author
- Musolesi, Mirco
- ORCID
- Musolesi, Mirco
- Venue
- Autonomous Agents and Multi-Agent Systems
- Published
- 2025-03-11
- Volume
- 39
- Issue
- 1
- ISSN
- 1387-2532 (ISSN-L)
- 1573-7454
- OpenAlex
- W4408283893
- Cited by
- 8
- References
- 62
- Citations by year
- 2026: 3 · 2025: 2
- FWCI
- 8.71
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Autonomous Agents and Multi-Agent Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Keywords
- emergence of cooperation, multi-agent reinforcement learning, third-party punishment, mate choice, reputation, social mechanisms, learning dynamics, evolutionary game theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems. https://app.cochat.ai/papers/10.1007/s10458-025-09698-5
- The paper
- Dasgupta, N., & Musolesi, M. (2025). Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems. Autonomous Agents and Multi-Agent Systems. https://doi.org/10.1007/s10458-025-09698-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.