Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 4 of the 5 sources its claims cite were checked for support.
Abstract
The significance of network structures in promoting group cooperation within social dilemmas has been widely recognized. Prior studies attribute this facilitation to the assortment of strategies driven by spatial interactions. Although reinforcement learning has been employed to investigate the impact of dynamic interaction on the evolution of cooperation, there remains a lack of understanding about how agents develop neighbour selection behaviours and the formation of strategic assortment within an explicit interaction structure. To address this, our study introduces a computational framework based on multi-agent reinforcement learning in the spatial Prisoner's Dilemma game. This framework allows agents to select dilemma strategies and interacting neighbours based on their long-term experiences, differing from existing research that relies on preset social norms or external incentives. By modelling each agent using two distinct Q-networks, we disentangle the coevolutionary dynamics between cooperation and interaction. The results indicate that long-term experience enables agents to develop the ability to identify non-cooperative neighbours and exhibit a preference for interaction with cooperative ones. This emergent self-organizing behaviour leads to the clustering of agents with similar strategies, thereby increasing network reciprocity and enhancing group cooperation.
Overview
CoChat Summary
TL;DR
The paper introduces a multi-agent reinforcement learning framework in a spatial Prisoner’s Dilemma where agents learn both dilemma strategies and neighbour-selection strategies from long-term experience. It finds that selective interaction with cooperative neighbours increases network reciprocity and group cooperation, outperforming an evolutionary game theory baseline.
Key findings
- RL agents maintain full cooperation until dilemma strength exceeds \(b > 1.2\), whereas the EGT baseline reaches only 54% cooperation even at \(b = 1.1\).
- As dilemma strength rises from \(b = 1.20\) to \(1.26\), cooperation falls from 0.987 to 0.294; from \(b = 1.20\) to \(1.22\), average payoff per episode drops from 2.67 to 2.25, with cooperative payoff falling from 3.52 to 3.41 and defector payoff from 1.44 to 1.10.
- Actual interaction frequency is 45.48% for defector–defector links but rises to 77.29% for cooperator–cooperator links, showing learned preference for cooperative partners.
- Longer memory improves cooperation: when dilemma strength increases from 1.2 to 1.22, agents recalling four timesteps maintain cooperation, while current-only agents drop from 0.64 to 0.19.
Methods
- Agents play a spatial Prisoner’s Dilemma on an \(L \times L\) square lattice with von Neumann neighbourhoods; each agent uses two DQN Q-networks, one for dilemma strategy and one for interaction selection, with replay memory, target networks, \(\epsilon\)-greedy exploration, and centralized training with decentralized execution.
- Training used 900 agents, 10 parallel arenas, 6,000 episodes of 10 timesteps each (60,000 steps), Adam optimizer, prioritized experience replay, and five replications; results average the final 10 episodes, with memory weight 0.6 (previous four rounds) unless stated.
- The utility function combines self-learning and social learning by comparing actual payoffs with counterfactual dilemma actions and average population payoffs; the EGT baseline uses Fermi-rule imitation of successful neighbours.
Limitations
- The paper does not present a dedicated limitations section; it states that integrating psychological complexity into the training framework is a promising future direction.
- Evident from the text: evaluation is simulation-only, restricted to a weak spatial PDG on a square lattice with fixed von Neumann neighbourhoods and specific parameters, with no human experiments or alternative network topologies tested.
- Evident from the text: results depend on hyperparameters such as memory length and on centralized training access to global average payoffs; scalability beyond 900 agents and 10 arenas is not tested.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
3 Valid · 0 Not valid · 3 Not checked · 50 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“our findings suggest that the performance of the learned interaction mechanism in promoting cooperation is attributed to its ability to help populations form homogeneous strategic clusters.”
What the sources say
- Partly supportedcites Szolnoki and Chen, 2020 sourceExperimental — accuracy not yet measured
The Results (pp. 6–8) show distinct strategic domains and report that selective cooperation can prevail over defection. But the Conclusion (p. 8) attributes that success to biased weakening at domain boundaries, which blocks defector invasion—not to helping populations form homogeneous strategic clusters. The paper also does not describe its investment rule as a learned interaction mechanism. (LLM, conf=0.82)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unlike existing RL cooperation studies, our approach does not rely on predefined social norms or external incentives [Ueshima et al. , 2023; Tennant et al. , 2023].”
What the sources say
- Partly supportedcites Ueshima et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction and Related Work describe earlier cooperation studies that use predetermined behavioral policies or incentives. Ueshima et al. also test fixed ALLC and TFT policies (Methods, §3.2.2), which supports the contrast about predefined behavior. But their Methods (§3) explicitly say they do not incorporate incentive mechanisms that maximize group benefits, so this citation does not support treating Ueshima et al. as an example of a study reliant on external incentives. (LLM, conf=0.85)
- Supportedcites Tennant et al., 2023 source
The Background and Related Work section says that investigating pre-defined moral norms in social dilemmas is the paper’s focus. Section 3.2 implements a Deontological agent that follows a specified cooperation norm, so Tennant et al. is an example of an RL study relying on predefined norms. The citation need not establish that it also uses external incentives. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“which in turn, aids the evolution of cooperative behaviors [Park et al. , 2022].”
What the sources say
- Partly supportedcites Park et al., 2022 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says cooperation can evolve through repeated interactions, and the Discussion (page unavailable) says reciprocity can maintain cooperation. Neither passage identifies the correlation referred to as “this correlation” or shows that it aids cooperation’s evolution, so the specific causal claim is unsupported. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a general decrease in the average payoff as the dilemma increases, which aligns with expectations, given that cooperators are primarily the contributors to the group payoff.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, RL agents employing additional interaction strategies exhibit two evident phases in their learning process, aligning with observations from previous EGT experiments [Wang et al. , 2013]: the END period and the EXP period.”
What the sources say
- Not checked — no full text of the sourcecites Wang et al., 2013 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
33 more claims
Create a free account to see all 56 claims and every source verdict.
- It may state two separate claims as one. 15
- Probably not a substantial claim. 5
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Prisoner's Dilemma Game · p. 3
“while mutual defection result in P = 0 . Unilateral cooperation against a defector incurs a cost S = c”
§ 3.1 Prisoner's Dilemma Game · p. 3
“In the PD, the conditions T > R > P > S and 2 R > T + S hold.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.5 Role of Long-term Experiences · p. 7
“Three memory weights were assessed, with α varying from 0.6 to 0, representing input scenarios based on observations from the last, second, and fourth rounds, respectively.”
§ 5.1 Experimental Setup · p. 5
“Unless otherwise stated, a memory weight of 0.6 is assigned, incorporating experiences from the previous four rounds as network input.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Prisoner's Dilemma Game · p. 3
“where mutual cooperation yields a reward R = b -c”
§ 3.1 Prisoner's Dilemma Game · p. 3
“Our model employs a weak PDG with T = b (1 ≤ b ≤ 2) , R = 1 , and P = S = 0 [Nowak and May, 1993]. Here, the parameter b directly assesses the strength of the dilemma, given c = 0 .”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We compute the gradient by using the Adam optimizer [Kingma and Ba, 2014] with a linear annealing schedule of learning rate.”
MethodHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 2×
“In this study, we construct a computational model using deep Q-learning (DQN) [Mnih et al. , 2015] to explore how agents can simultaneously learn both interaction and dilemma strategies from their long-term experiences.”
MethodFive Rules for the Evolution of Cooperation
Nowak, M. A. (2006) · Science · doi:10.1126/science.1133755 · Cited by 5,894 · cited 1×
“Notably, Nowak [2006] identifies five mechanisms central to understanding cooperation evolution.”
BackgroundMulti-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017) · arXiv · doi:10.48550/arxiv.1706.02275 · Cited by 3,377 · cited 2×
“During the training stage of our RL experiments, we employ a centralized training with decentralized execution approach [Lowe et al. , 2017].”
BackgroundMethodSchaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614 · cited 1×
“Within MARL, the IPD is conceptualized as a multi-agent extension of Markov decision processes (MDPs), termed partially observable general-sum Markov games [Littman, 1994].”
BackgroundEvolutionary prisoner’s dilemma game on a square lattice
Szabó, G., & Tőke, C. (1998) · Physical Review E · doi:10.1103/physreve.58.69 · Cited by 1,659 · cited 1×
“In EGT research, the Fermi rule is commonly used to model the dynamics of strategy evolution [Szab´o and T˝oke, 1998], reflecting social learning where neighbours tend to imitate the most successful policy observed.”
MethodStatistical physics of human cooperation
Perc, M., Jordan, J. J., Rand, D. G., Wang, Z., Boccaletti, S., & Szolnoki, A. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.05.004 · Cited by 1,590
Evolutionary dynamics of group interactions on structured populations: a review
Perc, M., Gómez-Gardeñes, J., Szolnoki, A., Floría, L. M., & Moreno, Y. (2013) · Journal of The Royal Society Interface · doi:10.1098/rsif.2012.0997 · Cited by 1,340
Rand, D. G., & Nowak, M. A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.003 · Cited by 1,308
Research works printed but not linked: 2
- [Leibo et al. , 2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multiagent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 464-473, 2017.
- [Rapoport et al. , 1965] Anatol Rapoport, Albert M Chammah, and Carol J Orwant. Prisoner's dilemma: A study in conflict and cooperation . University of Michigan Press, 1965.
Cited by
ROSBag MCP Server: Analyzing Robot Data with LLMs for Agentic Embodied AI Applications
Fu, L., Salimpour, S., Militano, L., Edelman, H., Queralta, J. P., & Toffetti, G. (2025) · International Conference on Robotic Computing and Communication (RoboticCC) · doi:10.1109/roboticcc68732.2025.00025 · Cited by 7
“Other relevant works in this area included non-robotics agents for games such as Minecraft [19], [20].”
MulPlanLM: multimodal robotic task planning with vision-language models and physical feedback
Son, Y.-C., Lee, D.-H., & Lim, S. (2026) · Intelligent Service Robotics · doi:10.1007/s11370-026-00743-0
Towards Autonomous Building Construction: A Multi-agent Framework Leveraging Large Multimodal Models
Soma, W., Ding, D., & Ito, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_17
MIMIC: Integrating Diverse Personality Traits for Better Game Testing Using Large Language Model
Chen, Y., Habchi, S., & Wei, L.-L. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00012 · Cited by 2
“We further evaluated MIMIC in the widely used real-world game Minecraft [17], comparing it against the state-of-the-art agent ODYSSEY [18].”
InfluentialMMVPS: A Multimodal Large Model-Based Visual Perception System
Li, Z., Xie, X., Zeng, W., Yan, X., GuoFeng, L. U. O., Hou, X., Wang, C., Xu, L., & Chen, L. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00180
“Odyssey [17] addresses goal decomposition, skill execution, and policy adaptation; Zheng et al. [18] proposed a multimodal end-to-end model improving agent vision; and MP5 [8] simulates human-like perception via active strategies.”
Metadata
- Authors
- Ren, T., & Zeng, X.-J.
- Affiliations
- University of Manchester (GB)
- Venue
- Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
- Published
- 2024-08
- Pages
- 193-201
- OpenAlex
- W4401024623
- Cited by
- 8
- References
- 47
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 7.07
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- University of Manchester
- Topics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- evolution of cooperation, spatial prisoner's dilemma, multi-agent reinforcement learning, neighbor selection, long-term experience, network reciprocity, social dilemmas, Q-network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning. https://app.cochat.ai/papers/10.24963/ijcai.2024/22
- The paper
- Ren, T., & Zeng, X.-J. (2024). Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning. Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2024/22
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.