CoChat

Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning

Open access
Ren, Tianyu, Zeng, Xiao-Jun · 2024 · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 4 of the 5 sources its claims cite were checked for support.

Abstract

The significance of network structures in promoting group cooperation within social dilemmas has been widely recognized. Prior studies attribute this facilitation to the assortment of strategies driven by spatial interactions. Although reinforcement learning has been employed to investigate the impact of dynamic interaction on the evolution of cooperation, there remains a lack of understanding about how agents develop neighbour selection behaviours and the formation of strategic assortment within an explicit interaction structure. To address this, our study introduces a computational framework based on multi-agent reinforcement learning in the spatial Prisoner's Dilemma game. This framework allows agents to select dilemma strategies and interacting neighbours based on their long-term experiences, differing from existing research that relies on preset social norms or external incentives. By modelling each agent using two distinct Q-networks, we disentangle the coevolutionary dynamics between cooperation and interaction. The results indicate that long-term experience enables agents to develop the ability to identify non-cooperative neighbours and exhibit a preference for interaction with cooperative ones. This emergent self-organizing behaviour leads to the clustering of agents with similar strategies, thereby increasing network reciprocity and enhancing group cooperation.

Overview

CoChat Summary

TL;DR

The paper introduces a multi-agent reinforcement learning framework in a spatial Prisoner’s Dilemma where agents learn both dilemma strategies and neighbour-selection strategies from long-term experience. It finds that selective interaction with cooperative neighbours increases network reciprocity and group cooperation, outperforming an evolutionary game theory baseline.

Key findings

  • RL agents maintain full cooperation until dilemma strength exceeds \(b > 1.2\), whereas the EGT baseline reaches only 54% cooperation even at \(b = 1.1\).
  • As dilemma strength rises from \(b = 1.20\) to \(1.26\), cooperation falls from 0.987 to 0.294; from \(b = 1.20\) to \(1.22\), average payoff per episode drops from 2.67 to 2.25, with cooperative payoff falling from 3.52 to 3.41 and defector payoff from 1.44 to 1.10.
  • Actual interaction frequency is 45.48% for defector–defector links but rises to 77.29% for cooperator–cooperator links, showing learned preference for cooperative partners.
  • Longer memory improves cooperation: when dilemma strength increases from 1.2 to 1.22, agents recalling four timesteps maintain cooperation, while current-only agents drop from 0.64 to 0.19.

Methods

  • Agents play a spatial Prisoner’s Dilemma on an \(L \times L\) square lattice with von Neumann neighbourhoods; each agent uses two DQN Q-networks, one for dilemma strategy and one for interaction selection, with replay memory, target networks, \(\epsilon\)-greedy exploration, and centralized training with decentralized execution.
  • Training used 900 agents, 10 parallel arenas, 6,000 episodes of 10 timesteps each (60,000 steps), Adam optimizer, prioritized experience replay, and five replications; results average the final 10 episodes, with memory weight 0.6 (previous four rounds) unless stated.
  • The utility function combines self-learning and social learning by comparing actual payoffs with counterfactual dilemma actions and average population payoffs; the EGT baseline uses Fermi-rule imitation of successful neighbours.

Limitations

  • The paper does not present a dedicated limitations section; it states that integrating psychological complexity into the training framework is a promising future direction.
  • Evident from the text: evaluation is simulation-only, restricted to a weak spatial PDG on a square lattice with fixed von Neumann neighbourhoods and specific parameters, with no human experiments or alternative network topologies tested.
  • Evident from the text: results depend on hyperparameters such as memory length and on centralized training access to global average payoffs; scalability beyond 900 agents and 10 arenas is not tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.24963/ijcai.2024/22

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Training framework for developing dilemma and interaction strategies. Each iteration involves agent i choosing a dilemma strategy and selecting neighbouring agents for PDG engagement. Each agent uses two Q-networks: the dilemma policy network, which processes long-term actions in dilemmas by the agent and its neighbours, and the interaction selection network, which assesses neighbours' dilemma actions alongside the agent's previous interactions. The agent calculates the utility of its actions based on the rewards accumulated from past encounters.
Figure 1 Training framework for developing dilemma and interaction strategies. Each iteration involves agent i choosing a dilemma strategy and selecting neighbouring agents for PDG engagement. Each agent uses two Q-networks: the dilemma policy network, which processes long-term actions in dilemmas by the agent and its neighbours, and the interaction selection network, which assesses neighbours' dilemma actions alongside the agent's previous interactions. The agent calculates the utility of its actions based on the rewards accumulated from past encounters.From the PDF
Figure 2 RL-based training approach (ours) promotes cooperation more effectively than the EGT (baseline) method. The EGT (orange) represents agents solely calculating cumulative payoffs and adjusting dilemma actions through social learning. In contrast, the implementation of effective dilemma and selection policies, guided by RL (blue), has significantly enhanced the level of cooperation within the population. Our RL-based method maintains full cooperation in the population until dilemma strength exceeds 1 . 2 .
Figure 2 RL-based training approach (ours) promotes cooperation more effectively than the EGT (baseline) method. The EGT (orange) represents agents solely calculating cumulative payoffs and adjusting dilemma actions through social learning. In contrast, the implementation of effective dilemma and selection policies, guided by RL (blue), has significantly enhanced the level of cooperation within the population. Our RL-based method maintains full cooperation in the population until dilemma strength exceeds 1 . 2 .From the PDF
Figure 3 The evolution of cooperation and associated payoffs across varying dilemma strengths. In all scenarios, the fraction of cooperators first decreases and then increases over time, coinciding with reduced average individual payoffs and increased inequality as dilemma strength intensifies. The evaluation encompasses evolutionary trajectories of (a) cooperation level and (b) the Gini Coefficient, alongside metrics including (c) average group payoffs and (d)-(e) payoffs for trained cooperators and defectors, with dilemma strength varying from b = 1 . 20 to 1 . 26 .
Figure 3 The evolution of cooperation and associated payoffs across varying dilemma strengths. In all scenarios, the fraction of cooperators first decreases and then increases over time, coinciding with reduced average individual payoffs and increased inequality as dilemma strength intensifies. The evaluation encompasses evolutionary trajectories of (a) cooperation level and (b) the Gini Coefficient, alongside metrics including (c) average group payoffs and (d)-(e) payoffs for trained cooperators and defectors, with dilemma strength varying from b = 1 . 20 to 1 . 26 .From the PDF
Figure 4 Temporal evolution of strategy connectivity and actual link interactions. RL agents demonstrate enhanced interaction capabilities, increasing connections with cooperative neighbours. (a) The average connectivity ratio for cooperators and defectors in the first half of the total timestep. (b) The frequency of actual link connections between dilemma strategies during the first ten episodes. The dilemma strength is set to b = 1 . 20 .
Figure 4 Temporal evolution of strategy connectivity and actual link interactions. RL agents demonstrate enhanced interaction capabilities, increasing connections with cooperative neighbours. (a) The average connectivity ratio for cooperators and defectors in the first half of the total timestep. (b) The frequency of actual link connections between dilemma strategies during the first ten episodes. The dilemma strength is set to b = 1 . 20 .From the PDF
Figure 5 Snapshots of the spatial evolution of strategies and their connections. Cooperative individuals resist defector incursions by forming and expanding clusters. Panels (a)-(d) depict strategy distributions; (e)-(h) illustrate corresponding strategy connections at identical timesteps. Pixels represent agents as cooperators (blue) and defectors (red), with strategy connectivity ratio varying from 0 (shallow) to 1 (deep). The results are obtained for b = 1 . 20 .
Figure 5 Snapshots of the spatial evolution of strategies and their connections. Cooperative individuals resist defector incursions by forming and expanding clusters. Panels (a)-(d) depict strategy distributions; (e)-(h) illustrate corresponding strategy connections at identical timesteps. Pixels represent agents as cooperators (blue) and defectors (red), with strategy connectivity ratio varying from 0 (shallow) to 1 (deep). The results are obtained for b = 1 . 20 .From the PDF
Figure 6 Average cooperation level and effective connection for cooperators with varying experience lengths. Incorporating longer experiences as input enhances group cooperation and cooperative interactions. (a) Post-training cooperation levels in the population. (b) Frequency of cooperators participating in PDG across the initial 20 episodes. These results are obtained for four dilemma strength scenarios, with memory weight α ranging from 0 to 0.6.
Figure 6 Average cooperation level and effective connection for cooperators with varying experience lengths. Incorporating longer experiences as input enhances group cooperation and cooperative interactions. (a) Post-training cooperation levels in the population. (b) Frequency of cooperators participating in PDG across the initial 20 episodes. These results are obtained for four dilemma strength scenarios, with memory weight α ranging from 0 to 0.6.From the PDF

Verification

Claims the paper makes

3 Valid · 0 Not valid · 3 Not checked · 50 Own results and interpretations

33 more claims

Create a free account to see all 56 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 15
  • Probably not a substantial claim. 5

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 49 references

Research works printed but not linked: 2

  1. [Leibo et al. , 2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multiagent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 464-473, 2017.
  2. [Rapoport et al. , 1965] Anatol Rapoport, Albert M Chammah, and Carol J Orwant. Prisoner's dilemma: A study in conflict and cooperation . University of Michigan Press, 1965.

Cited by

See all 8

Metadata

Authors
Ren, T., & Zeng, X.-J.
Affiliations
University of Manchester (GB)
Venue
Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
Published
2024-08
Pages
193-201
OpenAlex
W4401024623
Cited by
8
References
47
Citations by year
2026: 4 · 2025: 4
FWCI
7.07
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • Research Explorer (The University of Manchester) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
University of Manchester
Topics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
evolution of cooperation, spatial prisoner's dilemma, multi-agent reinforcement learning, neighbor selection, long-term experience, network reciprocity, social dilemmas, Q-network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning. https://app.cochat.ai/papers/10.24963/ijcai.2024/22
The paper
Ren, T., & Zeng, X.-J. (2024). Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning. Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2024/22

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.