CoChat

Deep reinforcement learning can promote sustainable human behaviour in a common-pool resource problem

Open access
Koster, Raphael, Pîslar, Miruna, Tacchetti, Andrea, Balaguer, Jan, Liu, Leqi, Elie, Romuald, Hauser, Oliver P., Tuyls, Karl, Botvinick, Matt, Summerfield, Christopher · 2025 · Nature Communications

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 5 of the 41 sources its claims cite were checked for support.

Abstract

A canonical social dilemma arises when resources are allocated to people, who can either reciprocate with interest or keep the proceeds. The right resource allocation mechanisms can encourage levels of reciprocation that sustain the commons. Here, in an iterated multiplayer trust game, we use deep reinforcement learning (RL) to design a social planner that promotes sustainable contributions from human participants. We first trained neural networks to behave like human players, creating a stimulated economy that allows us to study the dynamics of receipt and reciprocation. We use RL to train a mechanism to maximise aggregate return to players. The RL mechanism discovers a redistributive policy that leads to a large but also more equal surplus. The mechanism outperforms baseline mechanisms by conditioning its generosity on available resources and temporarily sanctioning defectors. Examining the RL policy allows us to develop a similar but explainable mechanism that is more popular among players.

Overview

CoChat Summary

TL;DR

In an iterated multiplayer trust game with a common-pool resource, deep reinforcement learning designed a social planner that promoted sustainable contributions from human participants, producing higher surplus and lower inequality than baseline mechanisms. A simpler explainable interpolating mechanism approximated the RL policy, achieved comparable or slightly lower surplus with higher equality, and was preferred by players.

Key findings

  • In Exp. 1, the RL agent generated a surplus ~150% greater than the highest baseline (proportional) with a Gini coefficient just over ~0.2; it produced higher surplus than proportional (z = 3.25, p = 0.001), mixed (z = 4.38, p < 0.001), and equal (z = 6.31, p < 0.001) baselines, and lower Gini than proportional (z = 2.78, p < 0.01) but not than equal or mixed. 65% of RL-agent games were sustained to the end with at least one player, and 55% of these with all four players.
  • The RL policy was more egalitarian when the pool was abundant and more punitive when resources were scarce; it temporarily excluded defectors for ~1–5 rounds and then made more generous re-inclusion offers. Lagged regressions showed the agent’s offer predicted subsequent reciprocation rather than the reverse.
  • The explainable interpolating baseline, with mixing parameter w = (R/200)^22, matched the RL agent on surplus in Exp. 2 (both > proportional, z = 2.9, p < 0.01 and z = 2.52, p < 0.01; no difference between them) and had lower Gini than both proportional (z = 4.17, p < 0.001) and the RL agent (z = 3.67, p < 0.001). Human players preferred the interpolating agent as fairer, more understandable, and more prone to encourage cooperation.
  • In Exp. 3, a retrained RL agent (M2) achieved higher surplus than the interpolating baseline (z = 2.35, p < 0.05), but the interpolating baseline generated lower Gini (z = 5.64, p < 0.001). In Exp. 4, M2 produced larger surplus than a proportional baseline with augmented instructions in game 1 (z = 2.22, p = 0.026), game 2 (z = 4.99, p < 0.001), and game 3 (z = 6.08, p < 0.001), and surplus grew under M2 from game 1 to 3 (z = 2.84, p = 0.005).

Methods

  • The game was an infinitely repeated multiplayer trust game with p = 4 players, initial pool R0 = 200, growth factor r = 0.4, and a maximum pool size; each round the mechanism allocated endowments from the common pool, players chose how much to reciprocate, and the mechanism objective was to maximize aggregate player surplus over rounds and players.
  • Human participants were recruited online via Prolific Academic; the final dataset contains 4,952 participants. Exp. 1 recruited 640 participants (160 groups of 4), with 40 groups each under equal (w = 1), mixed (w = 0.5), proportional (w = 0), and RL-agent mechanisms; later experiments used 40–80 groups per condition. No demographic data were collected.
  • The pipeline collected human data under baseline mechanisms, trained recurrent neural network behavioral clones via imitation learning (BC1 and BC2), trained deep RL social planners using graph neural networks and policy gradient methods to maximize surplus against virtual players, then evaluated the mechanisms with new human participants. Analyses used Wilcoxon rank-sum tests, Gini coefficients, exclusion durations, and lagged regressions.

Limitations

  • Stated in the paper: the participant cohort was drawn exclusively from the UK and USA, and no fine-grained demographic data were collected, so generalizability across gender, age, or other variables is unknown.
  • Stated in the paper: the game differs from the classic CPR problem because players decide what to keep or reciprocate from a pre-set allocation rather than drawing down freely from the common pool; the mechanism’s success in this stylized game does not guarantee similar principles in more complex, naturalistic environments.
  • Evident from the text: the RL policy remains hard to understand or explain, motivating the interpolating approximation; results come from a stylized online game with no communication or self-organization among players, and about 10% of games were excluded because players dropped out.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-025-58043-7

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Figures

Fig. 1 Illustration of the game. A Two rounds (denoted t) are illustrated (columns). In each, a mechanism allocates resources (blue flowers) from a pool of size R to p=4 players, who each choose a quantity to reciprocate, with any remainder going to surplus (gold coins). For example, in the schematic, in round t the first player (left) receives 2 flowers, and reciprocates 1, generating a surplus of r=0.4 (amount due to growth factor shown in grey). The pool size is depleted by the allocation and replenished by the reciprocations. Note that players who receive no resources cannot reciprocate (e.g. centre left player on round t) (B) Illustration of our approach. First, (1) we collected data from human participants under a range of mechanisms defined by different values of w, and used imitation learning to create clones that behaved like people. Then (2) we used these clones to train the RL agent, and (3) conducted Exp.1, in which we compared the RL agent to baselines. Next, (4) we analysed the RL agent policy, and constructed a heuristic approximation that was more explainable (the ‘interpolation baseline’), which (5) we tested on behavioural clones, and (6) compared to the RL agent (and proportional mechanism) in Exp. 2. Finally, (7) we used all of the data so far to retrain a new version of the RL agent, and (8) compared it to the interpolation baseline in Exp. 3. C Example games (using behavioural clones). The game starts with R0=200. Left: offers (full lines) and reciprocations (dashed lines) to four players (lower panels) over 40 trials (x-axis) in an example game with the equal baseline. The grey shaded area indicates where each player receives an offer of zero. The top panel shows the size of the pool (blue) and the total per-trial surplus (red). The middle and right panels show example games under the proportional baseline and the RL agent, respectively. Note that in the example proportional baseline game, three players fall into poverty traps, leaving a single player to contribute, and increasing inequality.
Fig. 1 Illustration of the game. A Two rounds (denoted t) are illustrated (columns). In each, a mechanism allocates resources (blue flowers) from a pool of size R to p=4 players, who each choose a quantity to reciprocate, with any remainder going to surplus (gold coins). For example, in the schematic, in round t the first player (left) receives 2 flowers, and reciprocates 1, generating a surplus of r=0.4 (amount due to growth factor shown in grey). The pool size is depleted by the allocation and replenished by the reciprocations. Note that players who receive no resources cannot reciprocate (e.g. centre left player on round t) (B) Illustration of our approach. First, (1) we collected data from human participants under a range of mechanisms defined by different values of w, and used imitation learning to create clones that behaved like people. Then (2) we used these clones to train the RL agent, and (3) conducted Exp.1, in which we compared the RL agent to baselines. Next, (4) we analysed the RL agent policy, and constructed a heuristic approximation that was more explainable (the ‘interpolation baseline’), which (5) we tested on behavioural clones, and (6) compared to the RL agent (and proportional mechanism) in Exp. 2. Finally, (7) we used all of the data so far to retrain a new version of the RL agent, and (8) compared it to the interpolation baseline in Exp. 3. C Example games (using behavioural clones). The game starts with R0=200. Left: offers (full lines) and reciprocations (dashed lines) to four players (lower panels) over 40 trials (x-axis) in an example game with the equal baseline. The grey shaded area indicates where each player receives an offer of zero. The top panel shows the size of the pool (blue) and the total per-trial surplus (red). The middle and right panels show example games under the proportional baseline and the RL agent, respectively. Note that in the example proportional baseline game, three players fall into poverty traps, leaving a single player to contribute, and increasing inequality.CC BY · PubMed Central
Fig. 2 Results of first trained mechanism against baselines. A The surplus and Gini coefficient generated from games played under three baseline mechanisms (blue, purple and red) and the RL agent (green), for virtual players (left panel) and human participants (right panel) in Exp. 1. Each small dot is a game; the larger dot is the mean over games. B Correspondences between predicted outcomes (from virtual players, shading) and observed outcomes in Exp.1 (dots) for each baseline mechanism and the RL agent. Shown separately for games that were sustained to the end by at least one player (colours) and those where the pool was exhausted prematurely (grey). C The average Gini coefficient of the offer made to players as a function of the pool size, for individual trials (grey dots) and games (coloured dots), both for behavioural clones (upper panels) and human data in Exp.1 (lower panels). D Exclusions occur when a player receives nothing for one or more consecutive rounds. Here, we plot the duration of exclusions against the trial on which they are initiated (dots). Points on the diagonal indicate that the player was never reincluded (exclusion lasts until trial 40). The superimposed coloured histograms show the count of exclusions for each duration bin (of width 2). Note that unlike baselines the RL agent excludes frequently, but for short durations. E The offer made by each mechanism to each player as a function of the lagged contribution of that player over adjacent trials. Dots are individual coefficients; black line is the median.
Fig. 2 Results of first trained mechanism against baselines. A The surplus and Gini coefficient generated from games played under three baseline mechanisms (blue, purple and red) and the RL agent (green), for virtual players (left panel) and human participants (right panel) in Exp. 1. Each small dot is a game; the larger dot is the mean over games. B Correspondences between predicted outcomes (from virtual players, shading) and observed outcomes in Exp.1 (dots) for each baseline mechanism and the RL agent. Shown separately for games that were sustained to the end by at least one player (colours) and those where the pool was exhausted prematurely (grey). C The average Gini coefficient of the offer made to players as a function of the pool size, for individual trials (grey dots) and games (coloured dots), both for behavioural clones (upper panels) and human data in Exp.1 (lower panels). D Exclusions occur when a player receives nothing for one or more consecutive rounds. Here, we plot the duration of exclusions against the trial on which they are initiated (dots). Points on the diagonal indicate that the player was never reincluded (exclusion lasts until trial 40). The superimposed coloured histograms show the count of exclusions for each duration bin (of width 2). Note that unlike baselines the RL agent excludes frequently, but for short durations. E The offer made by each mechanism to each player as a function of the lagged contribution of that player over adjacent trials. Dots are individual coefficients; black line is the median.CC BY · PubMed Central
Fig. 3 Results of first trained mechanisms against novel, interpretable mechanism. A The family of exponential functions that determine w as a function of pool size (grey lines) and the one that produced the highest surplus with virtual players (yellow line; (R/200)²²). B The surplus and Gini coefficient generated from games played under two baseline mechanisms (proportional, red; interpolating, yellow) and the RL agent (green), for virtual players (left panel) and human participants (right panel) in Exp. 2. Each small dot is a game; the larger dot is the mean over games. C Same as Fig. 2C but for Proportional, Interpolating and RL agents. D Same as Fig. 2E. E Same as Fig. 2D.
Fig. 3 Results of first trained mechanisms against novel, interpretable mechanism. A The family of exponential functions that determine w as a function of pool size (grey lines) and the one that produced the highest surplus with virtual players (yellow line; (R/200)²²). B The surplus and Gini coefficient generated from games played under two baseline mechanisms (proportional, red; interpolating, yellow) and the RL agent (green), for virtual players (left panel) and human participants (right panel) in Exp. 2. Each small dot is a game; the larger dot is the mean over games. C Same as Fig. 2C but for Proportional, Interpolating and RL agents. D Same as Fig. 2E. E Same as Fig. 2D.CC BY · PubMed Central
Fig. 4 Results of second trained mechanism. A The surplus and Gini coefficient generated from games played under the interpolating baseline (yellow) and the RL agent M2 (light green), for virtual players (left panel) and human participants (right panel) in Exp. 3. Each small dot is a game; the larger dot is the mean over games. B Average reported preference for the interpolating (yellow) and RL agent M2 (green) agents on a number of dimensions. C Same as A; Results from human participants in Exp. 4., in which groups of players play three consecutive games.
Fig. 4 Results of second trained mechanism. A The surplus and Gini coefficient generated from games played under the interpolating baseline (yellow) and the RL agent M2 (light green), for virtual players (left panel) and human participants (right panel) in Exp. 3. Each small dot is a game; the larger dot is the mean over games. B Average reported preference for the interpolating (yellow) and RL agent M2 (green) agents on a number of dimensions. C Same as A; Results from human participants in Exp. 4., in which groups of players play three consecutive games.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 4 Not valid · 19 Not checked · 98 Own results and interpretations

41 more claims

Create a free account to see all 125 claims and to verify the 36 references still open.

See all claims — free account
  • Probably not a substantial claim. 48
  • It may state two separate claims as one. 42

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 38 references

Research works printed but not linked: 6

  1. Trust: Making and Breaking Cooperative Relations, Vol. 201–203 (B. Blackwell, New York, NY, USA, 1988).
  2. Ross, J. The economic theory of agency: the principal’s problem. Am. Econ. Rev. 63, 134–139 (1973).
  3. Dasgupta, P. & Heal, G. Economic Theory and Exhaustible Resources, Vol. 516 (Cambridge Univ. Pr, Cambridge, 2001).
  4. Kuznets, S. Economic gowth and income inequality. Am. Econ. Rev. 45, 1–28 (1955).
  5. Nowak, M. A. Evolutionary Dynamics: Exploring the Equations of Life Illustrated edition, Vol. 312 (Belknap Press of Harvard University Press, Cambridge, Mass, 2006).
  6. Axelrod, R. M. The Evolution of Cooperation Revised edition, Vol. 264 (Penguin Books, London; New York, 1990).

Cited by

See all 13

Metadata

Authors
Koster, R., Pîslar, M., Tacchetti, A., Balaguer, J., Liu, L., Elie, R., Hauser, O. P., Tuyls, K., Botvinick, M., & Summerfield, C.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • University of Exeter (GB)
  • University of Oxford (GB)
Venue
Nature Communications
Published
2025-03-22
Volume
16
Issue
1
Pages
2824-2824
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4408739195
Cited by
12
References
38
Citations by year
2026: 8 · 2025: 4
FWCI
12.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Open Research Exeter (University of Exeter) (submittedVersion, cc-by)
Topics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
common-pool resources, social dilemmas, reciprocity, trust game, deep reinforcement learning, social planner, resource allocation mechanisms, redistribution, sustainable behavior, repeated games
MeSH terms
Deep Learning, Reinforcement Machine Learning, Cooperative Behavior, Game Theory, Humans, Reinforcement, Psychology, Neural Networks, Computer, Trust, Resource Allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep reinforcement learning can promote sustainable human behaviour in a common-pool resource problem. https://app.cochat.ai/papers/10.1038/s41467-025-58043-7
The paper
Koster, R., Pîslar, M., Tacchetti, A., Balaguer, J., Liu, L., Elie, R., Hauser, O. P., Tuyls, K., Botvinick, M., & Summerfield, C. (2025). Deep reinforcement learning can promote sustainable human behaviour in a common-pool resource problem. Nature Communications. https://doi.org/10.1038/s41467-025-58043-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.