CoChat
Sign Up
Cited by 34

Scaffolding cooperation in human groups with deep reinforcement learning

McKee, Kevin R., Tacchetti, Andrea, Bakker, Michiel A., Balaguer, Jan, Campbell-Gillingham, Lucy, Everett, Richard, Botvinick, Matthew·2023·Nature Human Behaviour
DOI 10.1038/s41562-023-01686-7PMC PMC10593606

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Effective approaches to encouraging group cooperation are still an open challenge. Here we apply recent advances in deep learning to structure networks of human participants playing a group cooperation game. We leverage deep reinforcement learning and simulation methods to train a ‘social planner’ capable of making recommendations to create or break connections between group members. The strategy that it develops succeeds at encouraging pro-sociality in networks of human participants ( N = 208 participants in 13 groups) playing for real monetary stakes. Under the social planner, groups finished the game with an average cooperation rate of 77.7%, compared with 42.8% in static networks ( N = 176 in 11 groups). In contrast to prior strategies that separate defectors from cooperators (tested here with N = 384 in 24 groups), the social planner learns to take a conciliatory approach to defectors, encouraging them to act pro-socially by moving them to small highly cooperative neighbourhoods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41562-023-01686-7

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI systems could help to solve conflicts, for example, by searching over a larger space of potential solutions to disagreements, monitoring agreements, or acting as mediators.

    “AI systems could help to solve conflicts, for example, by searching over a larger space of potential solutions to disagreements, monitoring agreements, or acting as mediators (Bakker et al., 2022; Dafoe et al., 2020; McKee et al., 2023; Small et al., 2023).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as McKee et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. [1]Christakis NA Fowler JH Social contagion theory: examining dynamic social networks and human behavior Stat. Med. 2013 32 556 577 10.1002/sim.5408 22711416 PMC3830455 · DOI · Paper page
  2. [2]Keizer K Lindenberg S Steg L The importance of demonstratively restoring order PLoS ONE 2013 8 e65137 10.1371/journal.pone.0065137 23755182 PMC3673976 · DOI
  3. [3]Tsvetkova M Macy MW The social contagion of generosity PLoS ONE 2014 9 e87275 10.1371/journal.pone.0087275 24551053 PMC3923723 · DOI
  4. [5]Hatfield E Cacioppo JT Rapson RL Emotional contagion Curr. Dir. Psychol. Sci. 1993 2 96 100 10.1111/1467-8721.ep10770953 · DOI
  5. [6]Tsvetkova M Macy MW The social contagion of antisocial behavior Sociol. Sci 2015 2 36 49 10.15195/v2.a4 · DOI
  6. [7]Cacioppo JT Fowler JH Christakis NA Alone in the crowd: the structure and spread of loneliness in a large social network J. Pers. Soc. Psychol. 2009 97 977 10.1037/a0016076 19968414 PMC2792572 · DOI
  7. [8]Hill AL Rand DG Nowak MA Christakis NA Emotions as infectious diseases in a large social network: the SISa model Proc. Biol. Sci. 2010 277 3827 3835 20610424 PMC2992714
  8. [9]Lerman, K. & Ghosh, R. Information contagion: an empirical study of the spread of news on Digg and Twitter social networks. In Proc. 4th International AAAI Conference on Weblogs and Social Media 90–97 (AAAI Press, 2010).
  9. [10]Ugander J Backstrom L Marlow C Kleinberg J Structural diversity in social contagion Proc. Natl Acad. Sci. USA 2012 109 5962 5966 10.1073/pnas.1116502109 22474360 PMC3341012 · DOI
  10. [14]Rand DG Arbesman S Christakis NA Dynamic social networks promote cooperation in experiments with humans Proc. Natl Acad. Sci. USA 2011 108 19193 19198 10.1073/pnas.1108243108 22084103 PMC3228461 · DOI · Paper page
  11. [15]Shirado H Christakis NA Network engineering using autonomous agents increases cooperation in human groups iScience 2020 23 101438 10.1016/j.isci.2020.101438 32823053 PMC7452167 · DOI
  12. [16]Shirado H Fu F Fowler JH Christakis NA Quality versus quantity of social ties in experimental cooperative networks Nat. Commun. 2013 4 2814 10.1038/ncomms3814 24226079 PMC3868237 · DOI
  13. [18]Apicella CL Marlowe FW Fowler JH Christakis NA Social networks and cooperation in hunter-gatherers Nature 2012 481 497 501 10.1038/nature10736 22281599 PMC3340565 · DOI · Paper page
  14. [19]Smith KM Larroucau T Mabulla IA Apicella CL Hunter-gatherers maintain assortativity in cooperation despite high levels of residential change and mixing Curr. Biol. 2018 28 3152 3157 10.1016/j.cub.2018.07.064 30245106 · DOI
  15. [21]Zheng, S. et al. The AI economist: improving equality and productivity with AI-driven tax policies. Preprint at arXiv10.48550/arXiv.2004.13332 (2020).
  16. [22]Wagner, C. et al. Measuring algorithmically infused societies. Nature595, 197–204 (2021). 34194046
  17. [23]Sanz-Cruzado, J., Pepa, S. M. & Castells, P. Structural novelty and diversity in link prediction. In Companion Proceedings of the 2018 Web Conference 1347–1351 (International World Wide Web Conferences Steering Committee, 2018).
  18. [24]Sanz-Cruzado, J. & Castells, P. in Collaborative Recommendations: Algorithms, Practical Challenges and Applications (eds Berkovsky, S. et al.) Ch. 16 (World Scientific, 2019).
  19. [26]Battaglia, P. W. et al. Relational inductive biases, deep learning, and graph networks. Preprint at arXiv10.48550/arXiv.1806.01261 (2018).
  20. [27]Mnih, V. et al. Asynchronous methods for deep reinforcement learning. In Proc. 33rd International Conference on Machine Learning (eds Balcan, M. B. & Weinberger, K. Q.) 1928–1937 (PMLR, 2016).
  21. [28]Rand DG Nowak MA Fowler JH Christakis NA Static network structure can stabilize human cooperation Proc. Natl Acad. Sci. USA 2014 111 17093 17098 10.1073/pnas.1400406111 25404308 PMC4260616 · DOI
  22. [30]Scaffolding cooperation in human groups with deep reinforcement learning. OSF https://osf.io/8ahkg/ (2023). PMC10593606 37679439
  23. [31]Sanchez-Lengeling B Evaluating attribution for graph neural networks Adv. Neural Inf. Process. Syst. 2020 33 5898 5910
  24. [32]Zambaldi, V. et al. Deep reinforcement learning with relational inductive biases. In Proc. 7th International Conference on Learning Representations(ICLR 2019) 1–18 (OpenReview, 2019).
  25. [33]Borgatti SP Everett MG Models of core/periphery structures Soc. Netw. 2000 21 375 395 10.1016/S0378-8733(99)00019-2 · DOI
  26. [34]de Jeude JVL Caldarelli G Squartini T Detecting core–periphery structures by surprise Europhys. Lett. 2019 125 68001 10.1209/0295-5075/125/68001 · DOI
  27. [35]Heuillet A Couthouis F Díaz-Rodríguez N Explainability in deep reinforcement learning Knowl. Based Syst. 2021 214 106685 10.1016/j.knosys.2020.106685 · DOI
  28. [36]von Eschenbach WJ Transparency and the black box problem: why we do not trust AI Philos. Technol. 2021 34 1607 1622 10.1007/s13347-021-00477-0 · DOI · Paper page
  29. [37]Holm EA In defense of the black box Science 2019 364 26 27 10.1126/science.aax0162 30948538 · DOI
  30. [38]Centola DM Homophily, networks, and critical mass: solving the start-up problem in large group collective action Ration. Soc. 2013 25 3 40 10.1177/1043463112473734 · DOI
  31. [39]Sohn Y Choi J-K Ahn T-K Core–periphery segregation in evolving prisoner’s dilemma networks J. Complex Netw. 2020 8 cnz021 10.1093/comnet/cnz021 · DOI
  32. [40]Ketelaar T Tung Au W The effects of feelings of guilt on the behaviour of uncooperative individuals in repeated social bargaining games: an affect-as-information interpretation of the role of emotion in social interaction Cogn. Emot. 2003 17 429 453 10.1080/02699930143000662 29715746 · DOI · Paper page
  33. [41]Bardsley N Sausgruber R Conformity and reciprocity in public good provision J. Econ. Psychol. 2005 26 664 681 10.1016/j.joep.2005.02.001 · DOI
  34. [42]Cialdini, R. B. & Trost, M. R. Social Influence: Social Norms, Conformity and Compliance (McGraw-Hill, 1998).
  35. [43]Bapst, V. et al. Structured agents for physical construction. In Proc. 36th International Conference on Machine Learning (eds Chaudhuri, K. & Salakhutdinov, R.) 464–474 (PMLR, 2019).
  36. [44]Cranmer M Discovering symbolic models from deep learning with inductive biases Adv. Neural Inf. Process. Syst. 2020 33 17429 17442
  37. [45]Sanchez-Gonzalez, A. et al. Learning to simulate complex physics with graph networks. In Proc. 37th International Conference on Machine Learning (eds Daumé III, H., & Singh, A.) 8459–8468 (PMLR, 2020).
  38. [46]Comte, A. in Le Producteur, Journal Philosophique de l’Industrie, des Sciences et des Beaux Arts Ch. 5 (De Gruyter, 1825).
  39. [47]Kearns, M., Littman, M. L. & Singh, S. Graphical models for game theory. In Proc. 17th Conference on Uncertainty in Artificial Intelligence (eds Breese, J. S. & Koller, D.) 253–260 (Morgan Kaufmann, 2001).
  40. [48]Shirado H Iosifidis G Tassiulas L Christakis NA Resource sharing in technologically defined social networks Nat. Commun. 2019 10 1079 10.1038/s41467-019-08935-2 30842424 PMC6403336 · DOI
  41. [49]Mason WA Jones A Goldstone RL Propagation of innovations in networked groups J. Exp. Psychol. Gen. 2008 137 422 10.1037/a0012798 18729708 · DOI
  42. [50]Mason WA Watts DJ Collaborative learning in networks Proc. Natl Acad. Sci. USA 2012 109 764 769 10.1073/pnas.1110069108 22184216 PMC3271930 · DOI · Paper page
  43. [51]Ying, R. et al. Graph convolutional neural networks for web-scale recommender systems. In Proc. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 974–983 (Association for Computing Machinery, 2018).
  44. [52]Hu, W. et al. OGB-LSC: a large-scale challenge for machine learning on graphs. In Proc. 35th Conference on Neural Information Processing Systems: Datasets and Benchmarks Track 1–15 (OpenReview, 2021).
  45. [53]Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Burstein, J. et al.) 4171–4186 (Association for Computational Linguistics, 2019).
  46. [54]Brown T Language models are few-shot learners Adv. Neural Inf. Process. Syst. 2020 33 1877 1901
  47. [55]Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M., and Choi, Y. Reframing human–AI collaboration for generating free-text explanations. In Proc. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Carpuat, M. et al.) 632–658 (Association for Computational Linguistics, 2022).
  48. [56]Heer J Agency plus automation: designing artificial intelligence into interactive systems Proc. Natl Acad. Sci. USA 2019 116 1844 1850 10.1073/pnas.1807184115 30718389 PMC6369770 · DOI · Paper page
  49. [58]McKee, K. R., Bai, X. & Fiske, S. T. Warmth and competence in human-agent cooperation. In Proc. 21st International Conference on Autonomous Agents and MultiAgent Systems 898–907 (International Foundation for Autonomous Agents and Multiagent Systems, 2022).
  50. [59]Dobbe R Gilbert TK Mintz Y Hard choices in artificial intelligence Artific. Intell. 2021 300 103555 10.1016/j.artint.2021.103555 · DOI · Paper page
  51. [60]Garvey, C. A framework for evaluating barriers to the democratization of artificial intelligence. In Proc. 32nd AAAI Conference on Artificial Intelligence 8079–8080 (AAAI Press, 2018).
  52. [61]Birhane, A. et al. Power to the people? Opportunities and challenges for participatory AI. In Proc. Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '22) 1–8 (Association for Computing Machinery, 2022).
  53. [63]McKee KR Bai X Fiske ST Humans perceive warmth and competence in artificial intelligence iScience 2023 26 107256 10.1016/j.isci.2023.107256 37520710 PMC10371826 · DOI
  54. [65]Weidinger L Using the Veil of Ignorance to align AI systems with principles of justice Proc. Natl Acad. Sci. USA 2023 120 e2213709120 10.1073/pnas.2213709120 37094137 PMC10160973 · DOI
  55. [66]Carroll M On the utility of learning about humans for human–AI coordination Adv. Neural Inf. Process Syst. 2019 32 1 12
  56. [67]Strouse D McKee KR Botvinick M Hughes E Everett R Collaborating with humans without human data Adv. Neural Inf. Process. Syst. 2021 34 14502 14515
  57. [68]Paiva, A., Santos, F. & Santos, F. Engineering pro-sociality with autonomous agents. In Proc. 32nd AAAI Conference on Artificial Intelligence 7994–7999 (AAAI Press, 2018).
  58. [69]Espeholt, L. et al. IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 1407–1416 (PMLR, 2018).
  59. [70]Peer, E., Rothschild, D., Gordon, A., Evernden, Z. & Damer, E. Data quality of platforms and panels for online behavioral research. Behav. Res. Methods10.3758/s13428-021-01694-3 (2021). PMC8480459 34590289

Cited by

Sorted by citations

Authors
McKee, K. R., Tacchetti, A., Bakker, M. A., Balaguer, J., Campbell-Gillingham, L., Everett, R., & Botvinick, M.
ORCID
McKee, Kevin R.
Venue
Nature Human Behaviour
Publisher
Springer Science and Business Media LLC
Published
2023-09-07
Volume
7
Issue
10
Pages
1787-1796
ISSN
2397-3374
DOI
10.1038/s41562-023-01686-7
PMC
PMC10593606
Cited by
34
References
7
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review