Multi-agent Reinforcement Learning in Sequential Social Dilemmas
Abstract
Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Matrix game social dilemmas lack the complexity of real-world temporal and spatial dynamics.
“While MGSDs are valuable for abstracting key aspects of real-world multi-agent interactions-such as cooperation, risk, and trust-they lack the complexity of real-world temporal and spatial dynamics.”
From Environment Complexity and Nash Equilibria in a Sequential Social Dilemma (2024) · cites this paper as [12]
Such endogenous coordination failures blur the line between attack and accident.
“Such endogenous coordination failures blur the line between attack and accident (Leibo et al., 2017;Rahwan et al., 2019), reinforcing that defenses must account for both malicious swarms and emergent collective behaviors.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Leibo et al., 2017
Cites
Still being checked.
Cited by
Sorted by citations
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning
McKee, K. R., Gemp, I., McWilliams, B., Duèñez-Guzmán, E. A., Hughes, E., & Leibo, J. Z. (2020) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/dnuf7773 · Cited by 33 · see citing works
Self-citationShen, Y., Zhang, Y., Kang, H., Sun, X., Chen, Q., & Zhang, X. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3649080 · Cited by 7 · see citing works
Ren, T., & Zeng, X.-J. (2024) · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/22 · Cited by 5 · see citing works
“…spatial and temporal dynamics relevant to realistic scenarios [Ren and Zeng, 2024; Vinitsky et al. , 2023; Tennant et al. , 2023], moving beyond traditional matrix games through the fusion of complex incentive structures [Leibo et al. , 2017; Jaques et al. , 2019; Abeywickrama et al. , 2023].”
BackgroundDynamics of Q -Learning in Networked Stochastic Games
Yuan, Z., Jiang, G., Hu, S., Perc, M., Chu, C., & Liu, J. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3641365 · Cited by 4 · see citing works
MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning
Samvelyan, M., Khan, A., Dennis, M., Jiang, M., Parker-Holder, J., Foerster, J., Raileanu, R., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2303.03376 · Cited by 8 · see citing works
“It is inspired by prior singleton variation used in (Lanctot et al., 2017; Leibo et al., 2017) and developed using the Griddly sandbox framework (Bamford et al., 2020; 2022).”
MethodNormative Disagreement as a Challenge for Cooperative AI
Stastny, J., Riché, M., Lyzhov, A., Treutlein, J., Dafoe, A., & Clifton, J. (2021) · arXiv · doi:10.48550/arxiv.2111.13872 · Cited by 3 · see citing works
Coordination transparency: governing distributed agency in AI systems
Bohr, J. (2026) · AI & Society · doi:10.1007/s00146-026-02853-w · Cited by 5 · see citing works
Building Altruistic and Moral AI Agent With Brain-Inspired Emotional Empathy Mechanisms
Zhao, F., Feng, H., Tong, H., Han, Z., Lin, E., Lu, E., Sun, Y., & Zeng, Y. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3627936 · Cited by 3 · see citing works
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
Environment Complexity and Nash Equilibria in a Sequential Social Dilemma
Yasir, M., Howes, A., Mavroudis, V., & Hicks, C. (2024) · arXiv · doi:10.48550/arxiv.2408.02148 · Cited by 1 · see citing works
“Here we further characterise the Group B environments with an empirical game-theoretic analysis following the methodology of Leibo et al. [12].”
BackgroundMethodInfluentialHammurabi: Establish Cooperative Order From Pre-Trained Policies in Multi-UAV Networks
Chen, D., He, H., Qi, Q., Wang, J., Han, R., He, B., Zhuang, Z., Fu, Q., Liao, J., & Han, Z. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3654605 · Cited by 1 · see citing works
A successful strategy for iterated Prisoner’s dilemma with any number of channels
Cao, Z., Shi, J., Wang, Z., Hu, S., & Chu, C. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104572 · Cited by 1 · see citing works
Agentic AI Safety: A Structured Review of Open Problems and Their Regulatory Anchoring
Valenta, T., Rozinek, O., & Horálek, J. (2026) · Applied Informatics · doi:10.3390/ai7080298 · Cited by 1 · see citing works
Pasichnyk, I. (2026) · arXiv · doi:10.48550/arxiv.2603.07360
“Leibo et al. [2017] study cooperation and defection dynamics in sequential social dilemmas.”
QMIP: Enhancing Multiagent Reinforcement Learning Coordination Through Macro Intention Prediction
Cui, Y., Luo, B., Tang, R., Yang, C., & Gui, W. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3639190
Characterizing lane-changing behavior in mixed traffic
Chung, S., Talebpour, A., & Hamdar, S. H. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.106039
Token Adjustment for Peer Incentivization in Sequential Social Dilemma Environments
Namikoshi, K., Mizuno, S., & Sakurai, Y. (2025) · IEEE International Conference on Agents (ICA) · doi:10.1109/ica67499.2025.00022
Adaptive Incentive Mechanism for Multi-Agent Systems in Adversarial Environments
Chen, M., Zhang, X., Wang, L., & Zhang, J. (2025) · International Conference on Computer, Internet of Things and Smart City (CIoTSC) · doi:10.1109/ciotsc67482.2025.11413055
Conclusion and the Road Ahead: Open Challenges and Responsibility
Nagasubramanian, D. (2026) · Apress eBooks · doi:10.1007/979-8-8688-2361-9_14
CoGrid & the Multi-User Gymnasium: A framework for multi-agent experimentation
McDonald, C., & González, C. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03078-x
Social and emotional learning in artificial agents
Torres, N. (2026) · Scientific Reports · doi:10.1038/s41598-026-48309-5
Alqithami, S. (2026) · Mathematical and Computational Applications · doi:10.3390/mca31030069
Research on Multi-agent Game Methods Based on the Cournot Duopoly Strategy Model
Song, C., Yang, B., Liu, Y., Zhao, H., & Zhuang, Y. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-7648-7_41
Self-evolving Embodied Intelligence
Wang, X., Feng, T., Liu, H., & Zhu, W. (2026) · Springer eBooks · doi:10.1007/978-981-95-7750-7_4
Henz, P. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108939010.021
Pogrebna, G., & Hills, T. T. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108939010.015
UDRF: An Ubuntu Inspired Framework for Ethical Decentralized Decision Making and Resource Allocation
Sagor, M. T. (2026) · IEEE International Conference on Quantum Photonics, Artificial Intelligence & Networking (QPAIN) · doi:10.1109/qpain69676.2026.11545551
Incentive Mechanism Design in Multi-Agent Reinforcement Learning
Hua, L., Chen, S., & Chen, J. B. (2026) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc69976.2026.11560078
Adhikari, T. N., Sayers, W., & Zhang, S. (2026) · Biomimetics · doi:10.3390/biomimetics11070476
Bagal, S. (2026) · Global Journal of Computer Science and Technology · doi:10.34257/gjcstd258739
Computing exact optimal covers for cover function games with approximate ordinal input
Fatima, S. S. (2026) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-026-09756-6
Ethical Markov Decision Processes
Stojanovski, M., Bourdache, N., Bonnet, G., & Mouaddib, A. (2026) · The European Journal on Artificial Intelligence · doi:10.1177/30504554261460318
Salem, M., Pontes, D., Böhm, L., dos Reis, H., & Ferrugem, A. P. (2026) · Simpósio Brasileiro de Computação e Comunicação Quânticas (SBCCQ) · doi:10.5753/sbccq.2026.23434
Askar, S., Bakhtgerey, S., Chinchalinova, A., Kanysh, T., Dmitriy, M., & Aliya, O. (2026) · International Conference on Cybersecurity, Digital Forensics, and AI Applications (ICCSDFAI) · doi:10.1109/iccsdfai70505.2026.11647873
COALITION-VAST: Auditable Multi-Agent Alignment Under Byzantine Governance
Partow, S., & Nan, S. (2026) · International Conference on Intelligent Multimedia, Networking, and Security (IMNS) · doi:10.1109/imns67862.2026.11655305
Peng, X., Liu, Y., Liu, Y., Cao, Y. Q., Zhong, W., & Qian, F. (2026) · Chemical Engineering Science · doi:10.1016/j.ces.2026.125033
Sinchev, A. B., Синчев, Б. К., Chinchalinova, A., & Aliya, O. (2026) · НАУКА ИНТЕЛЛЕКТУАЛЬНЫХ СИСТЕМ · doi:10.66941/3106-9037/2026/iss1pp53-60
Adaptive Emission Trading System Policy Design via Bi-level Multi-agent Reinforcement Learning
Long, X., Zeng, X., & Meng, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38404-1_11
Learning Incentive Mechanisms in Multi-Sender Signaling Games
Hua, L., Hu, Z., & Wu, T. (2026) · International Conference on Machine Learning and Intelligent Computing · doi:10.1145/3829441.3829616
Shil, M., Pillai, G. N., & Gupta, M. K. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729655
- Authors
- Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Venue
- International Joint Conference on Autonomous Agents and Multiagent Systems
- Publisher
- IEEE Computer Society
- Published
- 2017-05-08
- Pages
- 464-473
- ISSN
- 1548-8403 (ISSN-L)
- 1558-2914
- OpenAlex
- W7124273160
- Cited by
- 35
- References
- 0
- Citations by year
- 2026: 29 · 2025: 4 · 2022: 1 · 2019: 1
- FWCI
- 0.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Adaptive Agents and Multi-Agents Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- multi-agent reinforcement learning, prisoner's dilemma, cooperation, deep Q-network, stochastic games, resource competition, conflict emergence, policy dynamics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
