Emergent Tool Use From Multi-Agent Autocurricula
Open accessAbstract
Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.
“For example, Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Baker et al., 2019
Agents develop exploitative strategies.
“Agents develop exploitative strategies (Baker et al., 2019), learn deception via theory-ofmind (Schulz et al., 2023), and discover betrayal without explicit training (Aitchison et al., 2021) -eluding threat models predicated on an identifiable adversary.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Baker et al., 2019
PPO is a competent agent used by OpenAI.
“Consequently, we used the PPO agent in Experiments III and IV, as PPO is a competent agent used by OpenAI [32].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4× · see citing works
“[75] Sepp Hochreiter and Jürgen Schmidhuber.”
MethodInfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 2× · see citing works
“[30] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.”
BackgroundMethodProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 4× · see citing works
“PPO [58], a policy gradient variant, penalizes large changes to the policy to prevent training instabilities.”
BackgroundInfluentialMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2× · see citing works
“[57] Emanuel Todorov, Tom Erez, and Yuval Tassa.”
BackgroundMethodArms races between and within species
Dawkins, R., & Krebs, J. R. (1979) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1979.0081 · Cited by 2,163 · cited 2× · see citing works
“The vast amount of complexity and diversity on Earth evolved due to co-evolution and competition between organisms, directed by natural selection [5].”
BackgroundHigh-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 3× · see citing works
“We use Generalized Advantage Estimation [3] with horizon length H to estimate the advantage function.”
BackgroundMethodTemporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 3× · see citing works
“In the context of multi-agent RL, [7], [30], [9], [13] and [10] used self-play with deep RL techniques to achieve super-human performance in Backgammon, Go, Dota, Capture-the-Flag and Starcraft, respectively.”
BackgroundMethodConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · see citing works
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06676 · Cited by 1,217 · cited 2× · see citing works
“[32] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson.”
Background(1978) · Medical Entomology and Zoology · Cited by 1,199 · see citing works
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 2× · see citing works
“[8] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.”
BackgroundSims, K. (1994) · Proceedings of the 21st annual conference on Computer graphics and interactive techniques - SIGGRAPH '94 · doi:10.1145/192161.192167 · Cited by 1,005 · cited 2× · see citing works
“Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acq…”
BackgroundLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847 · cited 2× · see citing works
“References [1] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.”
BackgroundEvolving 3D Morphology and Behavior by Competition
Sims, K. (1994) · Artificial Life · doi:10.1162/artl.1994.1.353 · Cited by 800 · see citing works
Animal tool behavior: the use and manufacture of tools by animals
(2011) · Choice Reviews Online · doi:10.5860/choice.49-1468 · Cited by 783 · see citing works
(2016) · neural information processing systems · Cited by 735 · see citing works
TrueSkill™: A Bayesian Skill Rating System
Herbrich, R., Minka, T., & Graepel, T. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0076 · Cited by 693 · cited 2× · see citing works
“Metrics like ELO (Elo, 1978) or Trueskill (Herbrich et al., 2007) can more reliably measure whether performance is improving relative to previous policy versions or other policies in a population; however, these metrics still do not give insight into whether improved performance stems from new adaptations or improving…”
BackgroundA possibility for implementing curiosity and boredom in model-building neural controllers
Schmidhuber, J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/3115.003.0030 · Cited by 669 · cited 1× · see citing works
“Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
BackgroundHuman-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662 · cited 2× · see citing works
“Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acq…”
BackgroundMethodEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · cited 2× · see citing works
“[4] Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al.”
BackgroundManufacture and use of hook-tools by New Caledonian crows
Hunt, G. R. (1996) · Nature · doi:10.1038/379249a0 · Cited by 645 · see citing works
Intrinsically Motivated Reinforcement Learning
Singh, S. P., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 614 · cited 2× · see citing works
“[35] Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh.”
BackgroundMethodIncentivizing Exploration In Reinforcement Learning With Deep Predictive ModelsPreprint
Stadie, B. C., Levine, S., & Abbeel, P. (2015) · arXiv.org · doi:10.48550/arxiv.1507.00814 · Cited by 545 · cited 2× · see citing works
“Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
Background#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · cited 2× · see citing works
“One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
BackgroundVariational Information Maximisation for Intrinsically Motivated Reinforcement LearningPreprint
Mohamed, S., & Rezende, D. J. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.08731 · Cited by 460 · cited 2× · see citing works
“Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
BackgroundOne-Shot Imitation LearningPreprint
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, O. J., Schneider, J., Sutskever, I., Abbeel, P., & Zaremba, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07326 · Cited by 449 · see citing works
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428 · cited 3× · see citing works
“One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
BackgroundMethodIntrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Singh, S. P., Lewis, R. L., Barto, A. G., & Sorg, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051031 · Cited by 418 · cited 2× · see citing works
“Intrinsic motivation methods have been widely studied in the literature [35, 36].”
BackgroundMethodCompetitive Coevolution through Evolutionary Complexification
Stanley, K. O., & Miikkulainen, R. P. (2004) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1338 · Cited by 415 · cited 2× · see citing works
“Early work explored self-play using genetic algorithms [16, 17, 18, 19].”
BackgroundCounterfactual Multi-Agent Policy GradientsPreprint
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08926 · Cited by 408 · cited 2× · see citing works
“At optimization time, the value function has access to the full unmasked environment state, similar to [59, 33, 60].”
BackgroundAvida: A Software Platform for Research in Computational Evolutionary Biology
Ofria, C., & Wilke, C. O. (2004) · Artificial Life · doi:10.1162/106454604773563612 · Cited by 378 · see citing works
Count-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377 · cited 2× · see citing works
“One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
BackgroundRelational Reinforcement Learning
Džeroski, S., De Raedt, L., & Driessens, K. (2001) · Machine Learning · doi:10.1023/a:1007694015589 · Cited by 371 · see citing works
Asymmetric Actor Critic for Image-Based Robot Learning
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., & Abbeel, P. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.008 · Cited by 369 · cited 2× · see citing works
“[59] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel.”
BackgroundSocial Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement LearningPreprint
Jaques, N., Lazaridou, A., Hughes, E., Gülçehre, Ç., Ortega, P. A., Strouse, D., Leibo, J. Z., & de Freitas, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08647 · Cited by 289 · see citing works
Reverse Curriculum Generation for Reinforcement LearningPreprint
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05300 · Cited by 277 · cited 2× · see citing works
“[27] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.”
BackgroundLearning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.00177 · Cited by 248 · cited 2× · see citing works
“[2] Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al.”
BackgroundSelf-citationParedis, J. (1995) · Artificial Life · doi:10.1162/artl.1995.2.355 · Cited by 229 · see citing works
An object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226 · cited 2× · see citing works
“[71] Carlos Diuk, Andre Cohen, and Michael L Littman.”
BackgroundMethodEmergence of Grounded Compositional Language in Multi-Agent PopulationsPreprint
Mordatch, I., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04908 · Cited by 201 · cited 1× · see citing works
“In addition, communication has also been shown to emerge from multi-agent RL [31, 32, 33, 34].”
BackgroundSelf-citationIntrinsically Motivated Goal Exploration Processes with Automatic Curriculum LearningPreprint
Forestier, S., Portelas, R., Mollard, Y., & Oudeyer, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02190 · Cited by 200 · see citing works
Methods for Competitive Co-Evolution: Finding Opponents Worth Beating
Rosin, C. D., & Belew, R. K. (1995) · Cited by 169 · see citing works
Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01753 · Cited by 166 · cited 2× · see citing works
“[29] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley.”
BackgroundRelational Deep Reinforcement LearningPreprint
Zambaldi, V., Raposo, D. C., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., Lockhart, E., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01830 · Cited by 153 · cited 2× · see citing works
“[73] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al.”
BackgroundMethodMulti-agent Reinforcement Learning in Sequential Social DilemmasPreprint
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017) · arXiv (Cornell University) · doi:10.5555/3091125.3091194 · Cited by 151 · see citing works
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139 · cited 1× · see citing works
“Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
BackgroundRapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912341117 · Cited by 120 · see citing works
A multi-agent reinforcement learning model of common-pool resource appropriation
Perolat, J., Leibo, J. Z., Zambaldi, V., Beattie, C., Tuyls, K., & Graepel, T. (2017) · arXiv · doi:10.48550/arxiv.1707.06600 · Cited by 109 · cited 2× · see citing works
“Many stages of emergent strategy can be mapped to behavioral shifts in the way agents interact with the tools in their environment, similar to [61, 62].”
BackgroundEmergent Coordination Through CompetitionPreprint
Liu, S., Lever, G., Merel, J., Tunyasuvunakool, S., Heess, N., & Graepel, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07151 · Cited by 103 · cited 3× · see citing works
“Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually enable agents to acq…”
BackgroundCoevolution of A Backgammon Player
Pollack, J. B., Blair, A., & Land, M. (1996) · Cited by 91 · see citing works
A Game-Theoretic Memory Mechanism for Coevolution
Ficici, S. G., & Pollack, J. B. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45105-6_35 · Cited by 81 · cited 2× · see citing works
“[69] Sevan G Ficici and Jordan B Pollack.”
BackgroundCuriosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
Large-Scale Study of Curiosity-Driven Learning
Burda, Y. (2025) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.57702/mspz8fly · Cited by 75 · see citing works
Intrinsic motivation and automatic curricula via asymmetric self-play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A. D., & Fergus, R. (2018) · International Conference on Learning Representations · Cited by 72 · see citing works
Improvisation through Physical Understanding: Using Novel Objects As Tools with Visual Foresight
Xie, A., Ebert, F., Levine, S., & Finn, C. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.001 · Cited by 66 · cited 2× · see citing works
“For example, in [54] a real-world robot uses familiar and novel tools to move objects using model-based RL and imitation learning.”
BackgroundLehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., Bentley, P. J., Bernard, S., Beslon, G., Bryson, D., et al. (2018) · arXiv (Cornell University) · Cited by 63 · see citing works
Leibo, J. Z., Hughes, E., Lanctot, M., & Graepel, T. (2019) · arXiv · doi:10.48550/arxiv.1903.00742 · Cited by 60 · cited 2× · see citing works
“[6] Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel.”
BackgroundSoros, L. B., & Stanley, K. O. (2014) · doi:10.1162/978-0-262-32621-6-ch128 · Cited by 51 · see citing works
Structured agents for physical constructionPreprint
Bapst, V., Sánchez‐González, Á., Doersch, C., Stachenfeld, K., Kohli, P., Battaglia, P., & Hamrick, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03177 · Cited by 49 · cited 2× · see citing works
“In [55], an agent solves construction tasks in a 2-D environment using both model-based and model-free methods.”
BackgroundMethodPerpetuating Evolutionary Emergence
Channon, A., & Damper, R. I. (1998) · The MIT Press eBooks · doi:10.7551/mitpress/3119.003.0080 · Cited by 28 · cited 1× · see citing works
“Open-ended evolution was further explored in the environments Polyworld [21] and Geb [22], where agents compete and mate in a 2D world, and in Tierra [23] and Avida [24], where computer programs compete for computational resources.”
BackgroundMalthusian Reinforcement Learning
Leibo, J. Z., Pérolat, J., Hughes, E., Wheelwright, S., Marblestone, A. H., Duéñez‐Guzmán, E. A., Sunehag, P., Dunning, I., & Graepel, T. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/fdir9565 · Cited by 24 · cited 2× · see citing works
“[51] Joel Z Leibo, Julien Perolat, Edward Hughes, Steven Wheelwright, Adam H Marblestone, Edgar Duéñez-Guzmán, Peter Sunehag, Iain Dunning, and Thore Graepel.”
BackgroundResultRequirements for Open-Ended Evolution in Natural and Artificial SystemsPreprint
Taylor, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.07403 · Cited by 16 · cited 1× · see citing works
“More recent work attempted to formulate necessary preconditions for open-ended evolution [25, 26].”
Background
Flagged first, then by citations
6 of the 70 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- Kelsey R Allen, Kevin A Smith, and Joshua B Tenenbaum. The tools challenge: Rapid trial-anderror learning in physical problem solving. arXiv preprint arXiv:1907.09620 , 2019. · arXiv
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Rene Baillargeon and Susan Carey. Core cognition and beyond: The acquisition of physical and numerical knowledge. In In S. Pauen (Ed.), Early Childhood Development and Later Outcome , pp. 33-65. University Press, 2012.
- Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations , 2018.
- Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2019b.
- Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsically-motivated, self-aware agents. In Advances in Neural Information Processing Systems , pp. 8388-8399, 2018.
- Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems , pp. 1109-1117, 2016.
- Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017. · arXiv
- Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , pp. 6379-6390, 2017.
- Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning , pp. 2778-2787, 2017.
- Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems , pp. 2244-2252, 2016.
- Larry Yaeger. Computational genetics, physiology, metabolism, neural systems, learning, vision, and behavior or poly world: Life in a new context. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXITY-PROCEEDINGS VOLUME, volume 17, pp. 263-263. ADDISON-WESLEY PUBLISHING CO, 1994.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationA Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S., & Kim, Y. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 1,926 · see citing works
“Hide and Seek is a simple yet effective simulation environment for multi-agent work that uses visual representations of objects and scenes from an egocentric perspective [218]–[221].”
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957 · see citing works
Unity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626 · see citing works
“The features of the toolkit include a set of example environments, state of the art RL algorithms Soft Actor-Critic (SAC) (Haarnoja et al., 2018) and Proximal Policy Optimization (PPO) (Schulman et al., 2017), the IL algorithms Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon, 2016) and Behavioral Clonin…”
BackgroundInfluentialDirect Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387 · see citing works
Attention, please! A survey of neural attention models in deep learning
Correia, A. D. S., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 330 · see citing works
Inductive biases for deep learning of higher-level cognition
Goyal, A., & Bengio, Y. (2022) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2021.0068 · Cited by 300 · see citing works
“…the non-stationary and competition systems put pressure on the learner to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with new environments (Bansal et al., 2017; Liu et al., 2019; Baker et al., 2019; Leibo et al., 2019).”
BackgroundCurriculum Learning for Reinforcement Learning Domains: A Framework and SurveyPreprint
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., & Stone, P. (2020) · Journal of machine learning research · doi:10.48550/arxiv.2003.04960 · Cited by 843 · see citing works
“Similar results have been shown in the context of RL agents by Baker et al. (2020). They demonstrated that increasingly complex behaviors can emerge in a physically grounded task.”
ResultCompositionally restricted attention-based network for materials property predictions
Wang, A., Kauwe, S. K., Murdock, R., & Sparks, T. D. (2021) · npj Computational Materials · doi:10.1038/s41524-021-00545-1 · Cited by 266 · see citing works
Deep reinforcement learning for inventory control: A roadmap
Boute, R. N., Gijsbrechts, J., van Jaarsveld, W. L., & Vanvuchelen, N. (2021) · European Journal of Operational Research · doi:10.1016/j.ejor.2021.07.016 · Cited by 254 · see citing works
Collision Avoidance in Pedestrian-Rich Environments With Deep Reinforcement Learning
Everett, M., Chen, Y. F., & How, J. P. (2021) · IEEE Access · doi:10.1109/access.2021.3050338 · Cited by 234 · see citing works
“of collision avoidance, recent work [33] introduced attention mechanisms, another tool popularized in NLP, as another method for embedding the variable number of other agents’ states.”
MethodGuillén, A., & Cano, M. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3169907 · Cited by 201 · see citing works
Explainable AI and Reinforcement Learning—A Systematic Review of Current Approaches and Trends
Wells, L., & Bednarz, T. P. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.550030 · Cited by 190 · see citing works
“A recent example of this was a game of hide-and-seek where agents learned strategies to exploit the physics system of the game to overcome what was intended by the developers to be walls that could not be passed (Baker et al., 2019).”
BackgroundSim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., & Batra, D. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3013848 · Cited by 201 · see citing works
“Notable examples include evolving tall creatures for locomotion that fall and somersault instead of learning active locomotion strategies [16] and OpenAI’s hide-and-seek agents abusing their physics engine to ‘surf’ on top of obstacles [17].”
BackgroundDecentralized Multi-Agent Pursuit Using Deep Reinforcement Learning
de Souza, C., Newbury, R., Cosgun, A., Castillo, P., Vidolov, B., & Kuli, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068952 · Cited by 184 · see citing works
A survey on multi-agent reinforcement learning and its application
Ning, Z., & Xie, L.-H. (2024) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2024.02.003 · Cited by 257 · see citing works
Learning agile soccer skills for a bipedal robot with deep reinforcement learning
Haarnoja, T., Moran, B., Lever, G., Huang, S. H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N., Hafner, R., et al. (2024) · Science Robotics · doi:10.1126/scirobotics.adi8022 · Cited by 191 · see citing works
Robust flow control and optimal sensor placement using deep reinforcement learning
Paris, R., Beneddine, S., & Dandois, J. (2021) · Journal of Fluid Mechanics · doi:10.1017/jfm.2020.1170 · Cited by 164 · see citing works
“Coupled with reinforcement methods, that use interactions with the controlled environment to improve performance, these techniques achieve autonomous learning of complex tasks (Baker et al. 2019; Kaiser et al. 2019), and often perform better than human experts (Mnih et al.”
MethodAndriotis, C. P., & Papakonstantinou, K. G. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.107551 · Cited by 141 · see citing works
Game-Theoretic Multiagent Reinforcement LearningPreprint
Yang, Y., Ma, C., Ding, Z., McAleer, S., Jin, C., Wang, J., & Sandholm, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00583 · Cited by 192 · see citing works
Botifoll, M., Pinto-Huguet, I., & Arbiol, J. (2022) · Nanoscale Horizons · doi:10.1039/d2nh00377e · Cited by 81 · see citing works
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128 · see citing works
“For example, Baker et al. (2020) noticed that the agents learned to exploit the physics engine to gain an unexpected advantage.”
BackgroundAre the Futures Computable? Knightian Uncertainty and Artificial Intelligence
Townsend, D. M., Hunt, R., Rady, J., Manocha, P., & Jin, J. (2024) · Academy of Management Review · doi:10.5465/amr.2022.0237 · Cited by 124 · see citing works
Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912341117 · Cited by 120 · see citing works
“Others learn from thousands of years of simulated experience (Baker et al., 2019), which is significantly longer than required for people.”
BackgroundROMA: Multi-Agent Reinforcement Learning with Emergent RolesPreprint
Wang, T., Dong, H., Lesser, V., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08039 · Cited by 120 · see citing works
“, 2020), the emergence of fairness (Jiang & Lu, 2019), and the emergence of tool usage (Baker et al., 2020) provide a deep learning perspective in understanding both natural and artificial multi-agent systems.”
BackgroundThe Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent GamesPreprint
Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A. M., & Wu, Y. (2021) · arXiv (Cornell University) · Cited by 121 · see citing works
Self-citationA review of research on reinforcement learning algorithms for multi-agents
Hu, K., Li, M., Song, Z., Xu, K., Xia, Q., Sun, N., Zhou, P., & Xia, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128068 · Cited by 113 · see citing works
RODE: Learning Roles to Decompose Multi-Agent TasksPreprint
Wang, T., Gupta, T., Mahajan, A., Peng, B., Whiteson, S., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01523 · Cited by 103 · see citing works
“Many interesting real-world phenomena, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social influence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be…”
BackgroundOpen Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93 · see citing works
Augmenting Automated Game Testing with Deep Reinforcement Learning
Bergdahl, J., Gordillo, C., Tollmar, K., & Gisslen, L. (2020) · 2020 IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231552 · Cited by 91 · see citing works
“Moreover, there are plenty of examples where RL algorithms have found unintended mechanics in complex scenarios [2].”
BackgroundUnveiling the Metaverse: Exploring Emerging Trends, Multifaceted Perspectives, and Future Challenges
Uddin, M., Manickam, S., Ullah, H., Obaidat, M. A., & Dandoush, A. (2023) · IEEE Access · doi:10.1109/access.2023.3281303 · Cited by 94 · see citing works
“Research conducted by Baker et al. [40] found that agents in a hide-and-seek type of multi-agent competition can generate an educational program for automation that develops new techniques at various stages.”
Explainable reinforcement learning in production control of job shop manufacturing system
Kuhnle, A., May, M. C., Schäfer, L., & Lanza, G. (2021) · International Journal of Production Research · doi:10.1080/00207543.2021.1972179 · Cited by 87 · see citing works
“Baker et al. (2019) model a simple game of hide-and-seek as a multi-agent RL-system.”
MethodAttacking Deep Reinforcement Learning With Decoupled Adversarial Policy
Mo, K., Tang, W., Li, J., & Yuan, X. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3143566 · Cited by 80 · see citing works
Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning
Lyu, X., Xiao, Y., Daley, B., & Amato, C. (2021) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/jcle4281 · Cited by 74 · see citing works
From motor control to team play in simulated humanoid football
Liu, S., Lever, G., Wang, Z., Merel, J., Eslami, S. M. A., Hennes, D., Czarnecki, W. M., Tassa, Y., Omidshafiei, S., Abdolmaleki, A., et al. (2022) · Science Robotics · doi:10.1126/scirobotics.abo0235 · Cited by 88 · see citing works
“behaviours and strategies have been shown to emerge as a result of competition between agents, rather than due to increasing difficulty of manually designed tasks (11, 21, 100, 108).”
BackgroundUddin, M., Obaidat, M. A., Manickam, S., Laghari, S. U. A., Dandoush, A., Ullah, H., & Ullah, S. S. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1556 · Cited by 87 · see citing works
“(Baker et al., 2019; Chen, Song, et al., 2020; Ilinykh et al., 2019) People and Society Metaverse for solving complex social problems; social change in the Metaverse; Metaverse for educational library orientation; museum experiences; museum social and cultural experiences (Dascalu et al., 2014; De…”
Transactive Energy for Flexible Prosumers Using Algorithmic Game Theory
Τσαούσογλου, Γ., Pinson, P., & Paterakis, N. G. (2021) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2021.3055764 · Cited by 73 · see citing works
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement LearningPreprint
Long, Q., Zhou, Z., Gupta, A., Fang, F., Wu, Y., & Wang, X.-L. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2003.10423 · Cited by 106 · see citing works
“Baker et al. (2019) show that multi-agent self-play can also lead to autocurricula in open-ended environments.”
Self-citationCentralized and Distributed Deep Reinforcement Learning Methods for Downlink Sum-Rate Optimization
Khan, A. A., & Adve, R. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3022705 · Cited by 61 · see citing works
A Survey of Domain-Specific Architectures for Reinforcement Learning
Rothmann, M., & Porrmann, M. (2022) · IEEE Access · doi:10.1109/access.2022.3146518 · Cited by 63 · see citing works
“Other interesting developments are agents with emergent tool use and cooperative behavior [3], [82].”
BackgroundAdaptive Multifactorial Evolutionary Optimization for Multitask Reinforcement Learning
Martinez, A. D., Ser, J. D., Osaba, E., & Herrera, F. (2021) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2021.3083362 · Cited by 62 · see citing works
Managing Uncertainty in AI-Enabled Decision Making and Achieving Sustainability
Wu, J., & Shang, S. S. C. (2020) · Sustainability · doi:10.3390/su12218758 · Cited by 58 · see citing works
Current cases of AI misalignment and their implications for future risks
Dung, L. (2023) · Synthese · doi:10.1007/s11229-023-04367-0 · Cited by 66 · see citing works
One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic ControlPreprint
Huang, W., Mordatch, I., & Pathak, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04976 · Cited by 64 · see citing works
Self-citationEffective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles
Zeng, X., Peng, H. M., & Li, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26390 · Cited by 54 · see citing works
Watch-And-Help: A Challenge for Social Perception and Human-AI CollaborationPreprint
Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y.-H., Tenenbaum, J. B., Fidler, S., & Torralba, A. B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09890 · Cited by 67 · see citing works
“There has been a large body of platforms for various multi-agent tasks (Jaderberg et al., 2019; Samvelyan et al., 2019; OpenAI, 2018; Lowe et al., 2017; Resnick et al., 2018; Shu & Tian, 2018; Carroll et al., 2019; Suarez et al., 2019; Baker et al., 2019; Bard et al., 2020).”
BackgroundMulti-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward
Sheikh, H. U., & Bölöni, L. L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206879 · Cited by 53 · see citing works
“Despite the recent success of multi-agent reinforcement learning (MARL) in multiplayer games like Dota 2 [4], Quake III Capture-the-Flag [5] and Starcraft II [6] or learning to use tools [7], learning multi-agent cooperation while simultaneously maximizing local rewards is still an open challenge.”
BackgroundHou, C., Zhou, M., Ji, Y., Daian, P., Tramèr, F., Fanti, G. C., & Juels, A. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24188 · Cited by 70 · see citing works
“Although PPO is not immune to this problem, it has been used successfully for multi-agent games [6], [7], [52], and we found it to be more stable than the alternatives in the multi-agent setting.”
MethodDOP: Off-Policy Multi-Agent Decomposed Policy Gradients
Wang, Y., Han, B., Wang, T., Dong, H., & Zhang, C. (2021) · International Conference on Learning Representations · Cited by 50 · see citing works
Orientation and Decision-Making for Soccer Based on Sports Analytics and AI: A Systematic Review
Pu, Z., Pan, Y., Wang, S., Liu, B., Chen, M., Ma, H., & Cui, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123807 · Cited by 49 · see citing works
A Cordial Sync: Going Beyond Marginal Policies for Multi-agent Embodied Tasks
Jain, U., Weihs, L., Kolve, E., Farhadi, A., Lazebnik, S., Kembhavi, A., & Schwing, A. G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_28 · Cited by 47 · see citing works
“Similarly, communication and language in the multi-agent setting has been investigated [32,45,10,61,53,27,83,67,7] in maze-based setups, tabular tasks, or Markov games.”
BackgroundOff-Policy Multi-Agent Decomposed Policy GradientsPreprint
Wang, Y., Han, B., Wang, T., Dong, H., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12322 · Cited by 45 · see citing works
“Cooperative multi-agent reinforcement learning (MARL) has achieved great progress in recent years (Hughes et al., 2018; Jaques et al., 2019; Vinyals et al., 2019; Zhang et al., 2019; Baker et al., 2020; Wang et al., 2020c).”
BackgroundCommunicating with Interactive Articles
Hohman, F., Conlen, M., Heer, J., & Chau, D. H. (2020) · Distill · doi:10.23915/distill.00028 · Cited by 46 · see citing works
Wildfire Front Monitoring With Multiple UAVs Using Deep Q-Learning
Viseras, A., Meißner, M., & Marchal, J. (2021) · IEEE Access · doi:10.1109/access.2021.3055651 · Cited by 44 · see citing works
“Inde-197 pendent learning has been combined with deep Q-learning [11], [32], and with policy gradient methods [31], [33].”
Distributed Deep Reinforcement Learning: A Survey and a Multi-player Multi-agent Learning Toolbox
Yin, Q., Yu, T., Shen, S., Yang, J. J., Zhao, M., Ni, W., Huang, K., Liang, B., & Wang, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1454-4 · Cited by 45 · see citing works
“Players trained for hide-and-seek [44] is similar with JueWu [36], where a SEEDRL [9] DDRL algorithm and a naive self-play are used to prove the multi-agent auto-curricula.”
BackgroundMethodScaling Up Multi-Agent Reinforcement Learning: An Extensive Survey on Scalability Issues
Liu, D., Ren, F., Yan, J. H., Su, G., Gu, W., & Kato, S. (2024) · IEEE Access · doi:10.1109/access.2024.3410318 · Cited by 46 · see citing works
“Although most works regard agents as the only variables and the characteristics of other entities in the environment are unchangeable, the dynamic environment is taken into account in tasks such as SAVETECITY and Hide-and-seek, allowing agents to change their environment [85], [102].”
BackgroundInfluentialReinforcement learning for pursuit and evasion of microswimmers at low Reynolds number
Borra, F., Biferale, L., Cencini, M., & Celani, A. (2022) · Physical Review Fluids · doi:10.1103/physrevfluids.7.023103 · Cited by 43 · see citing works
“Inspired by recent applications of Multi Agent Reinforcement Learning (MARL) [15] to hide-and-seek contests [16, 17], we explore its use as a general model-free framework for discovering effective chase-and-escape strategies at low Reynolds number.”
MethodRevisiting Parameter Sharing in Multi-Agent Deep Reinforcement LearningPreprint
Terry, J. K., Grammel, N., Son, S., Black, B., & Agrawal, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13625 · Cited by 60 · see citing works
Randomized Entity-wise Factorization for Multi-Agent Reinforcement LearningPreprint
Iqbal, S., de Witt, C. A. S., Peng, B., Böhmer, W., Whiteson, S., & Sha, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04222 · Cited by 39 · see citing works
“Several approaches devise attention or graph-neural-network based models for handling variable sized inputs and focus on learning curricula to progress on increasingly large/challenging settings (Long et al., 2020; Baker et al., 2019; Wang et al., 2020c; Agarwal et al., 2019).”
BackgroundBurggräf, P., Wagner, J., & Heinbach, B. (2021) · IEEE Access · doi:10.1109/access.2021.3054563 · Cited by 38 · see citing works
“…we have experienced a surge of Artifi-cial Intelligence (AI) reports in which AI algorithms have proven to outperform humans in complex, yet very specific tasks such as Go [6], complex online multiplayer games [7], Poker [8], hide-and-seek simulations [9] or emulations of old Atari games [10].”
BackgroundGames for Artificial Intelligence Research: A Review and Perspectives
Hu, C., Zhao, Y., Wang, Z., Du, H., & Liu, J. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3410935 · Cited by 51 · see citing works
“Pgx [90] Ludii [89] TAG [88] OpenSpiel [87] Botzone [86] GGP [85] Hanabi [57] ben [56] Fireplace [55] Sabberstone [54] Tales of Tributes [53] VGC [52] Diplomacy [51] Blood bowl [50] Mjx [49] Mjai [48] ColorShapeLinks [47] Grid-based 2D Griddly [98] Stratega [97] Melting Pot [96] GVGAI gym [94]…”
BackgroundOn the Critical Role of Conventions in Adaptive Human-AI CollaborationPreprint
Shih, A., Sawhney, A., Kondic, J., Ermon, S., & Sadigh, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02871 · Cited by 41 · see citing works
“In multi-agent reinforcement learning, techniques such as self-play, cross-training, or opponent learning (Foerster et al., 2018) have been used to observe emergent behavior in complex, physical settings (Nikolaidis & Shah, 2013; Liu et al., 2019; Baker et al., 2020).”
MethodAI in Human-computer Gaming: Techniques, Challenges and Opportunities
Yin, Q., Yang, J. J., Huang, K., Zhao, M., Ni, W., Liang, B., Huang, Y., Wu, S., & Wang, L. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1384-6 · Cited by 44 · see citing works
On the Use and Misuse of Absorbing States in Multi-agent Reinforcement LearningPreprint
Cohen, A., Teng, E., Berges, V.-P., Dong, R.-P., Henry, H., Mattar, M., Zook, A., & Ganguly, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05992 · Cited by 48 · see citing works
“2017) but without positional encodings (Baker et al. 2020).”
InfluentialZhang, Z., Guo, Z., Zheng, H., Li, Z., & Yuan, P. F. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105702 · Cited by 35 · see citing works
LEMMA: A Multi-view Dataset for L Earning Multi-agent Multi-task Activities
Jia, B., Chen, Y., Huang, S., Zhu, Y., & Zhu, S. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58574-7_46 · Cited by 36 · see citing works
“Although simulation-based multi-agent environments [3,59,6] can partially address such an issue, learning from noisy and real visual input in physical work is still essential for understanding collaborative planning behaviors of agents in the context of complex daily tasks.”
BackgroundMethodA social path to human-like artificial intelligence
Duéñez‐Guzmán, E. A., Sadedin, S., Wang, J. X., McKee, K. R., & Leibo, J. Z. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00754-x · Cited by 39 · see citing works
“Sub-stantial research effort has been devoted to balancing this trade-off 20,21 , and some bounds are known 22,23 .”
BackgroundInfluentialLearning Multipursuit Evasion for Safe Targeted Navigation of Drones
Xiao, J., & Feroskhan, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3366871 · Cited by 34 · see citing works
“Furthermore, OpenAI found that multi-agent self-play can induce distinctive emergent strategies toward open-ended growth in complexity [48], which helped solve evolution-required decision-making problems.”
BackgroundAli, M., Sokolov, A. S., Ko, M. J., & Choi, C. (2020) · Journal of Alloys and Compounds · doi:10.1016/j.jallcom.2020.157514 · Cited by 34 · see citing works
Scalable Reinforcement Learning Policies for Multi-Agent Control
Hsu, C. D., Jeong, H., Pappas, G. J., & Chaudhari, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636344 · Cited by 34 · see citing works
“There are also directions that focus on emergent behaviors in multi-agent systems to build cooperative and competitive strategies [34], identify emergent roles [35], coordination and emergent use of tools [36].”
BackgroundDogru, O., Velswamy, K., & Huang, B. (2021) · Engineering · doi:10.1016/j.eng.2021.04.027 · Cited by 34 · see citing works
Near-Optimal Reinforcement Learning with Self-PlayPreprint
Bai, Y., Jin, C., & Yu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12007 · Cited by 34 · see citing works
“Multi-agent RL has achieved significant recent success on traditionally challenging tasks, for example in the game of GO [30, 31], Poker [6], real-time strategy games [33, 22], decentralized controls or multiagent robotics systems [5], autonomous driving [27], as well as complex social scenarios such as hide-and-seek…”
BackgroundAcquiring musculoskeletal skills with curriculum-based reinforcement learning
Chiappa, A. S., Tano, P., Patel, N., Ingster, A., Pouget, A., & Mathis, A. (2024) · Neuron · doi:10.1016/j.neuron.2024.09.002 · Cited by 35 · see citing works
A Survey of Planning and Learning in Games
Duarte, F. F., Lau, N., Pereira, A., & Reis, L. P. (2020) · Applied Sciences · doi:10.3390/app10134529 · Cited by 33 · see citing works
Simultaneous Production and AGV Scheduling using Multi-Agent Deep Reinforcement Learning
Popper, J., Yfantis, V., & Ruskowski, M. (2021) · Procedia CIRP · doi:10.1016/j.procir.2021.11.257 · Cited by 34 · see citing works
“For example, in [14], a mixed competitive and cooperative physics-based environment was presented in which agents compete in a simple game of hide-and-seek.”
BackgroundMultiagent Deep Reinforcement Learning With Demonstration Cloning for Target Localization
Alagha, A., Mizouni, R., Bentahar, J., Otrok, H., & Singh, S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3262663 · Cited by 31 · see citing works
“To push the agents to cooperate, the reward function is also team-based (joint) [34].”
BackgroundMethodInfluentialLearning a Decentralized Multi-arm Motion PlannerPreprint
Ha, H., Xu, J., & Song, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02608 · Cited by 31 · see citing works
“Unlike masked observations as used in [35], our LSTM state encoder does not have a maximum limit on how many agent states can be encoded.”
MethodParameter Sharing is Surprisingly Useful for Multi-Agent Deep Reinforcement Learning.Preprint
Terry, J. K., Grammel, N., Hari, A., Santos, L., Black, B., & Manocha, D. (2020) · arXiv (Cornell University) · Cited by 28 · see citing works
Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach
Mak, S., Xu, L., Pearce, T., Ostroumov, M., & Brintrup, A. M. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104376 · Cited by 31 · see citing works
Reinforcement learning-driven local transactive energy market for distributed energy resources
Zhang, S., May, D. C.-M., Gül, M., & Musı́lek, P. (2022) · Energy and AI · doi:10.1016/j.egyai.2022.100150 · Cited by 28 · see citing works
Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement LearningPreprint
Xie, T., Jiang, N., Wang, H., Xiong, C., & Bai, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04895 · Cited by 28 · see citing works
“GO) [44, 45], robotic manipulation [3, 32], behavior learning in social scenarios [8], and more.”
BackgroundArtificial Intelligence: Arguments for Catastrophic Risk
Bales, A., D’Alessandro, W., & Kirk‐Giannini, C. D. (2024) · Philosophy Compass · doi:10.1111/phc3.12964 · Cited by 27 · see citing works
Spatial Intention Maps for Multi-Agent Mobile Manipulation
Wu, J., Sun, X., Zeng, A., Song, S., Rusinkiewicz, S. M., & Funkhouser, T. A. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561359 · Cited by 27 · see citing works
“Intention-aware multi-agent learning methods have shown promising success—from hide and seek [6] to Starcraft [7]—”
BackgroundHetzel, L., Dudley, J. J., Feit, A. M., & Kristensson, P. O. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3106494 · Cited by 27 · see citing works
Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework
Gizzi, E., Nair, L., Chernova, S., & Sinapov, J. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13864 · Cited by 26 · see citing works
“Baker et al. (2019) use LSTMs within their policy network architecture to enable a set of agents to learn creative policies that involve using objects to succeed in a game of hide-and-seek.”
BackgroundMethodInfluentialGao, H., Zhao, M., Zheng, X., Wang, C., Zhou, L., Wang, Y., Ма, Л., Cheng, B., Wu, Z., & Li, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128482 · Cited by 25 · see citing works
Cooperative dual-actor proximal policy optimization algorithm for multi-robot complex control task
Baltes, J., Akbar, I., & Saeedvand, S. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102960 · Cited by 25 · see citing works
Negotiation and honesty in artificial intelligence methods for the board game of Diplomacy
Kramár, J., Eccles, T., Gemp, I., Tacchetti, A., McKee, K. R., Malinowski, M., Graepel, T., & Bachrach, Y. (2022) · Nature Communications · doi:10.1038/s41467-022-34473-5 · Cited by 25 · see citing works
Navigating the landscape of multiplayer games
Omidshafiei, S., Tuyls, K., Czarnecki, W. M., Santos, F. C., Rowland, M., Connor, J., Hennes, D., Muller, P., Pérolat, J., Vylder, B. D., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19244-4 · Cited by 29 · see citing works
“Chicken Battle Hero Compromise Deadlock Prisoner’s dilemma Stag hunt Assurance Coordination Peace Harmony Concord (Ch) (Ba) (Hr) (Cm) (Dl) (Pd) (Sh) (As) (Co) (Pc) (Ha) (Nc) [ 2 3 1 4 ] [ 3 2 1 4 ] [ 3 1 2 4 ] [ 2 1 3 4 ] [ 1 2 3 4 ] [ 1 3 2 4 ] [ 1 4 2 3 ] [ 1 4 3 2 ] [ 2 4 3 1 ] [ 3 4 2 1 ] [ 3 4 1 2 ] [ 2 4 1 3 ]”
BackgroundA Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation AssurancePreprint
Brittain, M., Yang, X., & Wei, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08353 · Cited by 25 · see citing works
On Centralized Critics in Multi-Agent Reinforcement Learning
Lyu, X., Baisero, A., Xiao, Y., Daley, B., & Amato, C. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14386 · Cited by 24 · see citing works
“Extensive tests based on a centralized critic in a more realistic environment using self-play for hide-and-seek (Baker et al., 2020) have demonstrated impressive results showing emergent tool use.”
BackgroundConcentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems
Fu, Q., Qiu, T., Yi, J., Pu, Z., & Wu, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i9.21165 · Cited by 24 · see citing works
VMAS: A Vectorized Multi-agent Simulator for Collective Robot Learning
Bettini, M., Kortvelesy, R., Blumenkamp, J., & Prorok, A. (2024) · Springer proceedings in advanced robotics · doi:10.1007/978-3-031-51497-5_4 · Cited by 27 · see citing works
Learning to Play No-Press Diplomacy with Best Response Policy IterationPreprint
Anthony, T., Eccles, T., Tacchetti, A., Kramár, J., Gemp, I., Hudson, T. C., Porcel, N., Lanctot, M., Pérolat, J., Everett, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04635 · Cited by 24 · see citing works
“Game-playing has driven AI research since its inception: work on games delivered progress in search, RL and computing equilibria [101, 44, 61, 35, 13, 99, 102, 115, 104, 41, 34] leading to prominent successes in Chess [20], Go [108, 110], Poker [85, 17], multi-agent control domains [10, 79, 6, 112, 124] and video game…”
BackgroundPHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
Netanyahu, A., Shu, T., Katz, B., Barbu, A., & Tenenbaum, J. B. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i1.16167 · Cited by 23 · see citing works
“Recent work on deep reinforcement learning demonstrated promising results (Baker et al. 2019), but the trained policies are constrained to a very small goal space and cannot generalize to more diverse situations without re-training.”
Natural Emergence of Heterogeneous Strategies in Artificially Intelligent Competitive Teams
Deka, A., & Sycara, K. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78743-1_2 · Cited by 22 · see citing works
“[6] explored multi agent reinforcement learning for the game of hide and seek.”
BackgroundInfluentialCooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
Scheikl, P. M., Gyenes, B., Davitashvili, T., Younis, R., Schulze, A., Müller-Stich, B. P., Neumann, G. M., Wagner, M., & Mathis-Ullrich, F. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636193 · Cited by 22 · see citing works
Brandão, B., de Lima, T. W., Soares, A. D. S., Melo, L. C., & Máximo, M. R. O. A. (2022) · IEEE Access · doi:10.1109/access.2022.3189021 · Cited by 22 · see citing works
Multi-Agent Collaboration via Reward Attribution DecompositionPreprint
Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J. E., & Tian, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08531 · Cited by 26 · see citing works
“For example, in Hide and Seek (Baker et al., 2019), it takes agents 2 .”
BackgroundSelf-citationPotential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach
Suman, S., Etemad, A., & Rivest, F. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3127483 · Cited by 21 · see citing works
- Authors
- Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I.
- Affiliations
- OpenAI (United States) (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.07528
- OpenAlex
- W2973525135
- Semantic Scholar
- 33a2c0f3b9a0adc452a13d53f950c0c3c4abb11b
- Cited by
- 424
- References
- 70
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 63 · 2023: 60 · 2022: 69 · 2021: 117 · 2020: 88 · 2019: 7 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- multi-agent competition, hide and seek, multi-agent coordination, intrinsic motivation, intelligence tests, environmental complexity, overcoming obstacles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
