The DOI matches its DataCite record; 65 of the 70 references it cites were found in a registry or an index.
Emergent Tool Use From Multi-Agent Autocurricula
Open accessAbstract
Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For example, Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Baker et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Agents develop exploitative strategies (Baker et al., 2019), learn deception via theory-ofmind (Schulz et al., 2023), and discover betrayal without explicit training (Aitchison et al., 2021) -eluding threat models predicated on an identifiable adversary.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Baker et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Consequently, we used the PPO agent in Experiments III and IV, as PPO is a competent agent used by OpenAI [32].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“[75] Sepp Hochreiter and Jürgen Schmidhuber.”
MethodInfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 2×
“[30] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.”
BackgroundMethodProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 4×
“PPO [58], a policy gradient variant, penalizes large changes to the policy to prevent training instabilities.”
BackgroundInfluentialMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“[57] Emanuel Todorov, Tom Erez, and Yuval Tassa.”
BackgroundMethodArms races between and within species
Dawkins, R., & Krebs, J. R. (1979) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1979.0081 · Cited by 2,214 · cited 2×
“The vast amount of complexity and diversity on Earth evolved due to co-evolution and competition between organisms, directed by natural selection [5].”
BackgroundHigh-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 3×
“We use Generalized Advantage Estimation [3] with horizon length H to estimate the advantage function.”
BackgroundMethodTemporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 3×
“In the context of multi-agent RL, [7], [30], [9], [13] and [10] used self-play with deep RL techniques to achieve super-human performance in Backgammon, Go, Dota, Capture-the-Flag and Starcraft, respectively.”
BackgroundMethodConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Research works printed but not linked: 11
- Kelsey R Allen, Kevin A Smith, and Joshua B Tenenbaum. The tools challenge: Rapid trial-anderror learning in physical problem solving. arXiv preprint arXiv:1907.09620 , 2019. · arXiv
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Rene Baillargeon and Susan Carey. Core cognition and beyond: The acquisition of physical and numerical knowledge. In In S. Pauen (Ed.), Early Childhood Development and Later Outcome , pp. 33-65. University Press, 2012.
- Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations , 2018.
- Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2019b.
- Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsically-motivated, self-aware agents. In Advances in Neural Information Processing Systems , pp. 8388-8399, 2018.
- Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017. · arXiv
- Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , pp. 6379-6390, 2017.
- Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning , pp. 2778-2787, 2017.
- Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems , pp. 2244-2252, 2016.
- Larry Yaeger. Computational genetics, physiology, metabolism, neural systems, learning, vision, and behavior or poly world: Life in a new context. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXITY-PROCEEDINGS VOLUME, volume 17, pp. 263-263. ADDISON-WESLEY PUBLISHING CO, 1994.
Cited by
Sorted by citations
Items 401–428 of 428
캐릭터 복싱 과제에서 GAN 기반 접근법과 강화학습의 효과성 탐구
Son, S., & Kwon, T. (2023) · Journal of the Korea Computer Graphics Society · doi:10.15701/kcgs.2023.29.4.7
Path exploration with random network distillation on multi-agent reinforcement learning
Charoenpitaks, K. (2019) · Chula Digiverse · doi:10.58837/chula.the.2019.162
Adversarial Multi-Agent Reinforcement Learning for Fault-Tolerant Design of Complex Systems
Grimstad, J., & Morozov, A. S. (2023) · European Safety and Reliability Conference (ESREL) · doi:10.3850/978-981-18-8071-1_p375-cd
A STUDY OF EMERGENT COOPERATIVE BEHAVIORS OF MULTI-AGENT SYSTEMS USING REINFORCEMENT LEARNING
(2023) · Issues in Information Systems · doi:10.48009/1_iis_2023_116
“OpenAI’s competitive multi-agent hide and seek game using standard reinforcement algorithms outlines how even following a simple objective can cause agents to develop various strategies through their own trial and error (Baker et al., 2020).”
Moreno‐Bote, R., & Grytskyy, D. (2023) · Physics of Life Reviews · doi:10.1016/j.plrev.2023.10.024
Emergence of tool construction and tool use through hierarchical reinforcement learning
Li, Q., & Choe, Y. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-96104-2.00008-7
Simulation and Comparison of Reinforcement Learning Algorithms
Spyropoulos, K., & Sotiropoulos, D. N. (2023) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa59645.2023.10345966
“[2]–[4] This and other similar experiments, which can be resolved using reinforcement learning, can be implemented in real world problems.”
BackgroundEmergent Resource Exchange and Tolerated Theft Behavior Using Multiagent Reinforcement Learning
Garbus, J., & Pollack, J. B. (2024) · Artificial Life · doi:10.1162/artl_a_00423 · Cited by 1
“Indeed, a game in which half the time is spent doing nothing would likely not be very interesting for many; however, when agents are not given an easy way to further optimize an objective, they can find novel, sometimes unexpected methods to do so, given time (Baker et al., 2020).”
BackgroundLösel, G. (2024) · transcript Verlag eBooks · doi:10.1515/9783839470244
Lösel, G. (2024) · Theater · doi:10.14361/9783839470244
Psychological Restrain for Fault Detection in Reinforcement Learning
Phour, H., Gautam, N., & Gaurav, A. (2023) · International Conference on Computing, Communication, and Intelligent Systems (ICCCIS) · doi:10.1109/icccis60361.2023.10425110
“In [6], Bowen Baker et al. on “Emergent Tool Use From Multi-Agent Autocurricula” the seekers learn to bring a box to locked ramp in order to jump on the top of the box and then “surf” it to the hider’s shelter.”
BackgroundCommunicative and Cooperative Learning for Multi-agent Indoor Navigation
Zhu, F., Lee, V. C. S., & Liu, R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_22
PokerOWL: A Multi-Agent Poker Environment for Implementing and Evaluating Open-World LearningPreprint
Chiu, M., & Kejriwal, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4677706/v1
Learning to Play Foosball: System and Baselines
Moos, J., Derstroff, C., Schröder, N., & Clever, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611321 · Cited by 1
“Self-play has been widely used for a long time and has been successfully applied to a range of challenging domains [5], [6], [7], [8], [9], [10], [11].”
BackgroundEvolving Complex Environments in Evolution Gym using Large Language Models
Aki, F., Ikeda, R., Saito, T., Regan, C., & Oka, M. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626609
“…open-ended evolution (OEE) algorithms developed, co-evolution, the simultaneous evolution of the agent and the environment, is a remarkably advantageous approach, as demonstrated in studies such as hide and seek (Baker et al., 2019) or minimal criterion co-evolution (Brant and Stanley, 2017, 2020).”
BackgroundMulti-time scale hierarchical trust domain leads to the improvement of MAPPO algorithm
Guo, Z., Sun, L., Zhao, G., et al. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10662712
“DeepMind’s AlphaStar surpassed professional-level performance in StarCraft II [3], Ope-nAIFive defeated the world champion in Dota II [4], OpenAI demonstrates the emergence of human-like tools using agent behavior through multi-agent learning [5].”
BackgroundMarão, L. A. B., Cintra, L. E., & Caurin, G. A. P. (2024) · 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case59546.2024.10711704
“The goals are sequential and located at [40 , 20] , [30 , 2] , and [6 , 18] respectively.”
BackgroundStein, C. (2024) · Trayectorias Humanas Trascontinentales · doi:10.25965/trahs.6416
Using Curriculum to Train Multisensory Foraging DRL Agents
Filho, R. F. F., Santos, A. M. M., Rodrigues, H. R., Nogueira, Y. L. B., Vidal, C. A., Cavalcante-Neto, J. B., & Serafim, P. B. S. (2024) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames.2024.241119
“In some applications, agents receive additional numerical information, such as current energy level, and use proprioceptive sensors to gather information about their own state [Baker et al. 2020].”
Modernizing Smart Homes into Cognitive Residences
Sharma, G., Singh, K. R., Udayakumar, R., et al. (2024) · doi:10.1109/i3ceet61722.2024.10993752
Weyland, L. F., Schadd, M. P. D., & Henderson, H. C. (2025) · International Conference on Military Communication and Information Systems (ICMCIS) · doi:10.1109/icmcis64378.2025.11047897
NAMOSIM: a Robot Motion Planner for Navigation Among Movable Obstacles
Brown, D., Saraydaryan, J., Renault, B., et al. (2025) · The Journal of Open Source Software · doi:10.21105/joss.08816
“Another innovative related work (Baker et al., 2020) involved both multiple agents and movable obstacles, but was not designed for navigation nor for robotics applications.”
MADmax: Multi-agent Trust Dynamics and Influence Maximization
Sorensen, A., Sweitzer, M. D., Naugle, A., Doyle, C., & Krofcheck, D. (2025) · Springer proceedings in complexity · doi:10.1007/978-3-031-89692-7_5
Legal Zero-Days: a novel risk vector for advanced AI systems
Sadler, G., & Sherburn, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01189-x
“Current AI systems have demonstrated remarkable abilities to identify and exploit loopholes in rule-based systems, from video games [4] to trading algorithms.”
Ghosh, S., Chandroth, A. S., Ramakrishnan, N., & Chandrasekharan, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40581-4_2
Open-Ended Learning Leads to Generally Capable Agents
Open Ended Learning Team, Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.12808 · Cited by 56
“In addition, we see the agent being capable in tasks that not only are explicitly held-out from training, but lie far outside of its training distributions, including versions of hide and seek (Baker et al., 2020) and capture the flag (Jaderberg et al., 2019).”
BackgroundInfluentialCausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
Ahmed, O., Träuble, F., Goyal, A., Neitz, A., Bengio, Y., Schölkopf, B., Wüthrich, M., & Bauer, S. (2020) · arXiv · doi:10.48550/arxiv.2010.04296 · Cited by 50
“…in domains such as game playing (Mnih et al., 2013; Silver et al., 2017), learning motor control for high-dimensional simulated robots (Akkaya et al., 2019), multi-agent settings (Baker et al., 2019; Berner et al., 2019) and for studying transfer in the context of meta-learning (Yu et al., 2019).”
Background
Metadata
- Authors
- Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I.
- Affiliations
- OpenAI (United States) (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.07528
- OpenAlex
- W2973525135
- Semantic Scholar
- 33a2c0f3b9a0adc452a13d53f950c0c3c4abb11b
- Cited by
- 424
- References
- 70
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 63 · 2023: 60 · 2022: 69 · 2021: 117 · 2020: 88 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- multi-agent competition, hide and seek, multi-agent coordination, intrinsic motivation, intelligence tests, environmental complexity, overcoming obstacles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
