The DOI matches its DataCite record; 65 of the 70 references it cites were found in a registry or an index.
Emergent Tool Use From Multi-Agent Autocurricula
Open accessAbstract
Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For example, Baker et al. (2019) showed that even a simple game of hide and seek can lead to sophisticated tool use and coordination by MARL agents.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Baker et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Agents develop exploitative strategies (Baker et al., 2019), learn deception via theory-ofmind (Schulz et al., 2023), and discover betrayal without explicit training (Aitchison et al., 2021) -eluding threat models predicated on an identifiable adversary.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Baker et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Consequently, we used the PPO agent in Experiments III and IV, as PPO is a competent agent used by OpenAI [32].”
From Playtesting: What is Beyond Personas (2023) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“[75] Sepp Hochreiter and Jürgen Schmidhuber.”
MethodInfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 2×
“[30] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.”
BackgroundMethodProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 4×
“PPO [58], a policy gradient variant, penalizes large changes to the policy to prevent training instabilities.”
BackgroundInfluentialMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“[57] Emanuel Todorov, Tom Erez, and Yuval Tassa.”
BackgroundMethodArms races between and within species
Dawkins, R., & Krebs, J. R. (1979) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1979.0081 · Cited by 2,214 · cited 2×
“The vast amount of complexity and diversity on Earth evolved due to co-evolution and competition between organisms, directed by natural selection [5].”
BackgroundHigh-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 3×
“We use Generalized Advantage Estimation [3] with horizon length H to estimate the advantage function.”
BackgroundMethodTemporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 3×
“In the context of multi-agent RL, [7], [30], [9], [13] and [10] used self-play with deep RL techniques to achieve super-human performance in Backgammon, Go, Dota, Capture-the-Flag and Starcraft, respectively.”
BackgroundMethodConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Flagged first, then by citations
5 of the 70 references couldn’t be matched to a known paper.
Research works printed but not linked: 11
- Kelsey R Allen, Kevin A Smith, and Joshua B Tenenbaum. The tools challenge: Rapid trial-anderror learning in physical problem solving. arXiv preprint arXiv:1907.09620 , 2019. · arXiv
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Rene Baillargeon and Susan Carey. Core cognition and beyond: The acquisition of physical and numerical knowledge. In In S. Pauen (Ed.), Early Childhood Development and Later Outcome , pp. 33-65. University Press, 2012.
- Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations , 2018.
- Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2019b.
- Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsically-motivated, self-aware agents. In Advances in Neural Information Processing Systems , pp. 8388-8399, 2018.
- Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017. · arXiv
- Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems , pp. 6379-6390, 2017.
- Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning , pp. 2778-2787, 2017.
- Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems , pp. 2244-2252, 2016.
- Larry Yaeger. Computational genetics, physiology, metabolism, neural systems, learning, vision, and behavior or poly world: Life in a new context. In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXITY-PROCEEDINGS VOLUME, volume 17, pp. 263-263. ADDISON-WESLEY PUBLISHING CO, 1994.
Cited by
Sorted by citations
Items 201–300 of 428
Zero-shot sim-to-real transfer using Siamese-Q-Based reinforcement learning
Zhang, Z., Xie, S., Zhang, H., Luo, X., & Yu, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102664 · Cited by 7
Human strategy learning-based multi-agent deep reinforcement learning for online team sports game
Lee, S., Lee, G., Kim, W., Kim, J., Park, J., & Cho, K. (2025) · IEEE Access · doi:10.1109/access.2025.3531435 · Cited by 7
Collision Detection and Avoidance for Multi-UAV based on Deep Reinforcement Learning
Wang, G., Liu, Z., Xiao, K., Xu, Y., Yang, L., & Wang, X. (2021) · Chinese Control Conference (CCC) · doi:10.23919/ccc52363.2021.9550712 · Cited by 7
“Most of the researches are focusing on the single-UAV collision avoidance[19] or staying at the simple game environments for multiple agents[20].”
BackgroundReal-time Virtual Simulation Platform for Multi-UVA hunting target using Deep Reinforcement Learning
Zhu, K., Zong, Q., & Zhang, R. (2021) · Chinese Control Conference (CCC) · doi:10.23919/ccc52363.2021.9550261 · Cited by 7
“The current common environments include: (1) The Multiagent emergence environment[13] and the popular Gym environment developed by OpenAI.”
BackgroundCooperation and Learning Dynamics under Wealth Inequality and Diversity in Individual Risk
Merhej, R., Santos, F. P., Melo, F. S., & Santos, F. C. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13519 · Cited by 7
“Implementing a centralized critic with decentralized actors is another form of indirect communication and information sharing among agents that can increase performance and cooperation (Baker et al., 2019; Foerster et al., 2018b; Lowe et al., 2017).”
BackgroundHow a machine can understand the command intent
Schadd, M. P. D., Sternheim, A. M., Blankendaal, R., et al. (2022) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/15485129221115736 · Cited by 7
“14 Therefore, a requirement is that the function clearly defines areas, time-spans, unit indications, and all other aspects, so that no interpretation freedom remains.”
Exploring reinforcement learning approaches for drafting in collectible card games
Vieira, R. E. S., Tavares, A. R., & Chaimowicz, L. (2022) · Entertainment Computing · doi:10.1016/j.entcom.2022.100526 · Cited by 8
Hicks, C., Mavroudis, V., Foley, M., Davies, T., Highnam, K., & Watson, T. (2023) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3605764.3623986 · Cited by 10
“It is remarkable however, that the CAGE challenge has a large number of agents relative to the wider literature where 20 is usually the maximum (e.g., [48, 58]) and 2-6 is far more common (e.g., [6]).”
BackgroundMethodInfluentialPretorius, A., Cameron, S., van Biljon, E., Makkink, T., Mawjee, S., du Plessis, J., Shock, J. P., Laterre, A., & Beguir, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07777 · Cited by 7
“There is evidence to suggest that this hypothesis could be true for reinforcement learning agents, e.g. in Baker et al. (2019), and therefore MARL could play a key role in sustained innovation for networked system control.”
BackgroundEmergent Road Rules In Multi-Agent Driving EnvironmentsPreprint
Pal, A., Philion, J., Liao, Y.-H., & Fidler, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.10753 · Cited by 9
The Fittest Wins: A Multistage Framework Achieving New SOTA in ViZDoom Competition
Li, S., Xu, J., Dong, H., Yu, Y., Yuan, C., Sun, P., & Han, L. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3259724 · Cited by 6
“In addition to AlphaGo [8], SP has been demonstrated effective in many other complex games, such as hide-and-seek environment [17] and DOTA 2 [9].”
BackgroundMulti-Agent Visual Coordination Using Optical Wireless Communication
Nakagawa, H., & Kanezaki, A. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3304905 · Cited by 6
“Baker et al. [2] showed that agents with a limited 135 ◦ FOV learn how to use tools through cooperative and adversarial behavior.”
BackgroundOlmez, S., Birks, D., Heppenstall, A. J., & Ge, J. (2024) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2024.102141 · Cited by 6
“In response, and following recent advances in other areas of social simulation ((Baker et al., 2019; Sert et al., 2020)), this article proposes that using RL in ABMs of crime event dynamics may increase the accuracy of modelled behaviours as agents learn through assessing their dynamic environment…”
Han, Q., Wang, X., & Shen, W. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3444597 · Cited by 8
“For MAPPO implementation, we employ two separate networks for policy and value functions, alongside GAE, advantage normalization, and a PopArt value normalizer utilizing running statistics for target regression normalization [59].”
MethodAlagha, A., Mizouni, R., Singh, S., Bentahar, J., & Otrok, H. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101447 · Cited by 6
Sequential memory improves sample and memory efficiency in episodic control
Freire, I. T., Amil, A. F., & Verschure, P. F. M. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00950-3 · Cited by 6
“A number of sources of inefficiency and slowness in DRL have been already identified 8 .”
A deep reinforcement learning approach and its application in multi-USV adversarial game simulation
Rao, J., Wang, C., Liu, M., Chen, J., Lei, J., & Giernacki, W. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06380-x · Cited by 6
“The output of the critic network, i.e., the centralized state value function with training parameters 𝜙 , is: 1 ≤ i ≤ k t is the encoded observation of all surviving agents, and the RSA (residual self-attention) is used for encoding processing [29, 47].”
Reid, C., & Mukhopadhyay, S. (2020) · doi:10.1109/iccr51572.2020.9344374 · Cited by 6
“We believe that multi-agent systems, which naturally require multiple agents but which, at the time of this writing, often simply share parameters among those agents (see, for example, [22]), are a natural area for further exploration.”
BackgroundTowards Incorporating AI into the Mission Planning Process
Kane, S., Moody, V., & Harradon, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77772-2_14 · Cited by 6
A Deep Reinforcement Learning Environment for Particle Robot Navigation and Object Manipulation
Shen, J., Xiao, E., Liu, Y., & Feng, C. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811965 · Cited by 6
“Multi-Agent-Emergence-Environments [23] is solely a hide-and-seek game and every agent within a team shares its observations.”
D2A: Operating a Service Function Chain Platform With Data-Driven Scheduling Policies
Kramer, P., Diederich, P., Kramer, C., Pries, R., Kellerer, W., & Blenk, A. (2022) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3177694 · Cited by 6
“This module the current μ VNF to into relation to other μ VNFs [42], [43], [44].”
BackgroundScaling up Deep Reinforcement Learning for Intelligent Video Game Agents
Debner, A. (2022) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp55677.2022.00050 · Cited by 6
Blockchain-Assisted Demonstration Cloning for Multiagent Deep Reinforcement Learning
Alagha, A., Bentahar, J., Otrok, H., Singh, S., & Mizouni, R. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3316078 · Cited by 6
“Baker et al. [29] tackle the problem of Hide-and-Seek, where a vision-based reward is designed that rewards seekers in each step if they keep hiders within their sight, and rewards hiders in each step otherwise.”
BackgroundMethodAn online cost minimization of the slice broker based on deep reinforcement learning
Gohar, A., & Nencioni, G. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110198 · Cited by 7
Considerations for Comparing Video Game AI Agents with Humans
Madan, C. R. (2020) · Challenges · doi:10.3390/challe11020018 · Cited by 7
Artificial Agents Learn Flexible Visual Representations by Playing a Hiding Game.Preprint
Weihs, L., Kembhavi, A., Han, W., Herrasti, A., Kolve, E., Schwenk, D., Mottaghi, R., & Farhadi, A. (2019) · arXiv (Cornell University) · Cited by 6
Grounding Artificial Intelligence in the Origins of Human BehaviorPreprint
Nisioti, E., & Moulin-Frier, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08564 · Cited by 13
“In particular, multiagent autocurricula have leveraged the feedback effect that multiagent dynamics have on resource availability [46], as well as arms races between competing groups [5].”
BackgroundInfluentialThe Power of Exploiter: Provable Multi-Agent RL in Large State SpacesPreprint
Jin, C., Liu, Q., & Yu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03352 · Cited by 31
“…(Brown & Sandholm, 2019), real-time strategy games (Vinyals et al., 2019; OpenAI, 2018), decentralized controls or multiagent robotics systems (Brambilla et al., 2013), autonomous driving (Shalev-Shwartz et al., 2016), as well as complex social scenarios such as hide-and-seek (Baker et al., 2020).”
BackgroundCompositionally-Restricted Attention-Based Network for Materials Property PredictionPreprint
Wang, A., Kauwe, S., Murdock, R., & Sparks, T. D. (2020) · ChemRxiv · doi:10.26434/chemrxiv.11869026 · Cited by 6
Existential Risk from Power-Seeking AI
Carlsmith, J. (2025) · Oxford University Press eBooks · doi:10.1093/9780191979972.003.0025 · Cited by 5
M3ARL: Moment-Embedded Mean-Field Multi-Agent Reinforcement Learning for Continuous Action Space
Tang, H., Hu, Y., Zhao, F., Yan, J., Dong, T., & Ding, W. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448058 · Cited by 6
“With the blooming of deep reinforcement research in recent years, multi-agent reinforcement learning (MARL) has enjoyed great achievements in strategic games [1, 2], multi-player computer games [3, 4], autonomous driving [5], and emergent behaviors [6, 7].”
BackgroundCompeting for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation
Saeed, S. U., Huang, S.-Q., Ramalhinho, J., Gayo, I. J. M. B., Montaña-Brown, N., Bonmati, E., Pereira, S. P., Davidson, B. R., Barratt, D., Clarkson, M. J., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3474094 · Cited by 8
“Thus far, the focus for self-play research has mostly been to use learning in the context of existing zero-sum games e.g., Chess [80], Go [79], Poker [29] and Hide and Seek [5].”
Credible Negotiation for Multi-agent Reinforcement Learning in Long-term Coordination
Gu, T., Zhi, T., Bao, X., & Chang, L. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3706110 · Cited by 8
Deep replacement: Reinforcement learning based constellation management and autonomous replacement
Kopacz, J., Roney, J. A., & Herschitz, R. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104316 · Cited by 5
Visual Perspective Taking for Opponent Behavior Modeling
Chen, B., Hu, Y., Kwiatkowski, R., Song, S., & Lipson, H. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9562028 · Cited by 5
“[11] trains the agents to play the game on low-dimensional observations and found that more complex tool usage behaviors can emerge.”
BackgroundReinforcement Learning with Success Induced Task Prioritization
Nesterova, M. V., Skrynnik, A., & Panov, A. I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19493-1_8 · Cited by 5
“Several papers address the last one, showing that it is possible to generate a curriculum automatically by confronting several agents [8,1].”
DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
Yang, Z., Jin, H., Ding, R. G., You, H., Fan, G., Wang, X., & Zhou, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26288 · Cited by 6
“Recent years have seen great success of multi-agent reinforcement learning (MARL) in unconstrained multi-agent systems (MAS), such as video games (Hughes et al. 2018; Jaques et al. 2019; Baker et al. 2020), and many others.”
BackgroundStackelberg Games for Learning Emergent Behaviors During Competitive Autocurricula
Yang, B., Zheng, L., Ratliff, L., Boots, B., & Smith, J. R. (2023) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra48891.2023.10160875 · Cited by 9
Fuzzy Feedback Multiagent Reinforcement Learning for Adversarial Dynamic Multiteam Competitions
Fu, Q., Pu, Z., Pan, Y., Qiu, T., & Yi, J. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3363053 · Cited by 6
“Similar problems have been investigated in single-agent studies [4] as well as small-scale multiagent studies [5].”
BackgroundMethodResultInfluentialChoi, M., Guo, S., Koilias, A., Volonte, M., Kao, D., & Mousas, C. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3688006 · Cited by 5
“Baker et al. [5] presented a reinforcement learning algorithm to train virtual agents to play hide and seek cooperatively.”
Applied Machine Learning for Games: A Graduate School Course
Zeng, Y., Shah, A., Thai, J., & Zyda, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17849 · Cited by 5
“For instance, multi-agent Hide-and-Seek (Baker et al. 2019), OpenAI Five (Berner et al.”
BackgroundData-driven behavioural modelling for military applications
Schadd, M. P. D., de Reus, N. M., Uilkema, S., et al. (2022) · Journal of Defence & Security Technologies · doi:10.46713/jdst.004.02 · Cited by 5
“A difficulty with reinforcement learning is that the reward function has to be carefully crafted and any error in the simulator can be exploited and lead to learning undesired behaviour [6].”
BackgroundDrafting in Collectible Card Games via Reinforcement Learning
Vieira, R. E. S., Tavares, A. R., & Chaimowicz, L. (2021) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames_estendido.2021.19743 · Cited by 4
Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement LearningPreprint
Shao, J., Zhang, H., Jiang, Y., He, S., & Ji, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12957 · Cited by 6
“Recently, multi-agent reinforcement learning has shown strong performance in games Berner et al. (2019); Samvelyan et al. (2019); Ye et al. (2020); Jaderberg et al. (2019) and real-world phenomena such as the emergence of tool usage Baker et al.”
InfluentialLearning-based decision making for safe and scalable autonomous separation assurance
Brittain, M. (2021) · doi:10.31274/etd-20210609-24 · Cited by 6
Discovering Multi-Agent Auto-Curricula in Two-Player Zero-Sum Games.Preprint
Feng, X., Slumbers, O., Yang, Y., Wan, Z., Liu, B., McAleer, S., Wen, Y., & Wang, J. (2021) · arXiv (Cornell University) · Cited by 5
Curiosity-driven learning in artificial intelligence and its applications
Sun, C. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/172831 · Cited by 6
Exploring the Benefits of Teams in Multiagent Learning
Radke, D., Larson, K. S., & Brecht, T. B. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/65 · Cited by 5
Effect of Artificial intelligence in the field of games on humanity
Khder, M. A., Bahar, A. Y., & Fujo, S. W. (2022) · ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS) · doi:10.1109/icetsis55481.2022.9888927 · Cited by 4
“Hide and seek is a great idea from OpenAI to test unexpected moves and enforce collaboration between the AI agents [13].”
BackgroundDEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards
Wan, S., Tang, Y., Tian, Y., & Kaneko, T. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/477 · Cited by 8
Mean-Field Aided QMIX: A Scalable and Flexible Q-Learning Approach for Large-Scale Agent Groups
Zhang, E., Tang, H., Zhang, X.-P. S., & Ding, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887881 · Cited by 4
“Many critical tasks require multiple agents to operate within the same environment, including multiplayer computer games [1, 2], autonomous driving [3], and unmanned aerial vehicle control [4, 5].”
Curiosity Driven Multi-agent Reinforcement Learning for 3D Game Testing
Ferdous, R., Kifetew, F. M., Prandi, D., & Susi, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962505 · Cited by 5
“Recent years have seen remarkable progress in RL in solving immensely challenging multi-player real strategy video games [12], [14], [11], [2].”
Daylamani-Zad, D., & Angelides, M. C. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2020.2989636 · Cited by 5
“Yet, RL shows clear advantage in allowing emergent behaviour in multi-agent environment with rich reward signals and seemingly random environmental rewards [31], [32], [33].”
BackgroundLearning to Deceive in Multi-agent Hidden Role Games
Aitchison, M., Benke, L., & Sweetser, P. (2021) · Communications in Computer and Information Science · doi:10.1007/978-3-030-91779-1_5 · Cited by 4
“While many existing MARL environments have explored cooperation between teammates ([13, 30, 24, 19, 2] only ProAvalon [33] requires cooperation with unknown teammates.”
BackgroundResultEvaluating Navigation Behavior of Agents in Games using Non-Parametric Statistics
Colbert, I., & Saeedi, M. (2022) · IEEE Conference on Games (CoG) · doi:10.1109/cog51982.2022.9893600 · Cited by 5
“The development of non-playable characters (NPCs) has received increased attention in recent years due to impressive successes in training highly skilled agents to play complex games [1], [2].”
BackgroundMethodTaşbaş, A. S., Şahin, S. O., & Üre, N. K. (2023) · AIAA SciTech Forum · doi:10.2514/6.2023-1077 · Cited by 4
“For example, trajectory generation in drone racing [3], large-scale strategy management in Dota 2 game [4], and autocurricula training in contentious multi-agent environment [5] are the certain illustrative RL based solutions.”
BackgroundTowards a Better Understanding of Learning with Multiagent Teams
Radke, D., Larson, K. S., Brecht, T. B., & Tilbury, K. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/31 · Cited by 5
Curriculum based Reinforcement Learning for traffic simulations
Makri, S., & Charalambous, P. (2023) · Computers & Graphics · doi:10.1016/j.cag.2023.04.009 · Cited by 4
Mounsif, M., Lengagne, S., Thuilot, B., & Adouane, L. (2020) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit49905.2020.9263931 · Cited by 4
“In some cases, the author investigate whether the agent’s previously learned representations can be repurposed by modifying slightly the task [8], [9].”
BackgroundVeith, E. M., Wenninghoff, N., & Frost, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13601 · Cited by 13
“How much system analysis can benefit from a Deep Reinforcement Learning (DRL)-based agent exploring the system has been documented by Baker et al. [2], where a simulation of hide-and-seek games has uncovered bugs in the underlying 3D engine as an unintentional side effect.”
MethodCompeting AI: How does competition feedback affect machine learning?Preprint
Ginart, A. A., Zhang, E., Kwon, Y., & Zou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06797 · Cited by 4
“[30] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch.”
BackgroundDeep reinforcement learning models the emergent dynamics of human cooperationPreprint
McKee, K. R., Hughes, E., Zhu, T., Chadwick, M. J., Koster, R., Castañeda, A. G., Beattie, C., Graepel, T., Botvinick, M., & Leibo, J. Z. (2021) · arXiv (Cornell University) · Cited by 4
Augmenting Automated Game Testing with Deep Reinforcement LearningPreprint
Bergdahl, J., Gordillo, C., Tollmar, K., & Gisslén, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15819 · Cited by 10
“Moreover, there are plenty of examples where RL algorithms have found unintended mechanics in complex scenarios [2].”
BackgroundMinimizing Communication while Maximizing Performance in Multi-Agent Reinforcement LearningPreprint
Vijay, V. K., Sheikh, H. U., Majumdar, S., & Phielipp, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08482 · Cited by 4
“In Hide-and-Seek Baker et al. [2019], MARL has been shown to produce complex emergent behavior, including the use of tools in order to accomplish goals.”
BackgroundMethodInfluentialLevel generation and style enhancement -- deep learning for game development overviewPreprint
Migdał, P., Olechno, B., & Podgórski, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07397 · Cited by 2
“These methods learn from interacting with the game environment, learning human-like strategies [14], and exploiting unintended properties of game physics and level maps [15].”
AI in Games: Techniques, Challenges and OpportunitiesPreprint
Yin, Q., Yang, J. J., Ni, W., Liang, B., & Huang, K. (2021) · arXiv (Cornell University) · Cited by 4
MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning
Samvelyan, M., Khan, A., Dennis, M., Jiang, M., Parker-Holder, J., Foerster, J., Raileanu, R., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2303.03376 · Cited by 8
Abmarl: Connecting Agent-Based Simulations with Multi-Agent Reinforcement Learning
Rusu, E., & Glatt, R. (2021) · The Journal of Open Source Software · doi:10.21105/joss.03424 · Cited by 3
Tractable large-scale deep reinforcement learning
Sarang, N., & Poullis, C. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103689 · Cited by 3
ScaRLib: A Framework for Cooperative Many Agent Deep Reinforcement Learning in Scala
Domini, D., Cavallari, F., Aguzzi, G., & Mirko Viroli, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35361-1_3 · Cited by 3
Olmez, S., Heppenstall, A. J., Ge, J., et al. (2024) · Journal of Simulation · doi:10.1080/17477778.2024.2375446 · Cited by 3
“Due to computational complexity of training RL agents with neural networks (Baker et al., 2019; Juliani et al., 2018; Olmez, Birks, et al., 2022; Sutton & Barto, 2018), the experiments are kept concise.”
Butt, N. A., Awais, M. M., Shahzadi, S., Kim, T.-H., & Ashraf, I. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102182 · Cited by 3
“The study Baker et al. (2019) uses imitation learning as a training strategy to examine emergent behaviors in a multi-agent scenario.”
Martínez-Gil, F., & Gil-Magraner, E. (2025) · Entertainment Computing · doi:10.1016/j.entcom.2025.100941 · Cited by 4
Modelling Cooperation in Network Games with Spatio-Temporal Complexity
Bakker, M. A., Everett, R., Weidinger, L., Gabriel, I., Isaac, W. S., Leibo, J. Z., & Hughes, E. (2021) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/sjez3723 · Cited by 4
Ingel, A., Makkeh, A., Corcoll, O., & Vicente, R. (2022) · Entropy · doi:10.3390/e24030401 · Cited by 4
Whole-Body MPC and Dynamic Occlusion Avoidance: A Maximum Likelihood Visibility Approach
Ibrahim, I., Farshidian, F., Preisig, J., et al. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811536 · Cited by 3
Discovering Fatigued Movements for Virtual Character Animation
Cheema, N., Xu, R., Kim, N. H., Hämäläinen, P., Golyanik, V., Habermann, M., Theobalt, C., & Slusallek, P. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618176 · Cited by 4
“2020] and physics-based simulation, paired with Deep Reinforcement Learning (DRL) [Baker et al. 2019; Jiang et al. 2019; Pathak et al. 2017; Peng et al. 2021; Yin et al. 2021].”
MethodInfluentialLearning Diverse Risk Preferences in Population-Based Self-Play
Jiang, Y., Liu, Q., Ma, X., Li, C., Yang, Y., Yang, J. J., Liang, B., & Zhao, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29188 · Cited by 3
Ibrahim, I., Gillis, J., Decré, W., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611529 · Cited by 3
“In [19], the authors endow the agents with sight by using an array of 30 simulated rays arranged uniformly around the agent that cover a cone of interest, representing the agent’s field of view, similar to a simulated lidar.”
BackgroundTraining Interactive Agent in Large FPS Game Map with Rule-enhanced Reinforcement Learning
Zhang, C., Hu, H., Zhou, Y., Cao, Q., Liu, R., Wei, W., & Liu, E. S. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645654 · Cited by 3
“The depth map and the Lindar [8] is input into the feature extraction module.”
MethodPleines, M., Jitsev, J., Preuß, M., & Zimmer, F. (2020) · IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231802 · Cited by 4
“Such game environments are also more and more combined with additional information beyond pure pixels as for hide-and-seek [5] and AlphaStar [6].”
BackgroundShould I Tear down This Wall? Optimizing Social Metrics by Evaluating Novel Actions
Kramár, J., Rabinowitz, N. C., Eccles, T., & Tacchetti, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72376-7_7 · Cited by 4
“Multi-agent reinforcement learning has received considerable attention in recent years, both as a training paradigm to construct powerful individual agents [2,35,3], and as a tool to study the role and emergence of human pro-social inductive biases and institutions in complex environments [20,25,26,32].”
BackgroundThe Virtual Machine Learning Laboratory with Visualization of Algorithms Execution Process
Kholoshnia, V. D., & Boldyreva, E. A. (2020) · Smart innovation, systems and technologies · doi:10.1007/978-981-15-5584-8_19 · Cited by 3
Investigating Deep Q-Network Agent Sensibility to Texture Changes on FPS Games
Serafim, P. B. S., Nogueira, Y. L. B., Vidal, C. A., Cavalcante-Neto, J. B., & Filho, R. F. F. (2020) · Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) · doi:10.1109/sbgames51465.2020.00025 · Cited by 3
“They cover a broad range of challenges, like solving dozens of Atari games [2], specific harder Atari games [3], all Atari games [4], First-Person Shooters (FPS) [5], board games [6]–[8], Multi-Agent games [9], [10], and Real-Time Strategy games [11].”
BackgroundFew-Shot System Identification for Reinforcement Learning
Farid, K., & Sakr, N. (2021) · Asia-Pacific Conference on Intelligent Robot Systems (ACIRS) · doi:10.1109/acirs52449.2021.9519314 · Cited by 3
“RL research shows the reliability in learning strategies and policies in a vast range of planning and control tasks in different fields like finance, games, and robotics [8] [21] [22] [23] [24] [29].”
BackgroundKoutras, D. I., Kapoutsis, A. C., Amanatiadis, A. A., & Kosmatopoulos, E. B. (2021) · Electronics · doi:10.3390/electronics10222751 · Cited by 4
“The recent breakthroughs in Reinforcement Learning (RL), in terms of both algorithms and hardware acceleration, have spawned methodologies capable of achieving above human-level performance in high-dimensional, non-linear setups, such as the game of Go [11], atari games [12], multi-agent collaboration [13], robotic ma…”
BackgroundTran, J., Gajewski, L., Varghese, N., Pfeifer, P., Krahl, S., & Moser, A. (2021) · IEEE PES Innovative Smart Grid Technologies Europe · doi:10.1109/isgteurope52324.2021.9640167 · Cited by 3
Curiosity-Driven Exploration Effectiveness on Various Environments
Charoenpitaks, K., & Limpiyakorn, Y. (2019) · International Conference on Vision, Image and Signal Processing (ICVISP) · doi:10.1145/3387168.3387235 · Cited by 3
“[2] compared both types of exploration: count-based and prediction-based reward exploration.”
BackgroundMethodResultInfluentialVision Based Autonomous Tracking of UAVs Based on Reinforcement Learning
Xiong, G., & Dong, L. (2020) · Chinese Automation Congress · doi:10.1109/cac51589.2020.9326946 · Cited by 3
“It is used to train agents to do many unexpected things such as playing Go, autonomous driving and multi-agent competition [4].”
Backgroundsfc2cpu: Operating a Service Function Chain Platform with Neural Combinatorial Optimization
Krämer, P., Diederich, P., Kramer, C., Pries, R., Kellerer, W., & Blenk, A. (2021) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 3
Multi-Radar Tracking Optimization for Collaborative Combat
Nour, N., Belhaj-Soullami, R., Buron, C. L. R., Peres, A., & Barbaresco, F. (2021) · International Radar Symposium · doi:10.23919/irs51887.2021.9466227 · Cited by 3
“Recent successes in multi-agent RL were obtained by adapting mono-agent deep RL methods to the multiagent case, most of them based on policy gradient approaches [7] with a centralized learning and decentralized execution framework [8, 9, 10].”
MethodCoachGAN: Fast Adversarial Transfer Learning between Differently Shaped Entities
Mounsif, M., Lengagne, S., Thuilot, B., & Adouane, L. (2020) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0009972200890096 · Cited by 3
“In the RL paradigm, it is more common to train an agent on a given task and modify it afterward to see whether the agent’s previously learned representations can be repurposed (Trapit et al., 2017; Baker et al., 2019).”
BackgroundSalfinger, A. (2020) · IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) · doi:10.1109/cogsima49017.2020.9216026 · Cited by 3
“…was due to the rich environment the agents had been placed into, which was using a realistic physics engine (nonetheless, agents learned to exploit weaknesses in the physics engine, by learning box-surfing, a non-realistic behavior, representing another example towards reward hacking) [14].”
BackgroundInfluentialDisentangling causal effects for hierarchical reinforcement learningPreprint
Corcoll, O., & Vicente, R. (2020) · arXiv.org · doi:10.48550/arxiv.2010.01351 · Cited by 11
“Agents are provided with a discrete, entity-centric representation of the world as in Baker et al. (2019).”
BackgroundReinforcement Learning Agents for Ubisoft's Roller ChampionsPreprint
Iskander, N., Simoni, A., Alonso, E., & Peter, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06031 · Cited by 4
“Reinforcement Learning (RL) is an active topic of research that has received a lot of attention in the literature [1] [2] [3] [4].”
BackgroundOn the Regularity of AttentionPreprint
Vuckovic, J., Baratin, A., & Combes, R. T. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05628 · Cited by 15
“Recently, Baker et al. (2019) used attention in the policy architecture of a multi-agent reinforcement learning problem.”
BackgroundHow Biological Concepts and Evolutionary Theories Are Inspiring Advances in Machine IntelligencePreprint
Gutai, A. R., & Gorochowski, T. E. (2021) · Preprints.org · doi:10.20944/preprints202109.0234.v1 · Cited by 2
Curiosity-driven exploration: Diversity of mechanisms and functionsPreprint
Ten, A., Oudeyer, P.-Y., & Moulin-Frier, C. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/n2byt · Cited by 4
Dynamics of niche construction in adaptable populations evolving in diverse environments
Nisioti, E., & Moulin-Frier, C. (2023) · The 2023 Conference on Artificial Life · doi:10.1162/isal_a_00668 · Cited by 2
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
“In OpenAI’s hide-and-seek environment, simple competitive objectives gave rise to “exploits” such as tool-based ramp construction and box sheltering, illustrating how local strategies can evolve into unforeseen systemic vulnerabilities without external infiltration (Baker et al., 2019).”
Metadata
- Authors
- Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I.
- Affiliations
- OpenAI (United States) (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.07528
- OpenAlex
- W2973525135
- Semantic Scholar
- 33a2c0f3b9a0adc452a13d53f950c0c3c4abb11b
- Cited by
- 424
- References
- 70
- Citations by year
- 2026: 3 · 2025: 16 · 2024: 63 · 2023: 60 · 2022: 69 · 2021: 117 · 2020: 88 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- multi-agent competition, hide and seek, multi-agent coordination, intrinsic motivation, intelligence tests, environmental complexity, overcoming obstacles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
