Dota 2 with Large Scale Deep Reinforcement Learning
Open accessAbstract
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DRL algorithms have been successfully deployed in various types of simulation environments, yet their success beyond simulated worlds has been limited.
“Deep reinforcement learning (DRL) algorithms have been successfully deployed in various types of simulation environments, yet their success beyond simulated worlds has been limited.”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [20]
The Introduction (p. 1) and Related Work (p. 16) describe successful reinforcement-learning applications in games, supporting the simulation-environments portion of the claim. The paper does not say that success beyond simulated worlds has been limited; Environment Randomization (p. 62) instead notes a successful transfer from simulation to real-world robotics. (LLM, conf=0.84)
Experimental — accuracy not yet measuredSuccessful application of deep RL often requires large teams of experts with significant computational resources.
“As a result, successful application of deep RL often requires large teams of experts with significant computational resources (Berner et al., 2019; Schrittwieser et al., 2020; Ouyang et al., 2022).”
From TD-MPC2: Scalable, Robust World Models for Continuous Control (2023) · cites this paper as Berner et al., 2019
The Introduction (p. 1) says solving Dota 2 required thousands of GPUs over multiple months, and Author Contributions (p. 18) document a large team developing and training the system. These passages support OpenAI Five as an example, but do not establish that successful deep RL applications often require large expert teams and substantial compute. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Self-citationProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Self-citationPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · see citing works
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202 · see citing works
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 2× · see citing works
“For comparison Chess requires around one thousand values per observation (mostly 6-possibility categorical values) and Go around six thousand values (all binary)[12].”
BackgroundMethodGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · see citing works
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
Activity Analysis of Production and Allocation.
Carter, C. F., & Koopmans, T. C. (1952) · The Economic Journal · doi:10.2307/2226909 · Cited by 1,941 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,586 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
Dalvi, N. N., Domingos, P. M., Mausam, Sanghai, S. K., & Verma, D. (2004) · Knowledge Discovery and Data Mining · doi:10.1145/1014052.1014066 · Cited by 1,085 · see citing works
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner ArchitecturesPreprint
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01561 · Cited by 851 · see citing works
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. J. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 803 · see citing works
DeepStack: Expert-level artificial intelligence in heads-up no-limit poker
Moravčík, M., Schmid, M., BURCH, N. R., Lisý, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., & Bowling, M. (2017) · Science · doi:10.1126/science.aam6960 · Cited by 800 · see citing works
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662 · see citing works
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602 · see citing works
Self-citationAn Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories
Williams, R. J., & Peng, J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.4.490 · Cited by 593 · see citing works
Distributed Prioritized Experience ReplayPreprint
Horgan, D., Quan, J., Budden, D. M., Barth-Maron, G., Hessel, M., van Hasselt, H. P., & Silver, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00933 · Cited by 557 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Levine, S., & Koltun, V. (2013) · Cited by 481 · see citing works
An Empirical Model of Large-Batch TrainingPreprint
McCandlish, S., Kaplan, J., Amodei, D., & OpenAI Dota Team (2018) · arXiv.org · doi:10.48550/arxiv.1812.06162 · Cited by 464 · see citing works
Jain, A., Bansal, R., Kumar, A., & Singh, K. (2015) · International Journal of Applied and Basic Medical Research · doi:10.4103/2229-516x.157168 · Cited by 423 · see citing works
You, Y., Zhang, Z., Hsieh, C., Demmel, J. W., & Keutzer, K. (2018) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 393 · see citing works
Searching for solutions in games and artificial intelligence
Allis, L. V. (1994) · doi:10.26481/dis.19940923la · Cited by 350 · see citing works
Quantifying Generalization in Reinforcement LearningPreprint
Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02341 · Cited by 348 · see citing works
Self-citationEmergent Complexity via Multi-Agent CompetitionPreprint
Bansal, T., Pachocki, J., Sidor, S., Sutskever, I., & Mordatch, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03748 · Cited by 266 · see citing works
Self-citationDeep Reinforcement Learning from Self-Play in Imperfect-Information GamesPreprint
Heinrich, J., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01121 · Cited by 240 · see citing works
Scaling SGD Batch Size to 32K for ImageNet Training.Preprint
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · Cited by 215 · see citing works
Growing a Brain: Fine-Tuning by Increasing Model Capacity
Wang, Y.-X., Ramanan, D., & Hebert, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.323 · Cited by 154 · cited 1× · see citing works
“Past methods have grown neural networks by incrementally training and freezing parts of the network [49], [50], [51].”
MethodIntrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., & Fergus, R. (2017) · arXiv · doi:10.48550/arxiv.1703.05407 · Cited by 149 · see citing works
Open-ended Learning in Symmetric Zero-sum Games
Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W. M., Perolat, J., Jaderberg, M., & Graepel, T. (2019) · arXiv · doi:10.48550/arxiv.1901.08106 · Cited by 88 · see citing works
The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
Guss, W. H., Codel, C., Hofmann, K., Houghton, B., Kuno, N., Milani, S., Mohanty, S., Liebana, D. P., Salakhutdinov, R., Topin, N., et al. (2019) · arXiv · doi:10.48550/arxiv.1904.10079 · Cited by 46 · see citing works
Mix&Match - Agent Curricula for Reinforcement LearningPreprint
Czarnecki, W. M., Jayakumar, S. M., Jaderberg, M., Hasenclever, L., Teh, Y. W., Osindero, S., Heess, N., & Pascanu, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01780 · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- [15]15. Konda, V. R. & Tsitsiklis, J. N. Actor-critic algorithms in Advances in neural information processing systems (2000), 1008-1014.
- [16]16. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D. & Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning in International conference on machine learning (2016), 1928-1937.
- [17]17. Schulman, J., Moritz, P., Levine, S., Jordan, M. I. & Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. CoRR abs/1506.02438 (2016). · arXiv
- [23]23. Chen, T., Goodfellow, I. J. & Shlens, J. Net2Net: Accelerating Learning via Knowledge Transfer in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016). < http://arxiv.org/abs/ 1511.05641 >.
- [27]27. Herbrich, R., Minka, T. & Graepel, T. TrueSkill: a Bayesian skill rating system in Advances in neural information processing systems (2007), 569-576.
- [38]38. Anthony, T., Tian, Z. & Barber, D. Thinking fast and slow with deep learning and tree search in Advances in Neural Information Processing Systems (2017), 5360-5370.
- [41]41. Kulkarni, T. D., Narasimhan, K., Saeedi, A. & Tenenbaum, J. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation in Advances in neural information processing systems (2016), 3675-3683.
- [47]47. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D. & Kavukcuoglu, K. Asynchronous Methods for Deep Reinforcement Learning in Proceedings of The 33rd International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 48 (PMLR, New York, New York, USA, June 2016), 1928-1937. < http://proceedings.mlr.press/ v48/mniha16.html >.
- [49]49. Fahlman, S. E. & Lebiere, C. in (ed Touretzky, D. S.) 524-532 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1990). isbn : 1-55860-100-7. < http://dl.acm.org/ citation.cfm?id=109230.107380 >.
- [53]53. Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R. & Hadsell, R. Progressive Neural Networks. arXiv preprint arXiv:1606.04671 (2016). · arXiv
- [54]54. Hinton, G., Vinyals, O. & Dean, J. Distilling the Knowledge in a Neural Network in NIPS Deep Learning and Representation Learning Workshop (2015). < http://arxiv.org/abs/ 1503.02531 >.
- [59]59. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J. & Zaremba, W. OpenAI Gym. CoRR abs/1606.01540. arXiv: 1606.01540 . < http://arxiv.org/ abs/1606.01540 > (2016). · arXiv
Cited by
Sorted by citations
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,062 · see citing works
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works
Toward Causal Representation Learning
Scholkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3058954 · Cited by 1,170 · see citing works
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972 · see citing works
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957 · see citing works
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · see citing works
Champion-level drone racing using deep reinforcement learning
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., & Scaramuzza, D. (2023) · Nature · doi:10.1038/s41586-023-06419-4 · Cited by 739 · see citing works
Fusing Blockchain and AI With Metaverse: A Survey
Yang, Q., Zhao, Y., Huang, H., Xiong, Z., Kang, J., & Zheng, Z. (2022) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2022.3188249 · Cited by 576 · see citing works
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01294-x · Cited by 552 · see citing works
Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2021) · doi:10.2139/ssrn.3804983 · Cited by 552 · see citing works
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.10470 · Cited by 509 · see citing works
Machine learning and deep learning—A review for ecologists
Pichler, M., & Hartig, F. (2023) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14061 · Cited by 504 · see citing works
A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
Huang, S., & Ontañón, S. (2022) · The International FLAIRS Conference Proceedings · doi:10.32473/flairs.v35i.130584 · Cited by 435 · see citing works
Outracing champion Gran Turismo drivers with deep reinforcement learning
Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlić, A., Eckert, F., Fuchs, F., et al. (2022) · Nature · doi:10.1038/s41586-021-04357-7 · Cited by 372 · see citing works
Jukebox: A Generative Model for MusicPreprint
Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00341 · Cited by 602 · see citing works
Self-citationCompute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
Object-Centric Learning with Slot AttentionPreprint
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., & Kipf, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15055 · Cited by 399 · see citing works
Semantic memory: A review of methods, models, and current challenges
Kumar, A. A. (2020) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01792-x · Cited by 311 · see citing works
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · see citing works
A Survey of End-to-End Driving: Architectures and Training Methods
Tampuu, A., Matiisen, T., Semikin, M., Fishman, D., & Muhammad, N. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3043505 · Cited by 301 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Deep Multi-Agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
Chen, D., Hajidavalloo, M. R., Li, Z., Chen, K., Wang, Y., Jiang, L., & Wang, Y. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3285442 · Cited by 272 · see citing works
A Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 264 · see citing works
A Survey of Machine Learning for Computer Architecture and Systems
Wu, N., & Xie, Y. (2022) · ACM Computing Surveys · doi:10.1145/3494523 · Cited by 223 · see citing works
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks
Doan, K., Lao, Y., Zhao, W., & Li, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01175 · Cited by 211 · see citing works
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021) · RePEc: Research Papers in Economics · Cited by 201 · see citing works
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · see citing works
“[34] OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P.”
BackgroundQuantum agents in the Gym: a variational quantum algorithm for deep Q-learning
Skolik, A., Jerbi, S., & Dunjko, V. (2022) · Quantum · doi:10.22331/q-2022-05-24-720 · Cited by 201 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“…Learning (RL) has the potential to be used in a wide range of applications from autonomous vehicles (Filos et al., 2020) and algorithm control (Biedenkapp et al., 2020) to robotics (OpenAI et al., 2019a), but to fulfil this potential we need RL algorithms that can be used in the real world.”
BackgroundMethodResource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
Naderializadeh, N., Sydir, J. J., Simsek, M., & Nikopour, H. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3051163 · Cited by 191 · see citing works
Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., & Dresp-Langley, B. (2021) · Robotics · doi:10.3390/robotics10010022 · Cited by 188 · see citing works
Retzlaff, C. O., Das, S., Wayllace, C., Mousavi, P., Afshari, M. P., Yang, T., Saranti, A., Angerschmid, A., Taylor, M. E., & Holzinger, A. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15348 · Cited by 194 · see citing works
A survey on multi-agent reinforcement learning and its application
Ning, Z., & Xie, L.-H. (2024) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2024.02.003 · Cited by 257 · see citing works
Wang, X., Wang, Y., Su, X., Wang, L., Lü, C., Peng, H., & Liu, J. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10620-2 · Cited by 177 · see citing works
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174 · see citing works
Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning
Ilahi, I., Usama, M., Qadir, J., Janjua, M. U., Al-Fuqaha, A., Hoang, D. T., & Niyato, D. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3111139 · Cited by 170 · see citing works
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169 · see citing works
Loss of plasticity in deep continual learning
Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024) · Nature · doi:10.1038/s41586-024-07711-7 · Cited by 172 · see citing works
Robust Deep Reinforcement Learning against Adversarial Perturbations on State ObservationsPreprint
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D. S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08938 · Cited by 164 · see citing works
Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks
Ding, J., Yu, Z., Tian, Y., & Huang, T. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/321 · Cited by 157 · see citing works
A Survey of Deep Reinforcement Learning in Video GamesPreprint
Shao, K., Tang, Z., Zhu, Y., Li, N., & Zhao, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10944 · Cited by 224 · see citing works
Yang, J., Zhang, J., & Wang, H. H. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3023788 · Cited by 157 · see citing works
Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy
Kalinin, S. V., Ziatdinov, M. A., Hinkle, J. D., Jesse, S., Ghosh, A., Kelley, K. P., Lupini, A. R., Sumpter, B. G., & Vasudevan, R. K. (2021) · ACS Nano · doi:10.1021/acsnano.1c02104 · Cited by 151 · see citing works
Transferring policy of deep reinforcement learning from simulation to reality for robotics
Ju, H., Juan, R., Gomez, R., Nakamura, K., & Li, G. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00573-6 · Cited by 147 · see citing works
Motion Planning for Mobile Robots—Focusing on Deep Reinforcement Learning: A Systematic Review
Sun, H., Zhang, W., Yu, R., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3076530 · Cited by 143 · see citing works
Alexandropoulos, G. C., Stylianopoulos, K., Huang, C., Yuen, C., Bennis, M., & Debbah, M. A. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3174030 · Cited by 143 · see citing works
Deep Reinforcement Learning at the Edge of the Statistical Precipice
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. G. (2021) · arXiv · doi:10.48550/arxiv.2108.13264 · Cited by 142 · see citing works
Reinforcement Learning with Quantum Variational Circuit
Lockwood, O., & Si, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v16i1.7437 · Cited by 140 · see citing works
The Modern Mathematics of Deep Learning
Berner, J., Grohs, P., Gitta Kutyniok, B., & Petersen, P. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009025096.002 · Cited by 144 · see citing works
Knowledge-integrated machine learning for materials: lessons from gameplaying and robotics
Hippalgaonkar, K., Li, Q., Wang, X., Fisher, J. S., Kirkpatrick, J., & Buonassisi, T. (2023) · Nature Reviews Materials · doi:10.1038/s41578-022-00513-1 · Cited by 145 · see citing works
Ten questions concerning reinforcement learning for building energy management
Nagy, Z., Henze, G. P., Dey, S., Arroyo, J., Helsen, L., Zhang, X., Chen, B., Amasyali, K., Kurte, K. R., Zamzam, A. S., et al. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110435 · Cited by 137 · see citing works
Multi-agent hierarchical policy gradient for Air Combat Tactics emergence via self-play
Sun, Z., Piao, H., Yang, Z. C., Zhao, Y., Zhan, G., Zhou, D., Meng, G., Chen, H., Chen, X., Qu, B., et al. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.104112 · Cited by 132 · see citing works
Action Space Shaping in Deep Reinforcement Learning
Kanervisto, A., Scheller, C., & Hautamäki, V. (2020) · IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231687 · Cited by 123 · see citing works
Training Question Answering Models From Synthetic Data
Puri, R., Spring, R., Shoeybi, M., Patwary, M. A., & Catanzaro, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.468 · Cited by 137 · see citing works
Decoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123 · see citing works
Reinforcement learning for ridesharing: An extended survey
Qin, Z. T., Hongtu Zhu, B., & Ye, J. (2022) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2022.103852 · Cited by 124 · see citing works
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent GamesPreprint
Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A. M., & Wu, Y. (2021) · arXiv (Cornell University) · Cited by 121 · see citing works
Wang, Y., Zheng, P., Peng, T., Yang, H., & Zou, J. T. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1581-2 · Cited by 108 · see citing works
Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 108 · see citing works
A review of research on reinforcement learning algorithms for multi-agents
Hu, K., Li, M., Song, Z., Xu, K., Xia, Q., Sun, N., Zhou, P., & Xia, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128068 · Cited by 113 · see citing works
Combating emerging financial risks in the big data era: A perspective review
Cheng, X., Liu, S., Sun, X., Wang, Z., Zhou, H., Shao, Y., & Shen, H. (2021) · Fundamental Research · doi:10.1016/j.fmre.2021.08.017 · Cited by 103 · see citing works
Data-Efficient Reinforcement Learning with Self-Predictive RepresentationsPreprint
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A. C., & Bachman, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05929 · Cited by 103 · see citing works
RODE: Learning Roles to Decompose Multi-Agent TasksPreprint
Wang, T., Gupta, T., Mahajan, A., Peng, B., Whiteson, S., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01523 · Cited by 103 · see citing works
On the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102 · see citing works
PowerNet: Multi-Agent Deep Reinforcement Learning for Scalable Powergrid Control
Chen, D., Chen, K., Li, Z., Chu, T., Yao, R., Qiu, F., & Lin, K. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3100898 · Cited by 100 · see citing works
Transformer in reinforcement learning for decision-making: a survey
Yuan, W., Chen, J., Chen, S., Lu, L., Hu, Z., Li, P., Feng, D., Liu, F., & Chen, J. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300548 · Cited by 112 · see citing works
Defining the Scope of AI Regulations
Schuett, J. (2019) · Law Innovation and Technology · doi:10.1080/17579961.2023.2184135 · Cited by 133 · see citing works
Humans perceive warmth and competence in artificial intelligence
McKee, K. R., Bai, X., & Fiske, S. T. (2023) · iScience · doi:10.1016/j.isci.2023.107256 · Cited by 97 · see citing works
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13596 · Cited by 98 · see citing works
Open Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93 · see citing works
Measuring the Algorithmic Efficiency of Neural Networks
Hernandez, D., & Brown, T. B. (2020) · arXiv · doi:10.48550/arxiv.2005.04305 · Cited by 89 · see citing works
Suphx: Mastering Mahjong with Deep Reinforcement LearningPreprint
Li, J., Koyamada, S., Ye, Q., Liu, G., Wang, C., Yang, R., Li, Z., Qin, T., Liu, T., & Hon, H.-W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13590 · Cited by 88 · see citing works
Inferring neural activity before plasticity as a foundation for learning beyond backpropagation
Song, Y., Millidge, B., Salvatori, T., Lukasiewicz, T., Xu, Z., & Bogacz, R. (2024) · Nature Neuroscience · doi:10.1038/s41593-023-01514-1 · Cited by 85 · see citing works
Brax -- A Differentiable Physics Engine for Large Scale Rigid Body SimulationPreprint
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., & Bachem, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13281 · Cited by 127 · see citing works
Beyond Peak Performance: Comparing the Real Performance of AI-Optimized FPGAs and GPUs
Boutros, A., Nurvitadhi, E., Ma, R., Gribok, S., Zhao, Z., Hoe, J. C., Betz, V., & Langhammer, M. (2020) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt51103.2020.00011 · Cited by 78 · see citing works
Caldwell, S., Sweetser, P., O’Donnell, N., Knight, M. J., Aitchison, M., Gedeon, T., Johnson, D. M., Brereton, M. F., Gallagher, M. R., & Conroy, D. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514257 · Cited by 82 · see citing works
Wang, H., Liu, C. H., Yang, H., Wang, G., & Leung, K. K. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2023.3289172 · Cited by 75 · see citing works
Target localization using multi-agent deep reinforcement learning with proximal policy optimization
Alagha, A., Singh, S., Mizouni, R., Bentahar, J., & Otrok, H. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.06.015 · Cited by 72 · see citing works
Mathematical Aspects of Deep Learning
Berner, J., Grohs, P., Gitta Kutyniok, B., & Petersen, P. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009025096 · Cited by 73 · see citing works
Masked Contrastive Representation Learning for Reinforcement Learning
Zhu, J., Xia, Y., Wu, L., Deng, J., Zhou, W., Qin, T., Liu, T., & Li, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3176413 · Cited by 72 · see citing works
Xiang, X., & Foo, S. Y. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030029 · Cited by 72 · see citing works
Towards a generalized theory comprising digital, neuromorphic and unconventional computing
Jaeger, H. (2021) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/abf151 · Cited by 67 · see citing works
Finding Effective Security Strategies through Reinforcement Learning and Self-Play
Hammar, K., & Stadler, R. (2020) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm50824.2020.9269092 · Cited by 70 · see citing works
GAIL-PT: An intelligent penetration testing framework with generative adversarial imitation learning
Chen, J., Hu, S., Zheng, H., Xing, C., & Zhang, G. (2022) · Computers & Security · doi:10.1016/j.cose.2022.103055 · Cited by 65 · see citing works
Deep reinforcement learning challenges and opportunities for urban water systems
Negm, A. S., Ma, X., & Aggidis, G. (2024) · Water Research · doi:10.1016/j.watres.2024.121145 · Cited by 65 · see citing works
A Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
A Review of Uncertainty for Deep Reinforcement Learning
Lockwood, O., & Si, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v18i1.21959 · Cited by 63 · see citing works
Wang, W., Zhang, Y., Gao, J., Jiang, Y., Yang, Y., Zheng, Z., Zou, W., Li, J., Zhang, C., Cao, W., et al. (2023) · Communications in Transportation Research · doi:10.1016/j.commtr.2023.100096 · Cited by 63 · see citing works
Queueing Network Controls via Deep Reinforcement Learning
Dai, J. G., & Gluzman, M. O. (2022) · Stochastic Systems · doi:10.1287/stsy.2021.0081 · Cited by 61 · see citing works
A Survey of Domain-Specific Architectures for Reinforcement Learning
Rothmann, M., & Porrmann, M. (2022) · IEEE Access · doi:10.1109/access.2022.3146518 · Cited by 63 · see citing works
Coordination among neural modules through a shared global workspace
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., & Bengio, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.01197 · Cited by 59 · see citing works
A self‐adaptive SAC‐PID control approach based on reinforcement learning for mobile robots
Yu, X., Fan, Y., Xu, S., & Ou, L. (2021) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.5662 · Cited by 61 · see citing works
Xiao, J., Pisutsin, P., & Feroskhan, M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3331370 · Cited by 58 · see citing works
Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality
Vasudevan, R. K., Ziatdinov, M. A., Vlček, L., & Kalinin, S. V. (2021) · npj Computational Materials · doi:10.1038/s41524-020-00487-0 · Cited by 57 · see citing works
Visual Navigation in Real-World Indoor Environments Using End-to-End Deep Reinforcement Learning
Kulhánek, J., Derner, E., & Babuška, R. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3068106 · Cited by 58 · see citing works
Open-Ended Learning Leads to Generally Capable Agents
Open Ended Learning Team, Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.12808 · Cited by 56 · see citing works
Exploring optimal control of epidemic spread using reinforcement learning
Ohi, A. Q., Mridha, M. F., Monowar, M. M., & Hamid, M. A. (2020) · Scientific Reports · doi:10.1038/s41598-020-79147-8 · Cited by 55 · see citing works
Improving Teamwork Competencies in Human-Machine Teams: Perspectives From Team Science
Stowers, K. L., Brady, L. L., MacLellan, C. J., Wohleber, R. W., & Eduardo Salas, A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.590290 · Cited by 57 · see citing works
Offline Pre-trained Multi-agent Decision Transformer
Meng, L., Wen, M., Le, C., Li, X., Xing, D., Zhang, W., Wen, Y., Zhang, H., Wang, J., Yang, Y., et al. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1383-7 · Cited by 54 · see citing works
Deep Reinforcement Learning for One-Warehouse Multi-Retailer inventory management
Kaynov, I., van Knippenberg, M., Menkovski, V., van Breemen, A., & van Jaarsveld, W. L. (2023) · International Journal of Production Economics · doi:10.1016/j.ijpe.2023.109088 · Cited by 57 · see citing works
- Authors
- OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., Pinto, H. P. D. O., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., & Zhang, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1912.06680
- OpenAlex
- W2996037775
- Cited by
- 1,065
- References
- 37
- Citations by year
- 2026: 6 · 2025: 66 · 2024: 178 · 2023: 186 · 2022: 197 · 2021: 273 · 2020: 152 · 2019: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- Dota 2, self-play, distributed training, continuous training, imperfect information games, long time horizons, continuous state and action spaces, esports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
