Asynchronous Methods for Deep Reinforcement Learning
Open accessAbstract
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PILCO underestimates state uncertainty at future time steps, possibly leading to a decrease in performance.
“the former underestimates state uncertainty at future time steps [119], thus possibly leading to a decrease in performance”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [119]
The ensemble strategy combines three DRL algorithms (PPO, A2C, DDPG) to improve robustness.
“The ensemble strategy [51] combines three DRL algorithms (PPO [42], A2C [32] and DDPG [26]) to improve the robustness.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [32]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Self-citationADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · see citing works
Model-Free reinforcement learning with continuous action in practice
Degris, T., Pilarski, P. M., & Sutton, R. S. (2012) · 2012 American Control Conference (ACC) · doi:10.1109/acc.2012.6315022 · Cited by 239 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
Self-citationRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
“Here, we take the standard batched A2C [52] agent as base model and add CPC as an auxiliary loss.”
MethodCARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · see citing works
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · see citing works
Self-citationSigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Elfwing, S., Uchibe, E., & Doya, K. (2018) · Neural Networks · doi:10.1016/j.neunet.2017.12.012 · Cited by 2,119 · see citing works
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
Self-citationDeep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498 · see citing works
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · see citing works
Resource Management with Deep Reinforcement Learning
Mao, H., Alizadeh, M., Menache, I., & Kandula, S. (2016) · Proceedings of the 15th ACM Workshop on Hot Topics in Networks · doi:10.1145/3005745.3005750 · Cited by 1,353 · see citing works
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · see citing works
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861 · see citing works
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889 · see citing works
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712 · see citing works
Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., & Shah, A. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793742 · Cited by 694 · see citing works
“Many variants of actor-critic methods exist, see e.g. [28], [29].”
BackgroundLearning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · see citing works
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review
Cao, D., Hu, W., Zhao, J., Zhang, G., Zhang, B., Liu, Z., Chen, Z., & Blaabjerg, F. (2020) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2020.000552 · Cited by 460 · see citing works
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · see citing works
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393 · see citing works
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358 · see citing works
Shi, Z., Yao, W., Li, Z., Zeng, L., Zhao, Y., Zhang, R., Tang, Y., & Wen, J. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115733 · Cited by 327 · see citing works
Reinforcement learning for intelligent healthcare applications: A survey
Coronato, A., Naeem, M., De Pietro, G., & Paragliola, G. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101964 · Cited by 320 · see citing works
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Tan, H., Yu, L., & Bansal, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1268 · Cited by 293 · see citing works
Virtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326 · see citing works
“that of [18] where the actor network is a 4-layer convolutional network (shown in figure 3) with ReLU activation functions in-between.”
BackgroundMethodInfluentialLiu, B., Wang, L., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2931179 · Cited by 242 · see citing works
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264 · see citing works
Preparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · see citing works
Deep Reinforcement Learning for Trading
Zhang, Z., Zohren, S., & Roberts, S. (2020) · The Journal of Financial Data Science · doi:10.3905/jfds.2020.1.030 · Cited by 227 · see citing works
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00323 · Cited by 220 · see citing works
Decision Making in Multiagent Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2840971 · Cited by 219 · see citing works
Reinforcement learning for iot security: A comprehensive survey
Uprety, A., & Rawat, D. B. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3040957 · Cited by 214 · see citing works
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · see citing works
“[48] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P.”
BackgroundStabilizing Transformers for Reinforcement Learning
Parisotto, E., Song, H. F., Rae, J. W., Pascanu, R., Gulcehre, C., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.06764 · Cited by 199 · see citing works
Deep reinforcement learning for automated stock trading: an ensemble strategy
Yang, H., Liu, X.-Y., Zhong, S., & Walid, A. (2020) · Proceedings of the First ACM International Conference on AI in Finance · doi:10.1145/3383455.3422540 · Cited by 196 · see citing works
“A2C [32] is a typical actor-critic algorithm and we use it a component in the ensemble strategy.”
BackgroundMethodAhmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197 · see citing works
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Neural Map: Structured Memory for Deep Reinforcement Learning
Parisotto, E., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1702.08360 · Cited by 178 · see citing works
Survey on reinforcement learning for language processing
Uc-Cetina, V., Navarro-Guerrero, N., Martin-Gonzalez, A., Weber, C., & Wermter, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10205-5 · Cited by 177 · see citing works
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09359 · Cited by 175 · see citing works
“deling of the behavior policy, AWAC avoids overly conservative updates. 5Related Work Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [19, 5, 25, 11, 7, 4, 32, 48, 44, 3]. While standard off-policy methods are able to benefit from including data seen dur…”
BackgroundMethodInfluentialLoss of plasticity in deep continual learning
Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024) · Nature · doi:10.1038/s41586-024-07711-7 · Cited by 172 · see citing works
“Altogether, these results, along with other extensive results in Methods, constitute substantial evidence of plasticity loss.”
Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues
Zhu, H., Cao, Y., Wang, W., Jiang, T., & Jin, S. (2018) · IEEE Network · doi:10.1109/mnet.2018.1800109 · Cited by 153 · see citing works
PCGRL: Procedural Content Generation via Reinforcement Learning
Khalifa, A., Bontrager, P., Earle, S., & Togelius, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v16i1.7416 · Cited by 156 · see citing works
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140 · see citing works
Futures of artificial intelligence through technology readiness levels
Martínez-Plumed, F., Gómez, E., & Hernández-Orallo, J. (2021) · Telematics and Informatics · doi:10.1016/j.tele.2020.101525 · Cited by 130 · see citing works
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
Deep Reinforcement Adversarial Learning Against Botnet Evasion Attacks
Apruzzese, G., Andreolini, M., Marchetti, M., Venturi, A., & Colajanni, M. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3031843 · Cited by 117 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Nguyen, K., Daumé III, H., & Boyd-Graber, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1153 · Cited by 110 · see citing works
VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control
Zhang, J., Tai, L., Yun, P., Xiong, Y., Liu, M., Boedecker, J., & Burgard, W. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2894216 · Cited by 109 · see citing works
Maintaining cooperation in complex social dilemmas using deep reinforcement learning
Lerer, A., & Peysakhovich, A. (2017) · arXiv · doi:10.48550/arxiv.1707.01068 · Cited by 101 · see citing works
Toward Communication-Efficient Federated Learning in the Internet of Things With Edge Computing
Sun, H., Li, S., Yu, F. R., Qi, Q., Wang, J., & Liao, J. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2994596 · Cited by 95 · see citing works
MineRL: A Large-Scale Dataset of Minecraft Demonstrations
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/339 · Cited by 94 · see citing works
FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Liu, X.-Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3737257 · Cited by 90 · see citing works
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2107.09645 · Cited by 89 · see citing works
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
Active visual information gathering for vision-language navigation
Wang, H., Wang, W., Shu, T., Liang, W., & Shen, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58542-6_19 · Cited by 74 · see citing works
Hua, M., Qi, X., Chen, D., Jiang, K., Liu, Z. E., Sun, H., Zhou, Q., & Xu, H. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3574280 · Cited by 88 · see citing works
“…Q ( s , a ) to select actions via maximization; Policy-based methods (e.g., DDPG): optimize a parameterized policy π θ directly using gradient ascent; Actor-critic methods (e.g., Advantage Actor-Critic (A2C) [35]): combine policy and value function learning to improve stability and reduce variance.”
Learning to run a power network challenge for training topology controllers
Marot, A., Donnot, B., Romero, C., Donon, B., Lerousseau, M., Veyrin-Forrer, L., & Guyon, I. (2020) · Electric Power Systems Research · doi:10.1016/j.epsr.2020.106635 · Cited by 70 · see citing works
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation
Gordon, D., Kadian, A., Parikh, D., Hoffman, J., & Batra, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00111 · Cited by 57 · see citing works
MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores
Kao, S.-C., & Krishna, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00065 · Cited by 58 · see citing works
Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
de Prado, M., Rusci, M., Capotondi, A., Donze, R., Benini, L., & Pazos, N. (2021) · Sensors · doi:10.3390/s21041339 · Cited by 57 · see citing works
Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images
Ganapathi Subramanian, S., & Crowley, M. (2018) · Frontiers in ICT · doi:10.3389/fict.2018.00006 · Cited by 56 · see citing works
Deep Reinforcement Learning for Syntactic Error Repair in Student Programs
Gupta, R., Kanade, A., & Shevade, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301930 · Cited by 54 · see citing works
FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Liu, X.-Y., Yang, H., Chen, Q., Zhang, R., Yang, L., Xiao, B., & Wang, C. (2025) · doi:10.2139/ssrn.3737859 · Cited by 51 · see citing works
“In the middle is the agent layer that provides fine-tuned standard DRL algorithms (DQN [29][34], DDPG [29], Adaptive DDPG [27], Multi-Agent DDPG [30], PPO [40], SAC [18], A2C [33] and TD3 [11], etc.), commonly used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the r…”
InfluentialThe MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
Guss, W. H., Codel, C., Hofmann, K., Houghton, B., Kuno, N., Milani, S., Mohanty, S., Liebana, D. P., Salakhutdinov, R., Topin, N., et al. (2019) · arXiv · doi:10.48550/arxiv.1904.10079 · Cited by 46 · see citing works
Discovering cognitive strategies with tiny recurrent neural networks
Ji-An, L., Benna, M. K., & Mattar, M. G. (2025) · Nature · doi:10.1038/s41586-025-09142-4 · Cited by 64 · see citing works
FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance
Liu, X.-Y., Yang, H., Gao, J., & Wang, C. (2021) · doi:10.2139/ssrn.3955949 · Cited by 46 · see citing works
Minimalistic Attacks: How Little It Takes to Fool Deep Reinforcement Learning Policies
Qu, X., Sun, Z., Ong, Y.-S., Gupta, A., & Wei, P. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.2974509 · Cited by 40 · see citing works
Wang, H., Tan, A. H., & Nejat, G. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3412638 · Cited by 40 · see citing works
Algorithmic and Ethical Aspects of Recommender Systems in e-Commerce
Paraschakis, D. (2018) · doi:10.24834/2043/24268 · Cited by 35 · see citing works
Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks
M. Hafiz, A., Hassaballah, M., Alqahtani, A., Alsubai, S., & Abdel Hameed, M. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.031720 · Cited by 33 · see citing works
Safe Reinforcement Learning via Probabilistic Logic Shields
Yang, W.-C., Marra, G., Rens, G., & De Raedt, L. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/637 · Cited by 34 · see citing works
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning
McKee, K. R., Gemp, I., McWilliams, B., Duèñez-Guzmán, E. A., Hughes, E., & Leibo, J. Z. (2020) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/dnuf7773 · Cited by 33 · see citing works
Ghaffari, A., & Savaria, Y. (2020) · Electronics · doi:10.3390/electronics9122200 · Cited by 31 · see citing works
Cyber-physical defense in the quantum Era
Barbeau, M., & Garcia-Alfaro, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-05690-1 · Cited by 22 · see citing works
FinMem: A Performance-Enhanced LLM Trading Agent With Layered Memory and Character Design
Yu, Y., Li, H., Chen, Z., Jiang, Y., Li, Y., Suchow, J. W., Zhang, D., & Khashanah, K. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3593370 · Cited by 23 · see citing works
Learning Reciprocity in Complex Sequential Social Dilemmas
Eccles, T., Hughes, E., Kramár, J., Wheelwright, S., & Leibo, J. Z. (2019) · arXiv · doi:10.48550/arxiv.1903.08082 · Cited by 19 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
“Most previous work on RLHF uses on-policy RL algorithms such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) or Advantage Actor Critic (A2C) (Mnih et al., 2016), but offline RL methods have also been proposed (Snell et al., 2022).”
MethodSelf-Net: Lifelong Learning via Continual Self-Modeling
Mandivarapu, J. K., Camp, B., & Estrada, R. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00019 · Cited by 14 · see citing works
In-context Reinforcement Learning with Algorithm Distillation
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.14215 · Cited by 14 · see citing works
“In the Dark Room and Dark Key-to-Door environments we collect 2000 learning histories with an Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016) with 100 actors, while in DMLab Watermaze we collect 4000 learning histories with a distributed DQN with 16 parallel actors (see Appendix F for asymptotic learning…”
BackgroundMethodInfluentialSelf-citationDeep reinforcement learning for task-based feature learning in prosthetic vision
White, J., Kameneva, T., & McCarthy, C. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856541 · Cited by 14 · see citing works
Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“…bonus is commonly added to the training loss to encourage exploration by increasing the entropy of actions—a well-established practice in traditional tasks (Schulman et al., 2017; Williams, 1992; Mnih et al., 2016), and recently applied to LLM reasoning (Sheng et al., 2024; Hu et al., 2024).”
Nguyen, C. H., Shihua, J. M., & Liem, R. P. (2025) · Communications in Transportation Research · doi:10.1016/j.commtr.2025.100165 · Cited by 12 · see citing works
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02151 · Cited by 10 · see citing works
“Popular RL algorithms include Policy Gradient (Sutton et al., 2000), Proximal Policy Optimization Algorithm (PPO) (Schulman et al., 2017), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Advantage Actor Critic methods (Mnih et al., 2016).”
MethodZhao, W., Queralta, J. P., Qingqing, L., & Westerlund, T. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.10.043 · Cited by 9 · see citing works
Podracer architectures for scalable Reinforcement Learning
Hessel, M., Kroiss, M., Clark, A., Kemaev, I., Quan, J., Keck, T., Viola, F., & van Hasselt, H. (2021) · arXiv · doi:10.48550/arxiv.2104.06272 · Cited by 8 · see citing works
Mutual influence between language and perception in multi-agent communication games
Ohmer, X., Marino, M., Franke, M., & König, P. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010658 · Cited by 6 · see citing works
“We add an entropy regularization term [48] of 0.”
Learning Not to Learn: Nature versus Nurture in Silico
Lange, R. T., & Sprekeler, H. (2020) · arXiv · doi:10.48550/arxiv.2010.04466 · Cited by 6 · see citing works
Deconstructing Deep Active Inference: A Contrarian Information Gatherer
Champion, T., Grześ, M., Bonheme, L., & Bowman, H. (2024) · Neural Computation · doi:10.1162/neco_a_01697 · Cited by 5 · see citing works
Towards Practical Credit Assignment for Deep Reinforcement Learning
Alipov, V., Simmons-Edler, R., Putintsev, N., Kalinin, P., & Vetrov, D. (2021) · arXiv · doi:10.48550/arxiv.2106.04499 · Cited by 3 · see citing works
- Authors
- Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1602.01783
- OpenAlex
- W2260756217
- Cited by
- 7,132
- References
- 5
- Citations by year
- 2026: 18 · 2025: 375 · 2024: 795 · 2023: 914 · 2022: 926 · 2021: 1188 · 2020: 1108 · 2019: 981 · 2018: 545 · 2017: 238 · 2016: 39 · 2015: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- deep reinforcement learning, actor-critic, neural network controller, Atari, visual input, training stability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
