The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.
Asynchronous Methods for Deep Reinforcement Learning
Open accessAbstract
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the former underestimates state uncertainty at future time steps [119], thus possibly leading to a decrease in performance”
From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [119]
In the citing paper
“The ensemble strategy [51] combines three DRL algorithms (PPO [42], A2C [32] and DDPG [26]) to improve the robustness.”
From FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2021) · cites this paper as [32]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Self-citationADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Model-Free reinforcement learning with continuous action in practice
Degris, T., Pilarski, P. M., & Sutton, R. S. (2012) · 2012 American Control Conference (ACC) · doi:10.1109/acc.2012.6315022 · Cited by 239
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–120 of 120
Deep reinforcement learning for task-based feature learning in prosthetic vision
White, J., Kameneva, T., & McCarthy, C. (2019) · 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856541 · Cited by 14
Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“…bonus is commonly added to the training loss to encourage exploration by increasing the entropy of actions—a well-established practice in traditional tasks (Schulman et al., 2017; Williams, 1992; Mnih et al., 2016), and recently applied to LLM reasoning (Sheng et al., 2024; Hu et al., 2024).”
Nguyen, C. H., Shihua, J. M., & Liem, R. P. (2025) · Communications in Transportation Research · doi:10.1016/j.commtr.2025.100165 · Cited by 12
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02151 · Cited by 10
“Popular RL algorithms include Policy Gradient (Sutton et al., 2000), Proximal Policy Optimization Algorithm (PPO) (Schulman et al., 2017), Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and Advantage Actor Critic methods (Mnih et al., 2016).”
MethodZhao, W., Queralta, J. P., Qingqing, L., & Westerlund, T. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.10.043 · Cited by 9
Podracer architectures for scalable Reinforcement Learning
Hessel, M., Kroiss, M., Clark, A., Kemaev, I., Quan, J., Keck, T., Viola, F., & van Hasselt, H. (2021) · arXiv · doi:10.48550/arxiv.2104.06272 · Cited by 8
Mutual influence between language and perception in multi-agent communication games
Ohmer, X., Marino, M., Franke, M., & König, P. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010658 · Cited by 6
“We add an entropy regularization term [48] of 0.”
Learning Not to Learn: Nature versus Nurture in Silico
Lange, R. T., & Sprekeler, H. (2020) · arXiv · doi:10.48550/arxiv.2010.04466 · Cited by 6
Deconstructing Deep Active Inference: A Contrarian Information Gatherer
Champion, T., Grześ, M., Bonheme, L., & Bowman, H. (2024) · Neural Computation · doi:10.1162/neco_a_01697 · Cited by 5
Towards Practical Credit Assignment for Deep Reinforcement Learning
Alipov, V., Simmons-Edler, R., Putintsev, N., Kalinin, P., & Vetrov, D. (2021) · arXiv · doi:10.48550/arxiv.2106.04499 · Cited by 3
Li, G., Zhang, G., & Qin, C. (2021) · Sensors · doi:10.3390/s21186100 · Cited by 2
“Another notable Actor-Critic algorithm is A3C [24], which has multiple agents executing in parallel.”
BackgroundDizon-Paradis, O., Wormald, S., Capecci, D., Bhandarkar, A., & Woodard, D. (2023) · doi:10.36227/techrxiv.23739099 · Cited by 1
Qian, C., Yang, H., & Li, J. (2023) · 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc53992.2023.10393973 · Cited by 1
Orchestrating Joint Offloading and Scheduling for Low-Latency Edge SLAM
Zhang, Y., Mao, Y., Wang, H., Yu, Z., Guo, S., Zhang, J., Wang, L., & Guo, B. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3547256 · Cited by 1
Kuo, P.-C., Hou, H., Dabney, W., & Walker, E. Y. (2025) · arXiv
“The network is trained end-to-end to maximize the discounted cumulative reward with the Advantage Actor Critic algorithm [50] (Parts of the implementation code are based on [53], under MIT license).”
A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agents
Liu, A., & Borisyuk, A. (2024) · arXiv · doi:10.48550/arxiv.2407.03436
Performative Reinforcement Learning
Mandal, D., Triantafyllou, S., & Radanovic, G. (2022) · arXiv · doi:10.48550/arxiv.2207.00046
Kuo, P.-C., Hou, H., Dabney, W., & Walker, E. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-0945
NavFormer: A Transformer Architecture for Robot Target-Driven Navigation in Unknown and Dynamic Environments
Wang, H., Tan, A. H., & Nejat, G. (2024) · arXiv · Cited by 40
“In [8], the Asynchronous Advantage Actor Critic (A3C) [16] was used to train scene-specific FCLs to generate navigation actions using joint embedding.”
MethodTassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
Self-citation
Metadata
- Authors
- Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1602.01783
- OpenAlex
- W2260756217
- Cited by
- 7,132
- References
- 5
- Citations by year
- 2026: 18 · 2025: 375 · 2024: 795 · 2023: 914 · 2022: 926 · 2021: 1188 · 2020: 1108 · 2019: 981 · 2018: 545 · 2017: 238 · 2016: 39
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Keywords
- deep reinforcement learning, actor-critic, neural network controller, Atari, visual input, training stability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
