Deep Reinforcement Learning: A Brief Survey
Open accessThe DOI matches its Crossref record; 175 of the 182 references it cites were found in a registry or an index.
Abstract
Deep reinforcement learning (DRL) is poised to revolutionize the field of artificial intelligence (AI) and represents a step toward building autonomous systems with a higher-level understanding of the visual world. Currently, deep learning is enabling reinforcement learning (RL) to scale to problems that were previously intractable, such as learning to play video games directly from pixels. DRL algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of RL, then progress to the main streams of value-based and policy-based methods. Our survey will cover central algorithms in deep RL, including the deep Q-network (DQN), trust region policy optimization (TRPO), and asynchronous advantage actor critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via RL. To conclude, we describe several current areas of research within the field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Foundational surveys on RL [7,72] provide broad overviews of classical and deep RL, respectively, but do not focus on human feedback or LLMs.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [7]
In the citing paper
“Classical approaches employ reinforcement learning (RL) [6, 7] and imitation learning [8-10] that operate on primitive actions, which could be challenging for systematic exploration [11-15], interpretability [16-18], and generalization [19-21].”
From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [7]
In the citing paper
“This can be frequently modeled as decision making and learning in a Markov decision process (MDP) [160].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [160]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 2×
“The workhorse of DRL, however, remains backpropagation [162, 111].”
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 10×
“[34] train locally linear models for use with the NAF algorithm— the continuous equivalent of the DQN [71]—to improve the algorithm’s sample complexity in the robotic domain where”
InfluentialReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 7×
“(temporal) credit assignment problem [115].”
InfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · cited 1×
“[22] devised a method for calculating gradients to optimize stochastic policies by “reparameterizing” [30], [60] the stochasticity away from the network, thereby allowing standard gradients to be used (instead of the high-variance REINFORCE estimator [97]).”
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 7×
“dynamic programming over all possible actions (Figure 3, top left), sample trajectories for heuristic search (as was done by AlphaGo [108]), or even perform an exhaustive search (Figure 3, top right).”
InfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“technique known as distillation [40] has been utilised to both transfer policies learned by large DQNs to smaller DQNs, and transfer policies learned by several DQNs trained on separate games to one single DQN [85, 95].”
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2×
“By means of representation learning, DRL can deal efficiently with the curse of dimensionality, unlike tabular and traditional non-parametric methods [11].”
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · cited 6×
“Figures reproduced from [8, 67, 124, 63, 142, 141], respectively.”
InfluentialProximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 8×
“Note that RNNs are not restricted to value-function-based methods but have also been successfully applied to policy search [138] and actor-critic methods [36, 72].”
InfluentialTaking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049 · cited 1×
“Bayesian optimisation [106]; future work in DRL may benefit from investigating other successful techniques that are used in Bayesian optimisation.”
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 1×
“Many DRL papers in continuous control [103, 37, 67, 72, 4, 111] have used the MuJoCo physics engine to obtain relatively realistic dynamics for multi-joint continuous control problems [122], and there has now been some effort”
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · cited 2×
“Many DRL papers in continuous control [103, 37, 67, 72, 4, 111] have used the MuJoCo physics engine to obtain relatively realistic dynamics for multi-joint continuous control problems [122], and there has now been some effort”
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 1×
“A generalisation of MDPs are partially observable MDPs (POMDPs), in which the agent receives an observation ot ∈ Ω, where the distribution of the observation p(ot+1|st+1,at) is dependent on the current state and the previous action [45].”
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974 · cited 1×
“From designing state-of-the-art machine translation models [143] to constructing new optimisation functions [65], DRL has already”
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · cited 1×
“Q learning requires an additional function to be learned, later work proposed using the already available target network from the DQN algorithm, resulting in significantly better results with only a small change in the update step [127].”
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300 · cited 4×
“Rather than having to come up with accurate Qvalues for all actions, the duelling DQN [134] benefits from a single baseline for the state in the form of V π , and easier-tolearn relative values in the form of A .”
InfluentialDeterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233 · cited 2×
“much early work focused on simple, custom MDPs, there shortly emerged control problems that could be used as standard benchmarks for testing new algorithms, such as the Cartpole [5], Acrobot [22] and Mountain Car [74] domains.”
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120 · cited 2×
“The combination of the duelling DQN with prioritised experience replay [100] is one of the state-of-the-art techniques in discrete action settings.”
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1×
“Early work in this area introduced options, in which, apart from primitive actions (single-timestep actions), policies could also run other policies (multi-timestep “actions”) [116].”
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,789
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568 · cited 1×
“manner, such as deciding where to look in an image to track [102], classify [70] or caption objects [141].”
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 2,405
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 1×
“considers multiple agents learning through RL, and often the non-stationarity introduced by other agents changing their behaviours as they learn [14].”
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103 · cited 10×
“Whilst TRPO can be used as a pure policy gradient method with a simple baseline, later work by Schulman et al. [123] introduced generalised advantage estimation (GAE), which proposed several, more advanced variance reduction baselines.”
InfluentialLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · cited 2×
“From managing power consumption [120] to picking and stowing objects [64], DRL stands to increase the amount”
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · cited 2×
“Ho and Ermon [51] showed that policies are uniquely characterised by their occupancies (visited state and action distributions) allowing IRL to be reduced to the problem of measure matching.”
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · cited 1×
“model prediction error [112, 86] or maximising information gain [73, 42].”
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · cited 1×
“The goal of IRL is to estimate an unknown reward function from observed trajectories that characterise a desired solution [78]; IRL can be used in combination with RL to improve upon demonstrated behaviour.”
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 2×
“Indeed, one of the earliest success stories in RL is TD-Gammon, a neural network that reached expert-level performance in Backgammon in the early 90s [119].”
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“two techniques: experience replay [68] and target networks.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 6×
“(e) A real room, in which a wheeled robot trained to navigate the building is given a visual cue as input, and must find the corresponding location [142].”
InfluentialBuilding Watson: An Overview of the DeepQA Project
Ferrucci, D., Brown, E., Chu‐Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,497
On-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426 · cited 2×
“An alternative is to construct an autoregressive policy, where each action in a single timestep is predicted conditionally on the state and previously chosen actions from the same timestep [106, 5, 168].”
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344 · cited 4×
“Neural networks that encode policies have been successfully trained using both gradient-free [17], [33] and gradient-based [22], [41], [44], [70], [71], [96], [97] methods.”
InfluentialALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,338
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307
Campbell, M., Hoane, A. J., & Hsu, F.-H. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00129-1 · Cited by 1,226
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06676 · Cited by 1,217 · cited 1×
“Several approaches have been proposed for this purpose, including passing messages to agents sequentially [15], using a bidirectional channel (providing ordering with less signal loss) [58], and an all-to-all channel [77].”
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114 · cited 1×
“obtain relatively realistic dynamics for multi-joint continuous control problems [144], and there has now been some effort to standardise these problems [28].”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Salimans, T., Ho, J., Chen, X., Sidor, S., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1703.03864 · Cited by 1,027 · cited 2×
“perform hill-climbing in a subspace of policies [98], whilst more complex methods, such as compressed network search, impose additional inductive biases [53].”
On the theory of dynamic programming
Bellman, R. (1952) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.38.8.716 · Cited by 998
Auto-Encoding Variational Bayes
Chen, Y., Liu, J., Peng, L., Wu, Y., Xu, Y., & Zhang, Z. (2024) · Cambridge Explorations in Arts and Sciences · doi:10.61603/ceas.v2i1.33 · Cited by 934
Levine, S., & Koltun, V. (2013) · International Conference on Machine Learning · Cited by 906
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463189 · Cited by 848
van Hasselt, H. P. (2010) · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands · Cited by 839
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798 · cited 2×
“Typically, a parameterised policy πθ is chosen, whose parameters are updated to maximise the expected return E[R|θ] using either gradient-based or gradient-free optimisation [21].”
Self-citationKakade, S. M. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 775
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770 · cited 1×
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 764
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · cited 1×
“language processing [106, 5], we refer readers to the overview by Li [78].”
Riedmiller, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564096_32 · Cited by 757 · cited 2×
“A forerunner of the DQN—neural fitted Q iteration (NFQ)—involved training a neural network to return the Qvalue given a state-action pair [93].”
(2016) · neural information processing systems · Cited by 735
Konda, V. R., & Tsitsiklis, J. N. (2003) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012901385691 · Cited by 709
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 701
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · cited 1×
“A less common but potentially useful paradigm exists between model-free and model-based methods—the successor representation (SR) [24].”
The Option-Critic Architecture
Bacon, P.-L., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 687 · cited 1×
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Deep Recurrent Q-Learning for Partially Observable MDPs
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 667 · cited 4×
“Due to the range of supported games that vary in genre, visuals and difficulty, the ALE has become a standard testbed for DRL algorithms [71, 81, 35, 103, 112, 134, 72].”
InfluentialEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · cited 3×
“For simplicity, the underlying algorithm may be used with just one agent, termed advantage actor-critic (A2C) [156].”
InfluentialPolicy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564 · cited 1×
“Although RL had some successes in the past [119, 109, 51, 79], previous approaches lacked scalablity and were inherently limited to fairly low-dimensional problems.”
Action-conditional video prediction using deep networks in Atari games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · neural information processing systems · Cited by 561
Autonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer Tracts in Advanced Robotics · doi:10.1007/11552246_35 · Cited by 553
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · neural information processing systems · Cited by 542
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaskowski, W. (2016) · 2016 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 541 · cited 1×
“ViZDoom provides an interface to the Doom first-person shooter [49], and echoing the popularity of e-”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1611.02779 · Cited by 503
Tamar, A., Wu, Y., Thomas, G., Levine, S., & Abbeel, P. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/700 · Cited by 484 · cited 1×
“search [108] and planning [118], a deeper integration with other traditional AI approaches promises benefits such as better sample complexity, generalisation and interpretability [30].”
VIME: Variational Information Maximizing Exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 472
Relative Entropy Policy Search
Peters, J., Mulling, K., & Altun, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 455 · cited 1×
“The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
Deep Exploration via Bootstrapped DQN
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 434 · cited 1×
“[83] to propose the bootstrapped DQN, which”
Deep spatial autoencoders for visuomotor learning
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487173 · Cited by 430 · cited 2×
“Although deep neural networks can potentially produce very complex and rich models [81, 112, 27], sometimes simpler, more data-efficient methods are preferable [34].”
An Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423 · cited 2×
“An alternative is to construct an autoregressive policy, where each action in a single timestep is predicted conditionally on the state and previously chosen actions from the same timestep [106, 5, 168].”
A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Schmidhuber, J. (1991) · From Animals to Animats · doi:10.7551/mitpress/3115.003.0030 · Cited by 421 · cited 1×
“UCB can also be considered one way of implementing intrinsic motivation, which is a general concept that advocates decreasing uncertainty/making progress in learning about the environment [101].”
Action-Conditional Video Prediction using Deep Networks in Atari Games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.08750 · Cited by 419 · cited 3×
“Although deep neural networks can potentially produce very complex and rich models [81, 112, 27], sometimes simpler, more data-efficient methods are preferable [34].”
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · cited 2×
“This can be achieved in a naive fashion, directly using the same network in both the simulated and real phases [142], or with more sophisticated training procedures that directly try to mitigate the problem of neural networks “catastrophically forgetting” old knowledge by adding extra layers when transferring domain […”
PAC model-free reinforcement learning
Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143955 · Cited by 384
Likelihood ratio gradient estimation for stochastic systems
Glynn, P. W. (1990) · Communications of the ACM · doi:10.1145/84537.84552 · Cited by 381 · cited 1×
“FORCE rule [139], elsewhere known as the score function [29] or likelihood-ratio estimator [31].”
Massively Parallel Methods for Deep Reinforcement Learning
Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., De Maria, A., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.04296 · Cited by 379 · cited 1×
“[77] developed a framework for training multiple DQNs in parallel, achieving both better performance and a reduction in training time.”
Dynamic Programming and Suboptimal Control: A Survey from ADP to MPC*
Bertsekas, D. P. (2005) · European Journal of Control · doi:10.3166/ejc.11.310-334 · Cited by 366 · cited 1×
“This approach to solving POMDPs is related to other problems using dynamical systems and state space models, where the true state can only be estimated [12].”
Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
Singh, S., Litman, D., Kearns, M., & Walker, M. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.859 · Cited by 352 · cited 1×
“Although RL had some successes in the past [31], [53], [74], [81], previous approaches lacked scalability and were inherently limited to fairly low-dimensional problems.”
Learning Continuous Control Policies by Stochastic Value Gradients
Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., & Erez, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.09142 · Cited by 328 · cited 3×
“Many DRL papers in continuous control [103, 37, 67, 72, 4, 111] have used the MuJoCo physics engine to obtain relatively realistic dynamics for multi-joint continuous control problems [122], and there has now been some effort”
Learning to Navigate in Complex Environments
Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03673 · Cited by 323
Path integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309 · cited 1×
“The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
Continuous Deep Q-Learning with Model-based Acceleration
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 304 · cited 6×
“[34] train locally linear models for use with the NAF algorithm— the continuous equivalent of the DQN [71]—to improve the algorithm’s sample complexity in the robotic domain where”
InfluentialA Deep Hierarchical Approach to Lifelong Learning in Minecraft
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., & Mannor, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10744 · Cited by 299
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 296 · cited 1×
“Whilst easily achievable in small tabular cases, the use of powerful density models [11], or conversely, hashing [139], has allowed this algorithm to scale to high-dimensional visual domains with DRL.”
Deep Reinforcement Learning in Large Discrete Action Spaces
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., & Coppin, B. (2015) · arXiv · doi:10.48550/arxiv.1512.07679 · Cited by 290
Efficient memory-based learning for robot control
Moore, A. W. (2021) · doi:10.48456/tr-209 · Cited by 282
Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
Stadie, B. C., Levine, S., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.00814 · Cited by 280 · cited 4×
“Although deep neural networks can potentially produce very complex and rich models [81, 112, 27], sometimes simpler, more data-efficient methods are preferable [34].”
InfluentialMaximum Entropy Deep Inverse Reinforcement Learning
Wulfmeier, M., Ondrúška, P., & Posner, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.04888 · Cited by 272
Multi-agent Reinforcement Learning in Sequential Social Dilemmas
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03037 · Cited by 263 · cited 1×
“Other DRL works of note in MARL investigate the effects of learning and sequential decision making in game theory [38, 60].”
Learning Multiagent Communication with Backpropagation
Sukhbaatar, S., Szlam, A. D., & Fergus, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07736 · Cited by 261 · cited 1×
“to agents sequentially [28], using a bidirectional channel (providing ordering with less signal loss) [87], and an all-toall channel [114].”
Deep Reinforcement Learning: An Overview
Mousavi, S. S., Schukat, M., & Howley, E. (2018) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-319-56991-8_32 · Cited by 261
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
Mohamed, S., & Rezende, D. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.08731 · Cited by 258 · cited 1×
“model prediction error [112, 86] or maximising information gain [73, 42].”
Reinforcement Learning with Unsupervised Auxiliary Tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 247 · cited 3×
“For example, the “unsupervised reinforcement and auxiliary learning” A3Cbased agent is additionally trained with “pixel control” (maximally changing pixel inputs), plus reward prediction and value function learning from experience replay [43].”
Peng, P., Wen, Y., Yang, Y., Yuan, Q., Tang, Z., Long, H., & Wang, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10069 · Cited by 241
A Distributional Perspective on Reinforcement Learning
Bellemare, M. G., Dabney, W., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06887 · Cited by 237
FeUdal Networks for Hierarchical Reinforcement Learning
Vezhnevets, A., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01161 · Cited by 233 · cited 2×
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
Watter, M., Springenberg, J. T., Boedecker, J., & Riedmiller, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.07365 · Cited by 212 · cited 1×
“Based on the deep dynamical model [131], where high-dimensional observations are embedded into a lower-dimensional space using autoencoders, several model-based DRL algorithms have been proposed for learning models and policies from pixel information [81, 137, 132].”
Memory-based control with recurrent neural networks
Heess, N., Hunt, J. J., Lillicrap, T. P., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1512.04455 · Cited by 211 · cited 3×
“deep learning is to utilise recurrent neural networks (RNNs) [163, 44, 45, 85, 96], which, unlike feedforward neural networks, are dynamical systems.”
InfluentialThe Malmo platform for artificial intelligence experimentation
Johnson, M., Hofmann, K., Hutton, T. J., & Bignell, D. E. (2016) · International Joint Conference on Artificial Intelligence · Cited by 210
Imagination-Augmented Agents for Deep Reinforcement Learning
Weber, T., Racanière, S., Reichert, D., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06203 · Cited by 204 · cited 2×
“using representation learning to create distributed embeddings is a particular strength of DRL, and has been successfully utilised for other purposes [161, 100].”
Autonomous reinforcement learning on raw visual input data in a real world application
Lange, S., Riedmiller, M., & Voigtlander, A. (2012) · The 2012 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252823 · Cited by 202 · cited 5×
“NFQ was later extended to train a network to drive a slot car using raw visual inputs from a camera over the race track, by combining a deep autoencoder to reduce the dimensionality of the inputs with a separate branch to predict Q-values [69].”
InfluentialBridging the Gap Between Value and Policy Based Reinforcement Learning
Nachum, O., Norouzi, M., Xu, K., & Schuurmans, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08892 · Cited by 198 · cited 11×
“Meanwhile, the A3C-based agent of Mirowski et al. [82] was additionally trained to construct a depth map given RGB inputs, which helps it in its task of learning to navigate a 3D environment.”
InfluentialSample Efficient Actor-Critic with Experience Replay
Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01224 · Cited by 192 · cited 2×
“Both experience replay and target networks have gone on to be used in subsequent DRL works [34, 67, 135, 76].”
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2012) · arXiv · doi:10.48550/arxiv.1207.4708 · Cited by 187
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06342 · Cited by 186 · cited 1×
“technique known as distillation [40] has been utilised to both transfer policies learned by large DQNs to smaller DQNs, and transfer policies learned by several DQNs trained on separate games to one single DQN [85, 95].”
Neural Map: Structured Memory for Deep Reinforcement Learning
Parisotto, E., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1702.08360 · Cited by 178 · cited 1×
“More recent work, where the memory was given a 2D structure in order to resemble a spatial map, hints at future research where more specialised memory structures will be developed to address specific problems, such as 2D or 3D navigation [84].”
From Pixels to Torques: Policy Learning with Deep Dynamical ModelsPreprint
Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · arXiv (Cornell University) · doi:10.25561/21120 · Cited by 177
Self-citationQ-PrOP: Sample-efficient policy gradient with an off-policy critic
Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.17863/cam.21294 · Cited by 174
Distral: Robust Multitask Reinforcement Learning
Teh, Y. W., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R. T., Heess, N., & Pascanu, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04175 · Cited by 172 · cited 1×
“Together, the combination of multitask and transfer learning can improve the sample efficiency and robustness of current DRL algorithms [140].”
Equivalence Between Policy Gradients and Soft Q-Learning
Schulman, J., Chen, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06440 · Cited by 171 · cited 10×
“Meanwhile, the A3C-based agent of Mirowski et al. [82] was additionally trained to construct a depth map given RGB inputs, which helps it in its task of learning to navigate a 3D environment.”
InfluentialGradient Estimation Using Stochastic Computation Graphs
Schulman, J., Heess, N., Weber, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.05254 · Cited by 166 · cited 2×
““reparameterisation trick” [61, 108], allows neural networks to be treated as stochastic computation graphs that can be optimised over [121], which is a key concept in algorithms such as stochastic value gradients (SVGs) [46].”
BackgroundMultiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games.Preprint
Peng, P., Yuan, Q., Wen, Y., Yang, Y., Tang, Z., Long, H., & Wang, Y.-X. (2017) · arXiv (Cornell University) · Cited by 162
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
Kansky, K., Silver, T., Mély, D. A., Eldawy, M., Lázaro-Gredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., & George, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04317 · Cited by 161 · cited 2×
“Recent work with (non-deep) generative causal models demonstrated superior generalisation over standard DRL algorithms [72, 96] in some benchmarks [8], achieved by reasoning about causes and effects in the environment [47].”
Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching
Lin, L.-J. (1992) · Reinforcement Learning · doi:10.1007/978-1-4615-3618-5_5 · Cited by 160
Control of Memory, Active Perception, and Action in Minecraft
Oh, J., Chockalingam, V., Singh, S. P., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09128 · Cited by 157 · cited 3×
“A more common approach in deep learning is to utilise recurrent neural networks (RNNs) [138, 35, 36, 72, 82], which, unlike feedforward neural”
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
Christiano, P. F., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P., & Zaremba, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03518 · Cited by 148 · cited 1×
“(c) Utilising the potentially unlimited amount of training data that can be amassed in robotic simulators, several methods aim to transfer knowledge from the simulator to the real world [18, 97, 124].”
Towards Deep Symbolic Reinforcement Learning
Garnelo, M., Arulkumaran, K., & Shanahan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05518 · Cited by 146 · cited 1×
“search [108] and planning [118], a deeper integration with other traditional AI approaches promises benefits such as better sample complexity, generalisation and interpretability [30].”
Self-citationBagnell, J. A., & Schneider, J. (2018) · Figshare · doi:10.1184/r1/6552458 · Cited by 146
Evolving large-scale neural networks for vision-based reinforcement learning
Koutník, J., Cuccu, G., Schmidhuber, J., & Gomez, F. (2013) · Proceedings of the 15th annual conference on Genetic and evolutionary computation · doi:10.1145/2463372.2463509 · Cited by 130 · cited 5×
“eschewed the commonly used backpropagation algorithm in favour of evolutionary algorithms [32, 19, 53], which are gradient-free policy search algorithms.”
InfluentialThird-Person Imitation Learning
Stadie, B. C., Abbeel, P., & Sutskever, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01703 · Cited by 127 · cited 3×
“Many DRL papers in continuous control [103, 37, 67, 72, 4, 111] have used the MuJoCo physics engine to obtain relatively realistic dynamics for multi-joint continuous control problems [122], and there has now been some effort”
Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
Heinrich, J., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01121 · Cited by 125 · cited 1×
“Other DRL works of note in MARL investigate the effects of learning and sequential decision making in game theory [38, 60].”
Learning from Demonstrations for Real World Reinforcement LearningPreprint
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., et al. (2017) · arXiv (Cornell University) · Cited by 119
Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning
Tesauro, G. J., Das, R., Chan, H., Kephart, J. O., Levine, D., Rawson, F. L., & Lefurgy, C. R. (2007) · Cited by 118
Learning model-based planning from scratchPreprint
Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racanière, S., Reichert, D., Weber, T., Wierstra, D., & Battaglia, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06170 · Cited by 113 · cited 2×
“using representation learning to create distributed embeddings is a particular strength of DRL, and has been successfully utilised for other purposes [161, 100].”
BackgroundMethodRecurrent Environment Simulators
Chiappa, S., Racanière, S., Wierstra, D., & Mohamed, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02254 · Cited by 107 · cited 1×
“Although deep neural networks can make reasonable predictions in simulated environments over hundreds of timesteps [17], they typically require many samples to tune the large”
Wierstra, D., Forster, A., Peters, J., & Schmidhuber, J. (2010) · Logic Journal of IGPL · doi:10.1093/jigpal/jzp049 · Cited by 105 · cited 2×
“Note that RNNs are not restricted to value-function-based methods but have also been successfully applied to policy search [138] and actor-critic methods [36, 72].”
Deep Attention Recurrent Q-Network
Sorokin, I., Seleznev, A., Pavlov, M., Fedorov, A., & Ignateva, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.01693 · Cited by 94 · cited 4×
“However, the DQN outperformed the DRQN and DARQN on games requiring quick reactions, where Q-values can fluctuate more rapidly.”
InfluentialUsunier, N., Synnaeve, G., Lin, Z., & Chintala, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02993 · Cited by 91 · cited 1×
“[125] later proposed a similar method that performed”
Safe and Efficient Off-Policy Reinforcement Learning
Munos, R., Stepleton, T., Harutyunyan, A., & Bellemare, M. G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02647 · Cited by 91 · cited 1×
“which adds Retrace(λ) off-policy bias correction [75] to A3C, allowing it to use experience replay in order to improve sample complexity.”
Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., Schölkopf, B., & Levine, S. (2017) · Cambridge University Engineering Department Publications Database · doi:10.17863/cam.21291 · Cited by 89
Deep Successor Reinforcement Learning
Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02396 · Cited by 82 · cited 2×
“also an important area of ongoing research [117, 66, 152].”
TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
Synnaeve, G., Nardelli, N., Auvolat, A., Chintala, S., Lacroix, T., Lin, Z., Richoux, F., & Usunier, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00625 · Cited by 80
Strategic Attentive Writer for Learning Macro-Actions
-, A., Vezhnevets, Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04695 · Cited by 78 · cited 1×
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al. (2016) · arXiv · doi:10.48550/arxiv.1612.03801 · Cited by 68
LEARNING TO GENERATE ARTIFICIAL FOVEA TRAJECTORIES FOR TARGET DETECTION
Schmidhuber, J., & Huber, R. (1991) · International Journal of Neural Systems · doi:10.1142/s012906579100011x · Cited by 66 · cited 1×
“manner, such as deciding where to look in an image to track [102], classify [70] or caption objects [141].”
Towards Adapting Deep Visuomotor Representations from Simulated to Real Environments.Preprint
Tzeng, E., Devin, C., Hoffman, J., Finn, C., Peng, X., Levine, S., Saenko, K., & Darrell, T. (2015) · arXiv (Cornell University) · Cited by 66
Baird, I., & Leemon, C. (1993) · Defense Technical Information Center (DTIC) · doi:10.21236/ada280862 · Cited by 61
Discrete Sequential Prediction of Continuous Actions for Deep RL
Metz, L., Ibarz, J., Jaitly, N., & Davidson, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05035 · Cited by 60 · cited 1×
“[81] used this idea in order to construct the sequential DQN,”
Learning deep dynamical models from image pixels
Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2015.12.271 · Cited by 58
Self-citationLearning to Perform Physics Experiments via Deep Reinforcement Learning
Denil, M., Agrawal, P., Kulkarni, T. D., Erez, T., Battaglia, P., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01843 · Cited by 54 · cited 1×
“The availability of high-quality visual renderers and physics engines now enables us to take steps in this direction, with works that try to learn intuitive models of physics in visual environments [23].”
Combining policy gradient and Q-learning
O’Donoghue, B., Munos, R., Kavukcuoglu, K., & Mnih, V. (2017) · International Conference on Learning Representations · Cited by 52
Model-based Adversarial Imitation LearningPreprint
Baram, N., Anschel, O., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02179 · Cited by 46 · cited 2×
“[4] exploit gradient information that was not used in”
Recurrent Reinforcement Learning: A Hybrid Approach
Li, X., Li, L. C., Gao, J., He, X., Chen, J., Deng, L., & He, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.03044 · Cited by 44 · cited 2×
“A different form of transfer can be utilised to help RL in the form of multitask training [66, 43, 69].”
A Deeper Look at Planning as Learning from Replay
van Seijen, H., & Sutton, R. (2015) · International Conference on Machine Learning · Cited by 40
Analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Roy, B. V. (1996) · Cited by 32
Evolving Modular Fast-Weight Networks for Control
Gomez, F. J., & Schmidhuber, J. (2005) · Lecture notes in computer science · doi:10.1007/11550907_61 · Cited by 31 · cited 2×
“eschewed the commonly used backpropagation algorithm in favour of evolutionary algorithms [32, 19, 53], which are gradient-free policy search algorithms.”
The Reactor: A Sample-Efficient Actor-Critic Architecture
Gruslys, A., Azar, M. G., Bellemare, M. G., & Munos, R. (2017) · Cited by 31
Classifying Options for Deep Reinforcement LearningPreprint
Arulkumaran, K., Dilokthanakul, N., Shanahan, M., & Bharath, A. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.08153 · Cited by 27 · cited 1×
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Self-citationIntrinsically motivated neuroevolution for vision-based reinforcement learning
Cuccu, G., Luciw, M., Schmidhuber, J., & Gomez, F. J. (2011) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2011.6037324 · Cited by 24 · cited 2×
“eschewed the commonly used backpropagation algorithm in favour of evolutionary algorithms [32, 19, 53], which are gradient-free policy search algorithms.”
Deep Active Learning for Autonomous Navigation
Hussein, A. M., Gaber, M. M., & Elyan, E. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-44188-7_1 · Cited by 14
StarCraft II: A New Challenge for Reinforcement Learning
Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., et al. (2017) · arXiv · doi:10.48550/arxiv.1708.04782 · cited 2×
“Facebook’s TorchCraft [137] and DeepMind’s StarCraft II Learning Environment [153] respectively provide interfaces to the StarCraft and StarCraft II real-time strategy games, presenting challenges in both micromanagement and long-term planning.”
Pritzel, A., Uria, B., Srinivasan, S., Puigdomènech, A., Vinyals, O., Hassabis, D., Wierstra, D., & Blundell, C. (2017) · arXiv · doi:10.48550/arxiv.1703.01988 · cited 1×
“Alternatively, differentiable memories can be used as approximate hash tables, allowing DRL algorithms to store and retrieve successful experiences to facilitate rapid learning [105].”
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next
Cuomo, S., Di Cola, V. S., Giampaolo, F., Rozza, G., Raissi, M., & Piccialli, F. (2022) · Journal of Scientific Computing · doi:10.1007/s10915-022-01939-z · Cited by 2,784
“RL enables agents to conduct experiments to comprehend their environment better, allowing them to acquire high-level causal links and reasoning about causes and effects (Arulkumaran et al, 2017).”
BackgroundSarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00815-1 · Cited by 2,678
“Deep reinforcement learning (DRL or deep RL) [9] integrates neural networks with a reinforcement learning architecture to allow the agents to learn the appropriate actions in a virtual environment, as shown in Fig.”
BackgroundFederated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
“to [54]–[59] for comprehensive discussions of DNN architectures and training strategies.”
BackgroundApplications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,060
“discuss the applications of DRL such as [7] and [8], but they are specifically for computer vision and natural language processing.”
Metadata
- Authors
- Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A.
- Affiliations
- University of Oxford (GB)
- Institution of Engineering and Technology (GB)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 26-38
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W3100789280
- Cited by
- 4,733
- References
- 182
- Citations by year
- 2026: 444 · 2025: 855 · 2024: 806 · 2023: 749 · 2022: 590 · 2021: 511 · 2020: 388 · 2019: 260 · 2018: 122 · 2017: 6
- FWCI
- 126.58
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- deep reinforcement learning, value-based methods, deep Q-network, trust region policy optimization, asynchronous advantage actor-critic, visual understanding, autonomous systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.