CoChat
Sign Up
PreprintCited by 4,733Reinforcement Learning in RoboticsArtificial Intelligence in GamesDomain Adaptation and Few-Shot Learning

A Brief Survey of Deep Reinforcement Learning

Open access
Arulkumaran, Kai, Deisenroth, Marc Peter, Brundage, Miles, Bharath, Anil Anthony·2017·arXiv
DOI 10.48550/arxiv.1708.05866arXiv 1708.05866
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep reinforcement learning is poised to revolutionise the field of AI and represents a step towards building autonomous systems with a higher level understanding of the visual world. Currently, deep learning is enabling reinforcement learning to scale to problems that were previously intractable, such as learning to play video games directly from pixels. Deep reinforcement learning algorithms are also applied to robotics, allowing control policies for robots to be learned directly from camera inputs in the real world. In this survey, we begin with an introduction to the general field of reinforcement learning, then progress to the main streams of value-based and policy-based methods. Our survey will cover central algorithms in deep reinforcement learning, including the deep $Q$-network, trust region policy optimisation, and asynchronous advantage actor-critic. In parallel, we highlight the unique advantages of deep neural networks, focusing on visual understanding via reinforcement learning. To conclude, we describe several current areas of research within the field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1708.05866

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some of the most common applications for DRL in robotics are navigation and dexterous manipulation.

    “Some of the most common applications for DRL in robotics are navigation and dexterous manipulation [1], [76].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [1]

    The Introduction (pp. 1–2) gives robotic navigation and manipulation examples, including a wheeled robot navigating a building and a robot screwing on a bottle cap. It does not establish that these are among the most common DRL applications in robotics. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • RL algorithms have been increasingly adopted by the robotics community over the past years to control complex robots or multi-robot systems, or provide end-to-end policies from perception to control.

    “Reinforcement learning (RL) algorithms have been increasingly adopted by the robotics community over the past years to control complex robots or multi-robot systems [1], [2], or provide end-to-end policies from perception to control [3].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [1]

  • DRL algorithms have been successfully deployed in various types of simulation environments, yet their success beyond simulated worlds has been limited.

    “Deep reinforcement learning (DRL) algorithms have been successfully deployed in various types of simulation environments, yet their success beyond simulated worlds has been limited.”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [1]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 146

  1. [1][1] Pieter Abbeel and John Schulman. Deep Reinforcement Learning through Policy Optimization, 2016. Tutorial at NIPS 2016. -
  2. [2][2] Kai Arulkumaran, Nat Dilokthanakul, Murray Shanahan, and Anil Anthony Bharath. Classifying Options for Deep Reinforcement Learning. In IJCAI Workshop on Deep Reinforcement Learning: Frontiers and Challenges , 2016. -
  3. [3][3] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. In AAAI , 2017. -
  4. [4][4] J Andrew Bagnell and Jeff Schneider. Covariant Policy Search. In IJCAI , 2003. -
  5. [5][5] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An ActorCritic Algorithm for Sequence Prediction. In ICLR , 2017. -
  6. [7][7] Nir Baram, Oron Anschel, and Shie Mannor. Model-Based Adversarial Imitation Learning. In NIPS Workshop on Deep Reinforcement Learning , 2016. -
  7. [8][8] Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems. IEEE Trans. on Systems, Man, and Cybernetics , (5):834846, 1983. -
  8. [9][9] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨ uttler, Andrew Lefrancq, Simon Green, V´ ıctor Vald´ es, Amir Sadik, et al. DeepMind Lab. arXiv:1612.03801 , 2016. - · arXiv
  9. [10][10] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. In IJCAI , 2015. -
  10. [11][11] Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and R´ emi Munos. Unifying Count-Based Exploration and Intrinsic Motivation. In NIPS , 2016. -
  11. [12][12] Marc G Bellemare, Will Dabney, and R´ emi Munos. A Distributional Perspective on Reinforcement Learning. In ICML , 2017. -
  12. [13][13] Richard Bellman. On the Theory of Dynamic Programming. PNAS , 38(8):716-719, 1952. -
  13. [14][14] Yoshua Bengio, J´ erˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum Learning. In ICML , 2009. -
  14. [15][15] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Trans. on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013. -
  15. [16][16] Dimitri P Bertsekas. Dynamic Programming and Suboptimal Control: A Survey from ADP to MPC. European Journal of Control , 11(4-5): 310-334, 2005. -
  16. [17][17] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv:1606.01540 , 2016. - · arXiv
  17. [18][18] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A Comprehensive survey of Multiagent Reinforcement Learning. IEEE Trans. on Systems, Man, And Cybernetics , 2008. -
  18. [19][19] Murray Campbell, A Joseph Hoane, and Feng-hsiung Hsu. Deep Blue. Artificial Intelligence , 134(1-2):57-83, 2002. -
  19. [20][20] Rich Caruana. Multitask Learning. Machine Learning , 28(1):41-75, 1997. -
  20. [21][21] Silvia Chiappa, S´ ebastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent Environment Simulators. In ICLR , 2017. -
  21. [22][22] Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model. arXiv:1610.03518 , 2016. - · arXiv
  22. [24][24] Peter Dayan. Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Computation , 5(4): 613-624, 1993. -
  23. [25][25] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large Scale Distributed Deep Networks. In NIPS , 2012. -
  24. [26][26] Marc P Deisenroth, Gerhard Neumann, and Jan Peters. A Survey on Policy Search for Robotics. Foundations and Trends R © in Robotics , 2 (1-2), 2013. -
  25. [27][27] Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, and Nando de Freitas. Learning to Perform Physics Experiments via Deep Reinforcement Learning. In ICLR , 2017. -
  26. [28][28] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. In ICML , 2016. -
  27. [29][29] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast Reinforcement Learning via Slow Reinforcement Learning. In NIPS Workshop on Deep Reinforcement Learning , 2016. -
  28. [30][30] Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep Reinforcement Learning in Large Discrete Action Spaces. arXiv:1512.07679 , 2015. - · arXiv
  29. [31][31] David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. Building Watson: An Overview of the DeepQA Project. AI Magazine , 31(3):59-79, 2010. -
  30. [32][32] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep Spatial Autoencoders for Visuomotor Learning. In ICRA , 2016. -
  31. [33][33] Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In NIPS , 2016. -
  32. [35][35] Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards Deep Symbolic Reinforcement Learning. In NIPS Workshop on Deep Reinforcement Learning , 2016. -
  33. [36][36] Peter W Glynn. Likelihood Ratio Gradient Estimation for Stochastic Systems. Communications of the ACM , 33(10):75-84, 1990. -
  34. [38][38] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NIPS , 2014. -
  35. [39][39] Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G Bellemare, and R´ emi Munos. The Reactor: A Sample-Efficient Actor-Critic Architecture. arXiv:1704.04651 , 2017. - · arXiv
  36. [40][40] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous Deep Q-Learning with Model-Based Acceleration. In ICLR , 2016. -
  37. [41][41] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-Prop: Sample-Efficient Policy Gradient with an Off-Policy Critic. In ICLR , 2017. -
  38. [42][42] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, Bernhard Sch¨ olkopf, and Sergey Levine. Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning. In NIPS , 2017. -
  39. [44][44] Matthew Hausknecht and Peter Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAI Fall Symposium Series , 2015. -
  40. [45][45] Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-Based Control with Recurrent Neural Networks. In NIPS Workshop on Deep Reinforcement Learning , 2015. -
  41. [46][46] Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning Continuous Control Policies by Stochastic Value Gradients. In NIPS , 2015. -
  42. [47][47] Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286 , 2017. - · arXiv
  43. [49][49] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, et al. Learning from Demonstrations for Real World Reinforcement Learning. arXiv:1704.03732 , 2017. - · arXiv
  44. [51][51] Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In NIPS , 2016. -
  45. [52][52] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. In NIPS , 2016. -
  46. [54][54] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement Learning with Unsupervised Auxiliary Tasks. In ICLR , 2017. -
  47. [55][55] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The Malmo Platform for Artificial Intelligence Experimentation. In IJCAI , 2016. -
  48. [56][56] Leslie P Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence , 101(1):99-134, 1998. -
  49. [57][57] Sham M Kakade. A Natural Policy Gradient. In NIPS , 2002. -
  50. [58][58] Ken Kansky, Tom Silver, David A M´ ely, Mohamed Eldawy, Miguel L´ azaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema Networks: Zero-Shot Transfer with a Generative Causal Model of Intuitive Physics. In ICML , 2017. -
  51. [59][59] Hilbert J Kappen. Path Integrals and Symmetry Breaking for Optimal Control Theory. JSTAT , 2005(11):P11011, 2005. -
  52. [61][61] Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR , 2014. -
  53. [62][62] Nate Kohl and Peter Stone. Policy Gradient Reinforcement Learning for Fast Quadrupedal Locomotion. In ICRA , volume 3, 2004. -
  54. [63][63] Vijay R Konda and John N Tsitsiklis. On Actor-Critic Algorithms. SICON , 42(4):1143-1166, 2003. -
  55. [65][65] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. In NIPS , 2016. -
  56. [66][66] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep Successor Reinforcement Learning. In NIPS Workshop on Deep Reinforcement Learning , 2016. -
  57. [67][67] Tze Leung Lai and Herbert Robbins. Asymptotically Efficient Adaptive Allocation Rules. Advances in Applied Mathematics , 6(1):4-22, 1985. -
  58. [70][70] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Learning. Nature , 521(7553):436-444, 2015. -
  59. [72][72] Sergey Levine and Pieter Abbeel. Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics. In NIPS , 2014. -
  60. [73][73] Sergey Levine and Vladlen Koltun. Guided Policy Search. In ICLR , 2013. -
  61. [74][74] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. Endto-End Training of Deep Visuomotor Policies. JMLR , 17(39):1-40, 2016. -
  62. [77][77] Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent Reinforcement Learning: A Hybrid Approach. arXiv:1509.03044 , 2015. - · arXiv
  63. [78][78] Yuxi Li. Deep Reinforcement Learning: An Overview. arXiv:1701.07274 , 2017. - · arXiv
  64. [79][79] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. In ICLR , 2016. -
  65. [80][80] Long-Ji Lin. Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Machine Learning , 8(3-4):293-321, 1992. -
  66. [81][81] Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete Sequential Prediction of Continuous Actions for Deep RL. arXiv:1705.05035 , 2017. - · arXiv
  67. [82][82] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray - Kavukcuoglu, et al. Learning to Navigate in Complex Environments. In ICLR , 2017. -
  68. [83][83] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent Models of Visual Attention. In NIPS , 2014. -
  69. [84][84] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-Level Control through Deep Reinforcement Learning. Nature , 518(7540):529-533, 2015. -
  70. [85][85] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In ICLR , 2016. -
  71. [86][86] Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In NIPS , 2015. -
  72. [87][87] Andrew William Moore. Efficient Memory-Based Learning for Robot Control. Technical report, University of Cambridge, Computer Laboratory, 1990. -
  73. [88][88] R´ emi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G Bellemare. Safe and Efficient Off-Policy Reinforcement Learning. In NIPS , 2016. -
  74. [89][89] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the Gap Between Value and Policy Based Reinforcement Learning. arXiv:1702.08892 , 2017. - · arXiv
  75. [90][90] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning. arXiv:1708.02596 , 2017. - · arXiv
  76. [91][91] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively Parallel Methods for Deep Reinforcement Learning. In ICML Workshop on Deep Learning , 2015. -
  77. [92][92] Andrew Y Ng and Stuart J Russell. Algorithms for Inverse Reinforcement Learning. In ICML , 2000. -
  78. [93][93] Andrew Y Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Autonomous Inverted Helicopter Flight via Reinforcement Learning. Experimental Robotics , pages 363-372, 2006. -
  79. [94][94] Brendan O'Donoghue, R´ emi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. PGQ: Combining Policy Gradient and Q-Learning. In ICLR , 2017. -
  80. [95][95] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-Conditional Video Prediction using Deep Networks in Atari Games. In NIPS , 2015. -
  81. [96][96] Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of Memory, Active Perception, and Action in Minecraft. In ICLR , 2016. -
  82. [97][97] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep Exploration via Bootstrapped DQN. In NIPS , 2016. -
  83. [98][98] Emilio Parisotto and Ruslan Salakhutdinov. Neural Map: Structured Memory for Deep Reinforcement Learning. arXiv:1702.08360 , 2017. - · arXiv
  84. [99][99] Emilio Parisotto, Jimmy L Ba, and Ruslan Salakhutdinov. ActorMimic: Deep Multitask and Transfer Reinforcement Learning. In ICLR , 2016. -
  85. [100][100] Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racani` ere, David Reichert, Th´ eophane Weber, Daan Wierstra, and Peter Battaglia. Learning Model-Based Planning from Scratch. arXiv:1707.06170 , 2017. - · arXiv
  86. [101][101] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-Driven Exploration by Self-supervised Prediction. In ICML , 2017. -
  87. [102][102] Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games. arXiv:1703.10069 , 2017. - · arXiv
  88. [103][103] Jan Peters, Katharina M¨ ulling, and Yasemin Altun. Relative Entropy Policy Search. In AAAI , 2010. -
  89. [105][105] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adri` a Puigdom` enech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural Episodic Control. In ICML , 2017. -
  90. [106][106] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence Level Training with Recurrent Neural Networks. - In ICLR , 2016. -
  91. [107][107] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. In NIPS , 2011. -
  92. [108][108] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In ICML , 2014. -
  93. [109][109] Martin Riedmiller. Neural Fitted Q Iteration-First Experiences with a Data Efficient Neural Reinforcement Learning Method. In ECML , 2005. -
  94. [110][110] St´ ephane Ross, Geoffrey J Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In AISTATS , 2011. -
  95. [111][111] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning Representations by Back-Propagating Errors. Cognitive Modeling , 5(3):1, 1988. -
  96. [112][112] Gavin A Rummery and Mahesan Niranjan. On-line Q-learning using Connectionist Systems . University of Cambridge, Department of Engineering, 1994. -
  97. [113][113] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy Distillation. In ICLR , 2016. -
  98. [114][114] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv:1606.04671 , 2016. - · arXiv
  99. [115][115] Andrei A Rusu, Matej Vecerik, Thomas Roth¨ orl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-Real Robot Learning from Pixels with Progressive Nets. In CoRL , 2017. -
  100. [116][116] Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv:1703.03864 , 2017. - · arXiv
  101. [117][117] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal Value Function Approximators. In ICML , 2015. -
  102. [118][118] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. In ICLR , 2016. -
  103. [121][121] John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient Estimation using Stochastic Computation Graphs. In NIPS , 2015. -
  104. [122][122] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In ICML , 2015. -
  105. [123][123] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-Dimensional Continuous Control using Generalized Advantage Estimation. In ICLR , 2016. -
  106. [124][124] John Schulman, Pieter Abbeel, and Xi Chen. Equivalence Between Policy Gradients and Soft Q-Learning. arXiv:1704.06440 , 2017. - · arXiv
  107. [125][125] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 , 2017. - · arXiv
  108. [126][126] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando de Freitas. Taking the Human out of the Loop: A Review of Bayesian Optimization. Proc. of the IEEE , 104(1):148-175, 2016. -
  109. [127][127] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic Policy Gradient Algorithms. In ICML , 2014. -
  110. [128][128] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature , 529(7587):484-489, 2016. -
  111. [130][130] Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva. Deep Attention Recurrent Q-Network. In NIPS Workshop on Deep Reinforcement Learning , 2015. -
  112. [131][131] Bradley C Stadie, Pieter Abbeel, and Ilya Sutskever. Third Person Imitation Learning. In ICLR , 2017. -
  113. [132][132] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing Exploration in Reinforcement Learning with Deep Predictive Models. In NIPS Workshop on Deep Reinforcement Learning , 2015. -
  114. [133][133] Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and - Michael L Littman. PAC Model-Free Reinforcement Learning. In ICML , 2006. -
  115. [134][134] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning Multiagent Communication with Backpropagation. In NIPS , 2016. -
  116. [135][135] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, 1998. -
  117. [136][136] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence , 112(1-2):181-211, 1999. -
  118. [137][137] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timoth´ ee Lacroix, Zeming Lin, Florian Richoux, and Nicolas Usunier. TorchCraft: A Library for Machine Learning Research on Real-Time Strategy Games. arXiv:1611.00625 , 2016. - · arXiv
  119. [138][138] Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value Iteration Networks. In NIPS , 2016. -
  120. [139][139] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning. In NIPS , 2017. -
  121. [140][140] Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust Multitask Reinforcement Learning. In NIPS , 2017. -
  122. [141][141] Gerald Tesauro. Temporal Difference Learning and TD-Gammon. Communications of the ACM , 38(3):58-68, 1995. -
  123. [142][142] Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey Kephart, David Levine, Freeman Rawson, and Charles Lefurgy. Managing Power Consumption and Performance of Computing Systems using Reinforcement Learning. In NIPS , 2008. -
  124. [143][143] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. A Deep Hierarchical Approach to Lifelong Learning in Minecraft. In AAAI , 2017. -
  125. [144][144] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A Physics Engine for Model-Based Control. In IROS , 2012. -
  126. [145][145] John N Tsitsiklis and Benjamin Van Roy. Analysis of TemporalDifference Learning with Function Approximation. In NIPS , 1997. -
  127. [147][147] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks. In ICLR , 2017. -
  128. [148][148] Hado van Hasselt. Double Q-Learning. In NIPS , 2010. -
  129. [149][149] Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Learning. In AAAI , 2016. -
  130. [150][150] Harm Vanseijen and Rich Sutton. A Deeper Look at Planning as Learning from Replay. In ICML , 2015. -
  131. [151][151] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, and Koray Kavukcuoglu. Strategic Attentive Writer for Learning Macro-Actions. In NIPS , 2016. -
  132. [152][152] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In ICML , 2017. -
  133. [153][153] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich K¨ uttler, John Agapiou, Julian Schrittwieser, et al. StarCraft II: A New Challenge for Reinforcement Learning. arXiv:1708.04782 , 2017. - · arXiv
  134. [154][154] Niklas Wahlstr¨ om, Thomas B Sch¨ on, and Marc P Deisenroth. Learning Deep Dynamical Models from Image Pixels. IFAC SYSID , 48(28), 2015. -
  135. [155][155] Niklas Wahlstr¨ om, Thomas B Sch¨ on, and Marc P Deisenroth. From Pixels to Torques: Policy Learning with Deep Dynamical Models. In ICML Workshop on Deep Learning , 2015. -
  136. [157][157] Ziyu Wang, Nando de Freitas, and Marc Lanctot. Dueling Network Architectures for Deep Reinforcement Learning. In ICLR , 2016. -
  137. [158][158] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, R´ emi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample Efficient Actor-Critic with Experience Replay. In ICLR , 2017. -
  138. [159][159] Christopher JCH Watkins and Peter Dayan. Q-Learning. Machine Learning , 8(3-4):279-292, 1992. -
  139. [160][160] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images. In NIPS , 2015. -
  140. [161][161] Th´ eophane Weber, S´ ebastien Racani` ere, David P Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdom` enech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. ImaginationAugmented Agents for Deep Reinforcement Learning. In NIPS , 2017. -
  141. [163][163] Daan Wierstra, Alexander F¨ orster, Jan Peters, and J¨ urgen Schmidhuber. Recurrent Policy Gradients. Logic Journal of the IGPL , 18(5):620-634, 2010. -
  142. [164][164] Ronald J Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning , 8(3-4): 229-256, 1992. -
  143. [165][165] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum Entropy Deep Inverse Reinforcement Learning. In NIPS Workshop on Deep Reinforcement Learning , 2015. -
  144. [166][166] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML , volume 14, 2015. -
  145. [167][167] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-Driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. In ICRA , 2017. -
  146. [168][168] Barret Zoph and Quoc V Le. Neural Architecture Search with Reinforcement Learning. In ICLR , 2017. Kai Arulkumaran (ka709@imperial.ac.uk) is a Ph.D. candidate in the Department of Bioengineering at Imperial College London. He received a B.A. in Computer Science at the University of Cambridge in 2012, and an M.Sc. in Biomedical Engineering at Imperial College London in 2014. He was a Research Intern in Twitter Magic Pony and Microsoft Research in 2017. His research focus is deep reinforcement learning and transfer learning for visuomotor control. Marc Peter Deisenroth (m.deisenroth@imperial.ac.uk) is a Lecturer in Statistical Machine Learning in the Department of Computing at Imperial College London and with PROWLER.io. He received an M.Eng. in Computer Science at the University of Karlsruhe in 2006 and a Ph.D. in Machine Learning at the Karlsruhe Institute of Technology in 2009. He has been awarded an Imperial College Research Fellowship in 2014 and received Best Paper Awards at ICRA 2014 and ICCAS 2016. He is a recipient of a Google Faculty Research Award and a Microsoft Ph.D. Scholarship. His research is centred around data-efficient machine learning for autonomous decision making. Miles Brundage (miles.brundage@philosophy.ox.ac.uk) is a Ph.D. candidate in Human and Social Dimensions of Science and Technology at Arizona State University, and a Research Fellow at the University of Oxford's Future of Humanity Institute. He received a B.A. in Political Science at George Washington University in 2010. His research focuses on governance issues related to artificial intelligence. Anil Anthony Bharath (a.bharath@imperial.ac.uk) is a Reader in the Department of Bioengineering at Imperial College London and a Fellow of the Institution of Engineering and Technology. He received a B.Eng. in Electronic and Electrical Engineering from University College London in 1988, and a Ph.D. in Signal Processing from Imperial College London in 1993. He was an academic visitor in the Signal Processing Group at the University of Cambridge in 2006. He is a co-founder of Cortexica Vision Systems. His research interests are in deep architectures for visual inference.

Cited by

Sorted by citations

Authors
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A.
Affiliations
  • University of Oxford (GB)
  • Institution of Engineering and Technology (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
34
Issue
6
Pages
26-38
ISSN
1053-5888 (ISSN-L)
DOI
10.48550/arxiv.1708.05866
arXiv
1708.05866
OpenAlex
W3100789280
Cited by
4,733
References
0
Citations by year
2026: 444 · 2025: 855 · 2024: 806 · 2023: 749 · 2022: 590 · 2021: 511 · 2020: 388 · 2019: 260 · 2018: 122 · 2017: 6
FWCI
126.58
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Signal Processing Magazine (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
deep reinforcement learning, value-based methods, deep Q-network, trust region policy optimization, asynchronous advantage actor-critic, visual understanding, autonomous systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review