CoChat
Sign Up
PreprintCited by 90Reinforcement Learning in RoboticsArtificial Intelligence in GamesAdaptive Dynamic Programming Control

Combining policy gradient and Q-learning

Open access
O'Donoghue, Brendan, Munos, Remi, Kavukcuoglu, Koray, Mnih, Volodymyr·2016·arXiv
DOI 10.48550/arxiv.1611.01626arXiv 1611.01626
Open this paper in CoChat — free accountOpen PDF

Abstract

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer. This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values. This connection allows us to estimate the Q-values from the action preferences of the policy, to which we apply Q-learning updates. We refer to the new technique as 'PGQL', for policy gradient and Q-learning. We also establish an equivalency between action-value fitting techniques and actor-critic algorithms, showing that regularized policy gradient techniques can be interpreted as advantage function learning algorithms. We conclude with some numerical examples that demonstrate improved data efficiency and stability of PGQL. In particular, we tested PGQL on the full suite of Atari games and achieved performance exceeding that of both asynchronous advantage actor-critic (A3C) and Q-learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1611.01626

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation , 10(2):251276, 1998.
  2. Mohammad Gheshlaghi Azar, Vicenc ¸ G´ omez, and Hilbert J Kappen. Dynamic policy programming. Journal of Machine Learning Research , 13(Nov):3207-3245, 2012.
  3. J Andrew Bagnell and Jeff Schneider. Covariant policy search. In IJCAI , 2003.
  4. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 2012.
  5. Richard Bellman. Dynamic programming . Princeton University Press, 1957.
  6. Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1207.4708, 2015. · arXiv
  7. Matthew Hausknecht and Peter Stone. On-policy vs. off-policy updates for deep reinforcement learning. Deep Reinforcement Learning: Frontiers and Challenges, IJCAI 2016 Workshop, 2016.
  8. Nicolas Heess, David Silver, and Yee Whye Teh. Actor-critic reinforcement learning with energybased policies. In JMLR: Workshop and Conference Proceedings 24 , pp. 43-57, 2012.
  9. Sham Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems , volume 14, pp. 1531-1538, 2001.
  10. Vijay R Konda and John N Tsitsiklis. On actor-critic algorithms. SIAM Journal on Control and Optimization , 42(4):1143-1166, 2003.
  11. Lucas Lehnert and Doina Precup. Policy gradient methods for off-policy control. arXiv preprint arXiv:1512.04105, 2015. · arXiv
  12. Sergey Levine and Vladlen Koltun. Guided policy search. In Proceedings of the 30th International Conference on Machine Learning (ICML) , pp. 1-9, 2013.
  13. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. · arXiv
  14. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. · arXiv
  15. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop . 2013.
  16. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 02 2015. URL http://dx.doi.org/10.1038/ nature14236 .
  17. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783 , 2016. · arXiv
  18. Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. arXiv preprint arXiv:1609.00150, 2016. · arXiv
  19. Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013. · arXiv
  20. Edwin Pednault, Naoki Abe, and Bianca Zadrozny. Sequential cost-sensitive decision making with reinforcement learning. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 259-268. ACM, 2002.
  21. Jing Peng and Ronald J Williams. Incremental multi-step Q-learning. Machine Learning , 22(1-3): 283-290, 1996.
  22. Jan Peters, Katharina M¨ ulling, and Yasemin Altun. Relative entropy policy search. In AAAI . Atlanta, 2010.
  23. Martin Riedmiller. Neural fitted Q iteration-first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005 , pp. 317-328. Springer Berlin Heidelberg, 2005.
  24. Brian Sallans and Geoffrey E Hinton. Reinforcement learning with factored states and actions. Journal of Machine Learning Research , 5(Aug):1063-1088, 2004.
  25. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. · arXiv
  26. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of The 32nd International Conference on Machine Learning , pp. 1889-1897, 2015.
  27. David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML) , pp. 387-395, 2014.
  28. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.
  29. R. Sutton and A. Barto. Reinforcement Learning: an Introduction . MIT Press, 1998.
  30. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3 (1):9-44, 1988.
  31. Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , volume 99, pp. 1057-1063, 1999.
  32. Philip Thomas. Bias in natural actor-critic algorithms. In Proceedings of The 31st International Conference on Machine Learning , pp. 441-448, 2014.
  33. Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Qlearning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) , pp. 2094-2100, 2016.
  34. Harm Van Seijen, Hado Van Hasselt, Shimon Whiteson, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning , pp. 177-184. IEEE, 2009.
  35. Yin-Hao Wang, Tzuu-Hseng S Li, and Chih-Jui Lin. Backward q-learning: The combination of sarsa algorithm and q-learning. Engineering Applications of Artificial Intelligence , 26(9):2184-2193, 2013.
  36. Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML) , pp. 1995-2003, 2016.
  37. Christopher John Cornish Hellaby Watkins. Learning from delayed rewards . PhD thesis, University of Cambridge England, 1989.
  38. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992.
  39. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science , 3(3):241-268, 1991.

Cited by

Sorted by citations

Authors
O'Donoghue, B., Munos, R., Kavukcuoglu, K., & Mnih, V.
Affiliations
Google (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1611.01626
arXiv
1611.01626
OpenAlex
W2554120691
Cited by
90
References
1
Citations by year
2025: 1 · 2024: 5 · 2023: 7 · 2022: 8 · 2021: 10 · 2020: 17 · 2019: 22 · 2018: 15 · 2017: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
Keywords
policy gradient, Q-learning, off-policy learning, replay buffer, actor-critic, data efficiency, Atari games

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review