CoChat
Sign Up
PreprintCited by 259Reinforcement Learning in RoboticsAdaptive Dynamic Programming ControlDomain Adaptation and Few-Shot Learning

Maximum a Posteriori Policy Optimisation

Open access
Abdolmaleki, Abbas, Springenberg, Jost Tobias, Tassa, Yuval, Munos, Remi, Heess, Nicolas, Riedmiller, Martin·2018·arXiv
DOI 10.48550/arxiv.1806.06920arXiv 1806.06920
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.06920

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Constrained optimization targets a fixed entropy on average across states regardless of achievable returns, which is robust but explores slowly under sparse rewards and converges lower under dense rewards.

    “Constrained optimization targets a fixed entropy on average across states30,31 regardless of achievable returns, which is robust but explores slowly under sparse rewards and converges lower under dense rewards.”

    From Mastering diverse control tasks through world models (2025) · cites this paper as [31]

    The Constrained E-step (p. 5) says the method imposes a hard constraint on KL divergence, not a fixed target for policy entropy. The results discussion (p. 9) describes MPO as stable across environments and attributes slow learning on sparse-reward tasks to DDPG, not to MPO's constraint. The paper does not establish the claimed lower convergence under dense rewards. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Dreamer sets a new state-of-the-art on Proprio Control, outperforming D4PG, DMPO, and MPO.

    “Dreamer sets a new state-of-the-art on this benchmark, outperforming D4PG, DMPO, and MPO 33 .”

    From Mastering Diverse Domains through World Models (2023) · cites this paper as [33]

  • The variability in terms of learning algorithms is higher, with DRL using PPO, TRPO, MPO, A3C, SAC, or DDPG, among others.

    “The variability in terms of learning algorithms is higher, with DRL using proximal policy optimization (PPO) [45], trust region policy optimization (TRPO) [46], maximum aposteriori policy optimization (MPO) [47], asynchronous actor critic (A3C) methods [48], soft actor critic (SAC) [49], or deep deterministic policy gradient (DDPG) [50], among others.”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [47]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis Paulo Reis, Gerhard Neumann, et al. Deriving and improving cma-es with information geometric trust regions. Proceedings of the Genetic and Evolutionary Computation Conference , 2017.
  2. Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML , 2017.
  3. Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. CoRR , abs/1610.00529, 2016. · arXiv
  4. Kamil Ciosek and Shimon Whiteson. Expected policy gradients. CoRR , abs/1706.05374, 2017. · arXiv
  5. C. Daniel, G. Neumann, O. Kroemer, and J. Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research (JMLR) , 2016.
  6. Peter Dayan and Geoffrey E Hinton. Using expectation-maximization for reinforcement learning. Neural Computation , 9(2):271-278, 1997.
  7. Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends in Robotics , 2(1-2):1-142, 2013.
  8. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, pp. 13291338. JMLR.org, 2016. URL http://dl.acm.org/citation.cfm?id=3045390. 3045531 .
  9. Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. CoRR , abs/1704.03012, 2017. · arXiv
  10. Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence UAI , 2016.
  11. Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation. arXiv preprint arXiv:1610.00633 , 2016a. · arXiv
  12. Shixiang Gu, Tim Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning (ICML) , 2016b.
  13. Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Qprop: Sample-efficient policy gradient with an off-policy critic. In 5th International Conference on Learning Representations (ICLR) , 2017.
  14. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. CoRR , abs/1702.08165, 2017. · arXiv
  15. Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NIPS) , pp. 2926-2934, 2015.
  16. Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 , 2017. · arXiv
  17. HJ Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment , 2005.
  18. Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Advances in neural information processing systems , pp. 849-856, 2009.
  19. Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimization. In Advances in Neural Information Processing Systems , pp. 207-215, 2013.
  20. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR) , 2016.
  21. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.
  22. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML) , 2016.
  23. William Montgomery and Sergey Levine. Guided policy search as approximate mirror descent. CoRR , abs/1607.04614, 2016. URL http://arxiv.org/abs/1607.04614 . · arXiv
  24. Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient offpolicy reinforcement learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (NIPS) , 2016.
  25. Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the 28th international conference on machine learning (ICML-11) , pp. 817-824, 2011.
  26. Brendan O'Donoghue, Rémi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. PGQ: combining policy gradient and q-learning. CoRR , abs/1611.01626, 2016. · arXiv
  27. Brendan O'Donoghue, Ian Osband, Rémi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. CoRR , abs/1709.05380, 2017. URL http://arxiv.org/abs/ 1709.05380 . · arXiv
  28. Xue Bin Peng, Glen Berseth, and Michiel van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) , 2016.
  29. Theodore J. Perkins and Doina Precup. A convergent form of approximate policy iteration. In Advances in Neural Information Processing Systems 15 (NIPS) . MIT Press, Cambridge, MA, 2002.
  30. Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI) , 2010a.
  31. Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI . Atlanta, 2010b.
  32. Nicolas Le Roux. Efficient iterative policy optimization. CoRR , abs/1612.08967, 2016. URL http://arxiv.org/abs/1612.08967 . · arXiv
  33. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015.
  34. John Schulman, Pieter Abbeel, and Xi Chen. Equivalence between policy gradients and soft qlearning. CoRR , abs/1704.06440, 2017a. · arXiv
  35. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017b. URL http://arxiv.org/abs/ 1707.06347 . · arXiv
  36. David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML) , 2014.
  37. Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning (ICML) , pp. 216-224, 1990.
  38. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018. URL http://arxiv.org/abs/1801.00690 . · arXiv
  39. Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. Journal of Machine Learning Research (JMLR) , 2010.
  40. Emanuel Todorov. General duality between optimal control and estimation. In Proceedings of the 47th IEEE Conference on Decision and Control, CDC 2008, December 9-11, 2008, Cancún, México , pp. 4286-4292, 2008.
  41. Ahmed Touati, Pierre-Luc Bacon, Doina Precup, and Pascal Vincent. Convergent tree-backup and retrace with function approximation. CoRR , abs/1705.09322, 2017. URL http://arxiv. org/abs/1705.09322 . · arXiv
  42. Marc Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, pp. 1049-1056, 2009. ISBN 978-1-60558-516-1.
  43. Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. 5th International Conference on Learning Representations (ICLR) , 2017.
  44. Christian Wirth, Johannes Furnkranz, and Gerhard Neumann. Model-free preference-based reinforcement learning. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016 , pp. 22222228, 2016.
  45. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI , pp. 1433-1438, 2008.

Cited by

Sorted by citations

Authors
Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., & Riedmiller, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.06920
arXiv
1806.06920
OpenAlex
W2785738552
Cited by
259
References
2
Citations by year
2026: 1 · 2025: 13 · 2024: 29 · 2023: 26 · 2022: 23 · 2021: 62 · 2020: 58 · 2019: 41 · 2018: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
reinforcement learning algorithms, coordinate ascent, Kullback-Leibler divergence, off-policy learning, continuous control, sample efficiency, premature convergence, hyperparameter robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review