CoChat
Sign Up
PreprintCited by 366Reinforcement Learning in RoboticsAdaptive Dynamic Programming ControlMachine Learning and Algorithms

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Open access
Levine, Sergey·2018·arXiv
DOI 10.48550/arxiv.1805.00909arXiv 1805.00909
Open this paper in CoChat — free accountOpen PDF

Abstract

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.00909

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Figure 7 shows that expected free energy contains terms that arise in various formulations of optimal behaviour, ranging from optimal Bayesian design to control as inference.

    “Figure 7 shows that the expected free energy contains terms that arise in various formulations of optimal behaviour; ranging from optimal Bayesian design [134] through to control as inference [132, 127].”

    From The free energy principle made simpler but not too simple (2023) · cites this paper as [127]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. (2018). Maximum a posteriori policy optimisation. In International Conference on Learning Representations (ICLR) .
  2. Attias, H. (2003). Planning by probabilistic inference. In Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics .
  3. Bagnell, J. A. and Schneider, J. (2003). Covariant policy search. In International Joint Conference on Artifical Intelligence (IJCAI) .
  4. Botvinick, M. and An, J. (2009). Goal-directed decision making in prefrontal cortex: a computational framework. In Advances in Neural Information Processing Systems (NIPS) .
  5. Botvinick, M. and Toussaint, M. (2012). Planning as inference. Trends in Cognitive Sciences , 16(10):485-488.
  6. Dragan, A. D., Lee, K. C. T., and Srinivasa, S. S. (2013). Legibility and predictability of robot motion. In International Conference on Human-Robot Interaction (HRI) .
  7. Dvijotham, K. and Todorov, E. (2010). Inverse optimal control with linearly-solvable mdps. In International Conference on International Conference on Machine Learning (ICML) .
  8. Finn, C., Christiano, P., Abbeel, P., and Levine, S. (2016a). A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. CoRR , abs/1611.03852. · arXiv
  9. Finn, C., Levine, S., and Abbeel, P. (2016b). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML) .
  10. Friston, K. (2009). The free-energy principle: A rough guide to the brain? Trends in Cognitive Sciences , 13(7):293-301.
  11. Fu, J., Luo, K., and Levine, S. (2018). Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations (ICLR) .
  12. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Neural Information Processing Systems (NIPS) .
  13. Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S. (2018). Meta-reinforcement learning of structured exploration strategies. CoRR , abs/1802.07245. · arXiv
  14. Haarnjoa, T., Pong, V., Zhou, A., Dalal, M., Abbeel, P., and Levine, S. (2018). Composable deep reinforcement learning for robotic manipulation. In International Conference on Robotics and Automation (ICRA) .
  15. Haarnoja, T., Hartikainen, K., Abbeel, P., and Levine, S. (2018a). Latent space policies for hierarchical reinforcement learning. CoRR , abs/1804.02808. · arXiv
  16. Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. (2017). Reinforcement learning with deep energybased policies. In International Conference on Machine Learning (ICML) .
  17. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018b). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In arXiv .
  18. Hachiya, H., Peters, J., and Sugiyama, M. (2009). Efficient sample reuse in em-based policy search. In European Conference on Machine Learning (ECML) .
  19. Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. (2018). Learning an embedding space for transferable robot skills. In International Conference on Learning Representations (ICLR) .
  20. Heess, N., Silver, D., and Teh, Y . W. (2013). Actor-critic reinforcement learning with energy-based policies. In European Workshop on Reinforcement Learning (EWRL) .
  21. Huang, D., Farahmand, A., Kitani, K. M., and Bagnell, J. A. (2015). Approximate MaxEnt inverse optimal control and its application for mental simulation of human interactions. In AAAI Conference on Artificial Intelligence (AAAI) .
  22. Huang, D. and Kitani, K. M. (2014). Action-reaction: Forecasting the dynamics of human interaction. In European Conference on Computer Vision (ECCV) .
  23. Kalman, R. (1960). A new approach to linear filtering and prediction problems. ASMETransactions journal of basic engineering , 82(1):35-45.
  24. Kappen, H. J. (2011). Optimal control theory and the linear bellman equation. Inference and Learning in Dynamic Models , pages 363-387.
  25. Kappen, H. J., Gómez, V., and Opper, M. (2012). Optimal control as a graphical model inference problem. Machine Learning , 87(2):159-182.
  26. Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques . The MIT Press.
  27. Levine, S. (2014). Motor skill learning with local trajectory methods . PhD thesis, Stanford University.
  28. Levine, S. and Abbeel, P. (2014). Learning neural network policies with guided policy search under unknown dynamics. In Neural Information Processing Systems (NIPS) .
  29. Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(1).
  30. Levine, S. and Koltun, V. (2012). Continuous inverse optimal control with locally optimal examples. In International Conference on Machine Learning (ICML) .
  31. Levine, S. and Koltun, V. (2013a). Guided policy search. In International Conference on International Conference on Machine Learning (ICML) .
  32. Levine, S. and Koltun, V. (2013b). Variational policy search via trajectory optimization. In Advances in Neural Information Processing Systems (NIPS) .
  33. Levine, S. and Koltun, V. (2014). Learning complex neural network policies with trajectory optimization. In International Conference on Machine Learning (ICML) .
  34. Levine, S., Popovi´ c, Z., and Koltun, V. (2011). Nonlinear inverse reinforcement learning with gaussian processes. In Neural Information Processing Systems (NIPS) .
  35. Minka, T. P. (2001). Expectation propagation for approximate bayesian inference. In Uncertainty in Artificial Intelligence (UAI) .
  36. Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. (2017a). Bridging the gap between value and policy based reinforcement learning. In arXiv .
  37. Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. (2017b). Trust-pcl: An off-policy trust region method for continuous control. CoRR , abs/1707.01891. · arXiv
  38. Neumann, G. (2011). Variational inference for policy search in changing situations. In International Conference on Machine Learning (ICML) .
  39. O'Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. (2017). Pgq: Combining policy gradient and q-learning. In International Conference on Learning Representations (ICLR) .
  40. Peters, J., Mülling, K., and Altün, Y. (2010). Relative entropy policy search. In AAAI Conference on Artificial Intelligence (AAAI) .
  41. Peters, J. and Schaal, S. (2007). Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning (ICML) .
  42. Sallans, B. and Hinton, G. E. (2004). Reinforcement learning with factored states and actions. Journal of Machine Learning Research , 5.
  43. Schulman, J., Chen, X., and Abbeel, P. (2017). Equivalence between policy gradients and soft q-learning. In arXiv .
  44. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2015). Trust region policy optimization. In International Conference on Machine Learning (ICML) .
  45. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR) .
  46. Solway, A. and Botvinick, M. (2012). Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol Rev. , 119(1):120-154.
  47. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning (ICML) .
  48. Theodorou, E. A., Buchli, J., and Schaal, S. (2010). Learning policy improvements with path integrals. In International Conference on Artificial Intelligence and Statistics (AISTATS 2010) .
  49. Todorov, E. (2006). Linearly-solvable markov decision problems. In Advances in Neural Information Processing Systems (NIPS) .
  50. Todorov, E. (2008). General duality between optimal control and estimation. In Conference on Decision and Control (CDC) .
  51. Todorov, E. (2010). Policy gradients in linearly-solvable mdps. In Neural Information Processing Systems (NIPS) .
  52. Toussaint, M. (2009). Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML) .
  53. Toussaint, M. and Storkey, A. (2006). Probabilistic inference for solving discrete and continuous state markov decision processes. In International Conference on Machine Learning (ICML) .
  54. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3-4):229-256.
  55. Williams, R. J. and Peng, J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science , 3(3):241-268.
  56. Wulfmeier, M., Ondruska, P., and Posner, I. (2015). Maximum entropy deep inverse reinforcement learning. In Neural Information Processing Systems Conference, Deep Reinforcement Learning Workshop .
  57. Ziebart, B. (2010). Modeling purposeful adaptive behavior with the principle of maximum causal entropy . PhD thesis, Carnegie Mellon University.
  58. Ziebart, B. D., Bagnell, J. A., and Dey, A. K. (2010). Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning (ICML) .
  59. Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. (2008). Maximum entropy inverse reinforcement learning. In International Conference on Artificial Intelligence (AAAI) .

Cited by

Sorted by citations

Authors
Levine, S.
Corresponding author
Levine, Sergey
ORCID
Levine, Sergey
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1805.00909
arXiv
1805.00909
OpenAlex
W2799151646
Cited by
366
References
2
Citations by year
2026: 3 · 2025: 24 · 2024: 36 · 2023: 44 · 2022: 71 · 2021: 74 · 2020: 64 · 2019: 38 · 2018: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
reinforcement learning, optimal control, probabilistic inference, maximum entropy reinforcement learning, variational inference, deterministic dynamics, stochastic dynamics, approximate inference, partial observability, compositionality, decision making under uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review