The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Reinforcement learning of motor skills in high dimensions: A path integral approach
Abstract
Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensional continuous state-action spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We believe that our new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1999) · Cited by 5,147
Stochastic Differential Equations: An Introduction with Applications.
Jacka, S. D., & Oksendal, B. (1987) · Journal of the American Statistical Association · doi:10.2307/2288814 · Cited by 3,964
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Stochastic Differential Equations
Øksendal, B. (2003) · Universitext · doi:10.1007/978-3-642-14394-6_5 · Cited by 3,081
Controlled Markov Processes and Viscosity Solutions
(2006) · Stochastic Modelling and Applied Probability · doi:10.1007/0-387-31071-1 · Cited by 2,684
Optimal Control and Estimation
Stengel, R. F. (1994) · Cited by 1,617
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Self-citationPeters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Self-citationLearning Attractor Landscapes for Learning Motor Primitives
Ijspeert, A. J., Nakanishi, J., Stefan Schaal, A., Becker, S., Thrun, S., & Obermayer, K. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 572
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Cited by 525
Constructive Incremental Learning from Only Local Information
Schaal, S., & Atkeson, C. G. (1998) · Neural Computation · doi:10.1162/089976698300016963 · Cited by 517
Self-citationPolicy Gradient Methods for Robotics
Peters, J., & Schaal, S. (2006) · 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2006.282564 · Cited by 498
Self-citationProbabilistic inference for solving discrete and continuous state Markov Decision Processes
Toussaint, M., & Storkey, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143963 · Cited by 482
Efficient computation of optimal actions
Todorov, E. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0710743106 · Cited by 417
Linearly-solvable Markov decision problems
Todorov, E. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0176 · Cited by 334
Path integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309
Linear Theory for Control of Nonlinear Stochastic Systems
Kappen, H. J. (2005) · Physical Review Letters · doi:10.1103/physrevlett.95.200201 · Cited by 299
Optimal control as a graphical model inference problem
Kappen, H. J., Gómez, V., & Opper, M. (2012) · Machine Learning · doi:10.1007/s10994-012-5278-7 · Cited by 297
A stochastic reinforcement learning algorithm for learning real-valued functions
Gullapalli, V. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90056-q · Cited by 295
General duality between optimal control and estimation
Todorov, E. (2008) · 2008 47th IEEE Conference on Decision and Control · doi:10.1109/cdc.2008.4739438 · Cited by 248
Learning motor primitives for robotics
Kober, J., & Peters, J. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152577 · Cited by 220
Gaussian process dynamic programming
Deisenroth, M. P., Rasmussen, C. E., & Peters, J. (2009) · Neurocomputing · doi:10.1016/j.neucom.2008.12.019 · Cited by 192
Natural Gradient Learning for Over- and Under-Complete Bases in ICA
Amari, S.-I. (1999) · Neural Computation · doi:10.1162/089976699300015990 · Cited by 173
Using Expectation-Maximization for Reinforcement Learning
Dayan, P., & Hinton, G. E. (1997) · Neural Computation · doi:10.1162/neco.1997.9.2.271 · Cited by 172
Learning to Control in Operational Space
Peters, J., & Schaal, S. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364907087548 · Cited by 162 · cited 3×
“…from statistical learning and empirical inference with classical RL approaches in order to minimize tuning parameters and numerical problems, such that ultimately more efficient algorithms can be developed that scale to significantly more complex learning systems [3], [4], [5], [6], [7], [8], [9].”
BackgroundMethodSelf-citationAn introduction to stochastic control theory, path integrals and reinforcement learning
Kappen, H. J. (2007) · AIP conference proceedings · doi:10.1063/1.2709596 · Cited by 125
Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems
van den Broek, B., Wiegerinck, W., & Kappen, B. (2008) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2473 · Cited by 99
Learning model-free robot control by a Monte Carlo EM algorithm
Vlassis, N. A., Toussaint, M., Kontes, G. D., & Piperidis, S. (2009) · Autonomous Robots · doi:10.1007/s10514-009-9132-0 · Cited by 72
Rollout sampling approximate policy iteration
Christos Dimitrakakis, A., & Lagoudakis, M. G. (2008) · Machine Learning · doi:10.1007/s10994-008-5069-3 · Cited by 64
Machine learning of motor skills for robotics
Schaal, S., & Peters, J. (2007) · Cited by 60
Self-citationMachine learning of motor skills for robotics
Peters, J. (2015) · MPG.PuRe (Max Planck Society) · doi:10.25549/usctheses-m397 · Cited by 35
Reinforcement Learning for Parameterized Motor Primitives
Peters, J., & Schaal, S. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2006.246662 · Cited by 29
Self-citationRelations among ODEs, PDEs, FSDEs, BSDEs, and FBSDEs
Yong, J. (2002) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1997.657832 · Cited by 22
Cited by
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
STOMP: Stochastic trajectory optimization for motion planning
Kalakrishnan, M., Chitta, S., Theodorou, E., Pastor, P., & Schaal, S. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980280 · Cited by 876
Self-citationLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
Information theoretic MPC for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., & Theodorou, E. A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989202 · Cited by 460
Self-citation
Metadata
- Authors
- Theodorou, E., Buchli, J., & Schaal, S.
- Venue
- 2010 IEEE International Conference on Robotics and Automation
- Publisher
- IEEE
- Published
- 2010-05
- Pages
- 2397-2403
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2042882799
- Cited by
- 252
- References
- 33
- FWCI
- 16.11
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.