The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Reinforcement learning of motor skills in high dimensions: A path integral approach
Abstract
Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensional continuous state-action spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We believe that our new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1999) · Cited by 5,147
Stochastic Differential Equations: An Introduction with Applications.
Jacka, S. D., & Oksendal, B. (1987) · Journal of the American Statistical Association · doi:10.2307/2288814 · Cited by 3,964
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Stochastic Differential Equations
Øksendal, B. (2003) · Universitext · doi:10.1007/978-3-642-14394-6_5 · Cited by 3,081
Controlled Markov Processes and Viscosity Solutions
(2006) · Stochastic Modelling and Applied Probability · doi:10.1007/0-387-31071-1 · Cited by 2,684
Optimal Control and Estimation
Stengel, R. F. (1994) · Cited by 1,617
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Self-citationPeters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Self-citationLearning Attractor Landscapes for Learning Motor Primitives
Ijspeert, A. J., Nakanishi, J., Stefan Schaal, A., Becker, S., Thrun, S., & Obermayer, K. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 572
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Cited by 525
Cited by
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
STOMP: Stochastic trajectory optimization for motion planning
Kalakrishnan, M., Chitta, S., Theodorou, E., Pastor, P., & Schaal, S. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980280 · Cited by 876
Self-citationLearning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
Information theoretic MPC for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., & Theodorou, E. A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989202 · Cited by 460
Self-citationPolicy search for motor primitives in robotics
Kober, J., & Peters, J. (2011) · Machine Learning · doi:10.1007/s10994-010-5223-6 · Cited by 427
Learning variable impedance control
Buchli, J., Stulp, F., Theodorou, E., & Schaal, S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911402527 · Cited by 337
Self-citationOptimal control as a graphical model inference problem
Kappen, H. J., Gómez, V., & Opper, M. (2012) · Machine Learning · doi:10.1007/s10994-012-5278-7 · Cited by 297
Learning contact-rich manipulation skills with guided policy search
Levine, S., Wagener, N., & Abbeel, P. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7138994 · Cited by 267
Robot motor skill coordination with EM-based Reinforcement Learning
Kormushev, P., Calinon, S., & Caldwell, D. G. (2010) · 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2010.5649089 · Cited by 258
Data-Efficient Generalization of Robot Skills with Contextual Policy Search
Kupcsik, A., Deisenroth, M., Peters, J., & Neumann, G. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8546 · Cited by 92
(2010) · Uncertainty in Artificial Intelligence · Cited by 56
Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
Xiao, T., Jang, E., Kalashnikov, D., Levine, S., Ibarz, J., Hausman, K., & Herzog, A. (2020) · arXiv · doi:10.48550/arxiv.2004.06089 · Cited by 25
Metadata
- Authors
- Theodorou, E., Buchli, J., & Schaal, S.
- Venue
- 2010 IEEE International Conference on Robotics and Automation
- Publisher
- IEEE
- Published
- 2010-05
- Pages
- 2397-2403
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2042882799
- Cited by
- 252
- References
- 33
- FWCI
- 16.11
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.