The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Policy Gradient Methods for Robotics
Abstract
The acquisition and improvement of motor skills and control policies for robotics from trial and error is of essential importance if robots should ever leave precisely pre-structured environments. However, to date only few existing reinforcement learning methods have been scaled into the domains of high-dimensional robots such as manipulator, legged or humanoid robots. Policy gradient methods remain one of the few exceptions and have found a variety of applications. Nevertheless, the application of such methods is not without peril if done in an uninformed manner. In this paper, we give an overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field. We outline previous applications to robotics and show how the most recently developed methods can significantly improve learning performance. Finally, we evaluate our most promising algorithm in the application of hitting a baseball with an anthropomorphic arm
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Practical Methods of Optimization
Fletcher, R. (2000) · Wiley eBooks · doi:10.1002/9781118723203 · Cited by 7,605
(1999) · Cited by 5,147
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Natural Gradient Works Efficiently in Learning
Amari, S.-I. (1998) · Neural Computation · doi:10.1162/089976698300017746 · Cited by 2,681
Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control
Spall, J. C. (2003) · Wiley eBooks · doi:10.1002/0471722138 · Cited by 1,890
Modelling and Control of Robot Manipulators
Sciavicco, L., & Siciliano, B. (2000) · Advanced Textbooks in Control and Signal Processing · doi:10.1007/978-1-4471-0449-0 · Cited by 1,701
On Choosing and Bounding Probability Metrics
Gibbs, A. L., & Su, F. E. (2002) · International Statistical Review · doi:10.1111/j.1751-5823.2002.tb00178.x · Cited by 1,163
INTRODUCTION TO STOCHASTIC SEARCH AND OPTIMIZATION: ESTIMATION, SIMULATION, AND CONTROL
Nowak, W. (2012) · Medical Entomology and Zoology · Cited by 1,033
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Reinforcement Learning · doi:10.1007/978-1-4615-3618-5_2 · Cited by 903
Kakade, S. M. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 775
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564
Differential dynamic programming
Denham, W. (1971) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1971.1099760 · Cited by 530
Differential dynamic programming
Jacobson, D. H., & Mayne, D. Q. (1970) · CERN Document Server (European Organization for Nuclear Research) · Cited by 521
Likelihood ratio gradient estimation for stochastic systems
Glynn, P. W. (1990) · Communications of the ACM · doi:10.1145/84537.84552 · Cited by 381
PEGASUS: A Policy Search Method for Large MDPs and POMDPs
Ng, A. Y., & Jordan, M. I. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3878 · Cited by 360
Reinforcement Learning for Humanoid Robotics
Peters, J., Vijayakumar, S., & Schaal, S. (2003) · Cited by 309
Self-citationPeters, J., Vijayakumar, S., & Schaal, S. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564096_29 · Cited by 307
Self-citationVariance Reduction Techniques for Gradient Estimates in Reinforcement Learning
Greensmith, E., Bartlett, P. L., & Baxter, J. (2004) · doi:10.5555/1005332.1044710 · Cited by 300
The Optimal Reward Baseline for Gradient-Based Reinforcement LearningPreprint
Weaver, L., & Tao, N. (2001) · Conference on Uncertainty in Artificial Intelligence · doi:10.48550/arxiv.1301.2315 · Cited by 288
Balasubramanian, V. (1997) · Neural Computation · doi:10.1162/neco.1997.9.2.349 · Cited by 236
Likelilood ratio gradient estimation: an overview
Glynn, P. W. (1987) · Proceedings of the 19th conference on Winter simulation - WSC '87 · doi:10.1145/318371.318612 · Cited by 219
Acquiring robot skills via reinforcement learning
(1994) · IEEE Control Systems · doi:10.1109/37.257890 · Cited by 206
Biped dynamic walking using reinforcement learning
Benbrahim, H., & Franklin, J. A. (1997) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(97)00043-2 · Cited by 198
Learning rhythmic movements by demonstration using nonlinear oscillators
Ijspeort, A. J., Nakanishi, J., & Schaal, S. (2003) · IEEE/RSJ International Conference on Intelligent Robots and System · doi:10.1109/irds.2002.1041514 · Cited by 183
Self-citationA Kendama Learning Robot Based on Bi-directional Theory
Miyamoto, H., Schaal, S., Gandolfo, F., Gomi, H., Koike, Y., Osu, R., Nakano, E., Wada, Y., & Kawato, M. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00043-3 · Cited by 171
The Computation and Theory of Optimal Control
Dyer, P., McReynolds, S. R., & Eggleston, D. M. (1972) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3426573 · Cited by 152
Bagnell, J. A., & Schneider, J. (2018) · Figshare · doi:10.1184/r1/6552458 · Cited by 148
Gradient Optimization and Nonlinear Control
Hasdorff, L. (1976) · Medical Entomology and Zoology · Cited by 137
Fast Biped Walking with a Sensor-driven Neuronal Controller and Real-time Online Learning
Geng, T., Porr, B., & Wörgötter, F. (2006) · The International Journal of Robotics Research · doi:10.1177/0278364906063822 · Cited by 136
The Computation and Theory of Optimal Control
(1970) · Mathematics in Science and Engineering/Mathematics in science and engineering · doi:10.1016/s0076-5392(08)x6199-5 · Cited by 136
Direct gradient-based reinforcement learning
Baxter, J., & Bartlett, P. L. (2002) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2000.856049 · Cited by 94
Reinforcement learning for a CPG-driven biped robot
Mori, T., Nakamura, Y., Sato, M.-A., & Ishii, S. (2004) · National Conference on Artificial Intelligence · Cited by 70
Mitsunaga, N., Smith, C., Kanda, T., Ishiguro, H., & Hagita, N. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2005.1545206 · Cited by 65
Machine learning of motor skills for robotics
Schaal, S., & Peters, J. (2007) · Cited by 60
Self-citationLearning CPG sensory feedback with policy gradient for biped locomotion for a full-body humanoid
Endo, G., Morimoto, J., Matsubara, T., Nakanishi, J., & Cheng, G. (2005) · National Conference on Artificial Intelligence · Cited by 35
Machine learning of motor skills for robotics
Peters, J. (2015) · MPG.PuRe (Max Planck Society) · doi:10.25549/usctheses-m397 · Cited by 35
Self-citationReinforcement Learning for Biped Locomotion
Sato, M.-A., Nakamura, Y., & Ishii, S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-46084-5_126 · Cited by 34
Real-time learning: a ball on a beam
Benbrahim, H., Doleac, J., Franklin, J. A., & Selfridge, O. G. (2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.287219 · Cited by 29
Efficient Gradient Estimation for Motor Control LearningPreprint
Lawrence, G., Cowan, N. J., & Russell, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.2475 · Cited by 28
Statistical Machine Learning and Combinatorial Optimization
Berny, A. (2001) · Natural computing series · doi:10.1007/978-3-662-04448-3_14 · Cited by 25
Associative reinforcement learning of real-valued functions
Gullapalli, V. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1991.169893 · Cited by 24
Natural Policy Gradient Reinforcement Learning for a CPG Control of a Biped Robot
Nakamura, Y., Mori, T., & Ishii, S. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30217-9_98 · Cited by 14
Fukushima-Kimura, B. H., Kamijima, Y., Kawamura, K., & Sakai, A. (2023) · Transactions of the Institute of Systems Control and Information Engineers · doi:10.5687/iscie.36.9 · Cited by 6
Cited by
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Self-citationA Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Self-citationGaussian Processes for Data-Efficient Learning in Robotics and Control
Deisenroth, M. P., Fox, D., & Rasmussen, C. E. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.218 · Cited by 659
A Survey on Machine-Learning Techniques in Cognitive Radios
Bkassiny, M., Li, Y., & Jayaweera, S. K. (2013) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2012.100412.00017 · Cited by 580
Metadata
- Authors
- Peters, J., & Schaal, S.
- Venue
- 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems
- Publisher
- IEEE
- Published
- 2006-10
- Pages
- 2219-2225
- OpenAlex
- W2127107099
- Cited by
- 498
- References
- 43
- FWCI
- 10.32
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.