Learning to Control in Operational Space
Abstract
One of the most general frameworks for phrasing control problems for complex, redundant robots is operational-space control. However, while this framework is of essential importance for robotics and well understood from an analytical point of view, it can be prohibitively hard to achieve accurate control in the face of modeling errors, which are inevitable in complex robots (e.g. humanoid robots). In this paper, we suggest a learning approach for operational-space control as a direct inverse model learning problem. A first important insight for this paper is that a physically correct solution to the inverse problem with redundant degrees of freedom does exist when learning of the inverse map is performed in a suitable piecewise linear way. The second crucial component of our work is based on the insight that many operational-space controllers can be understood in terms of a constrained optimal control problem. The cost function associated with this optimal control problem allows us to formulate a learning algorithm that automatically synthesizes a globally consistent desired resolution of redundancy while learning the operational-space controller. From the machine learning point of view, this learning problem corresponds to a reinforcement learning problem that maximizes an immediate reward. We employ an expectation-maximization policy search algorithm in order to solve this problem. Evaluations on a three degrees-of-freedom robot arm are used to illustrate the suggested approach. The application to a physically realistic simulator of the anthropomorphic SARCOS Master arm demonstrates feasibility for complex high degree-of-freedom robots. We also show that the proposed method works in the setting of learning resolved motion rate control on a real, physical Mitsubishi PA-10 medical robotics arm.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,934 · see citing works
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,535 · see citing works
Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Williams, C. K. I. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s269 · Cited by 4,390 · see citing works
Spong, M. W., Hutchinson, S., & Vidyasagar, M. (2006) · Cited by 3,273 · see citing works
Khatib, O. (1987) · IEEE J. Robotics Autom. · doi:10.1109/jra.1987.1087068 · Cited by 3,233 · see citing works
Introduction to Stochastic Search and Optimization
Spall, J. C. (2003) · Wiley eBooks · doi:10.1002/0471722138 · Cited by 1,912 · see citing works
Atkeson, C. G., Moore, A., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006559212014 · Cited by 1,684 · see citing works
Self-citationCabano, F. G., Müller, S., & Gelbrich, K. (2024) · Routledge eBooks · doi:10.4324/9781003506010-51 · Cited by 1,479 · see citing works
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451 · see citing works
Advanced robotics: redundancy and optimization
(1991) · Choice Reviews Online · doi:10.5860/choice.29-0333 · Cited by 1,137 · see citing works
INTRODUCTION TO STOCHASTIC SEARCH AND OPTIMIZATION: ESTIMATION, SIMULATION, AND CONTROL
Nowak, W. (2012) · Medical Entomology and Zoology · Cited by 1,030 · see citing works
Modeling and Control of Robot Manipulators
(2000) · Industrial Robot the international journal of robotics research and application · doi:10.1108/ir.2000.04927aae.003 · Cited by 989 · see citing works
Information Theory, Inference, and Learning Algorithms
(2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.834752 · Cited by 866 · see citing works
Adaptive Approximation Based Control
Farrell, J. A., & Polycarpou, M. M. (2006) · Wiley eBooks · doi:10.1002/0471781819 · Cited by 622 · see citing works
Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94)
(1994) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.1994.407417 · Cited by 551 · see citing works
Locally Weighted Learning for Control
Atkeson, C. G., Moore, A., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006511328852 · Cited by 551 · see citing works
Self-citationIncremental Online Learning in High Dimensions
Vijayakumar, S., D’Souza, A., & Stefan Schaal, A. (2005) · Neural Computation · doi:10.1162/089976605774320557 · Cited by 527 · see citing works
IEEE journal of robotics and automation
Robotics, I. (1985) · Institute of Electrical and Electronics Engineers eBooks · Cited by 494 · see citing works
Modelling and Control of Robot Manipulators
Siciliano, L. S. A. B. (2000) · Measurement Science and Technology · doi:10.1088/0957-0233/11/12/709 · Cited by 428 · see citing works
Udwadia, F. E., & Kalaba, R. E. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511665479 · Cited by 344 · see citing works
Modeling and control of robot manipulators
Lew, J. Y. (2002) · Automatica · doi:10.1016/s0005-1098(01)00301-6 · Cited by 342 · see citing works
A Self-Organizing Neural Model of Motor Equivalent Reaching and Tool Use by a Multijoint Arm
Bullock, D., Grossberg, S., & Guenther, F. H. (1993) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1993.5.4.408 · Cited by 316 · see citing works
A Theory of Generalized Inverses Applied to Robotics
Doty, K. L., Melchiorri, C., & Bonivento, C. (1993) · The International Journal of Robotics Research · doi:10.1177/027836499301200101 · Cited by 310 · see citing works
D'Souza, A., Vijayakumar, S., & Schaal, S. (2002) · doi:10.1109/iros.2001.973374 · Cited by 287 · see citing works
Self-citationScalable Techniques from Nonparametric Statistics for Real Time Robot Learning
Schaal, S., Atkeson, C. G., & Vijayakumar, S. (2002) · Applied Intelligence · doi:10.1023/a:1015727715131 · Cited by 236 · see citing works
Self-citationLocal and global sparse Gaussian process approximations.
Snelson, E., & Zoubin Ghahramani, A. (2007) · Cited by 236 · see citing works
Dynamic control of redundant manipulators
Hsu, P., Mauser, J., & Sastry, S. (1989) · Journal of Robotic Systems · doi:10.1002/rob.4620060203 · Cited by 223 · see citing works
Reinforcement learning by reward-weighted regression for operational space control
Peters, J., & Stefan Schaal, A. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273590 · Cited by 188 · see citing works
Self-citationUsing Expectation-Maximization for Reinforcement Learning
Dayan, P., & Hinton, G. E. (1997) · Neural Computation · doi:10.1162/neco.1997.9.2.271 · Cited by 175 · see citing works
Solution to the inverse kinematics problem in robotics by neural networks
Guez, & Ahmad (1988) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1988.23979 · Cited by 168 · see citing works
Inverse kinematics for humanoid robots
Tevatia, G., & Schaal, S. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2000.844073 · Cited by 148 · see citing works
Self-citationControl of Free-Floating Humanoid Robots Through Task Prioritization
Sentis, L., & Khatib, O. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2005.1570361 · Cited by 129 · see citing works
Modeling and Control of the Mitsubishi PA-10 Robot Arm Harmonic Drive System
Kennedy, C. W., & Desai, J. (2005) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2005.848290 · Cited by 114 · see citing works
Multiple paired forward-inverse models for human motor learning and control
Haruno, M., Wolpert, D. M., & Kawato, M. (1999) · Cited by 94 · see citing works
Statistical Learning for Humanoid Robots
Vijayakumar, S., D’Souza, A., Shibata, T., Conradt, J., & Schaal, S. (2002) · Autonomous Robots · doi:10.1023/a:1013258808932 · Cited by 89 · see citing works
Self-citationGauss' principle and the dynamics of redundant and constrained manipulators
Bruyninckx, H., & Khatib, O. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2000.846414 · Cited by 85 · see citing works
Comparative experiments on task space control with redundancy resolution
Nakanishi, J., Cory, R., Mistry, M., et al. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2005.1545203 · Cited by 82 · see citing works
A Bayesian Approach to Nonlinear Parameter Identification for Rigid Body Dynamics
Ting, J.-A., Mistry, M., Peters, J., et al. (2006) · doi:10.15607/rss.2006.ii.032 · Cited by 62 · see citing works
Self-citationFarrell, J. A., & Polycarpou, M. M. (2006) · Wiley-Interscience eBooks · Cited by 52 · see citing works
On dynamical decoupling of kinematically redundant manipulators
Park, J., Chung, W., & Youm, Y. (2003) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.1999.811690 · Cited by 41 · see citing works
A unifying methodology for the control of robotic systems
Peters, J., Mistry, M., Udwadia, F. E., et al. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2005.1545516 · Cited by 36 · see citing works
Self-citationAdaptive Approximation Based Control: General Theory
Farrell, J. A., & Polycarpou, M. M. (2006) · doi:10.1002/0471781819.ch7 · Cited by 35 · see citing works
Learning control for redundant manipulators
Luca, A. D., & Mataloni, F. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1991.131818 · Cited by 15 · see citing works
Learning composite adaptive control for a class of nonlinear systems
Nakanishi, J., Farrell, J. A., & Schaal, S. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307460 · Cited by 10 · see citing works
Self-citationNakanishi, J., Mistry, M., Peters, J., & Stefan Schaal, A. (2007) · Cited by 5 · see citing works
Self-citationReal-time learning of resolved velocity control on a Mitsubishi PA-10
Peters, J., & Nguyen-Tuong, D. (2008) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2008.4543645 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Self-citationModel learning for robot control: A survey
Nguyen-Tuong, D., & Peters, J. (2011) · Cognitive Processing · doi:10.1007/s10339-011-0404-1 · Cited by 503 · see citing works
Self-citationSalvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“In addition, policy search approaches result in more efficient trainings in terms of time needed for convergence [33, 34, 35, 36, 37, 38].”
Background
- Authors
- Peters, J., & Schaal, S.
- Affiliations
- University of Southern California (US)
- University of Southern California, 3641 Watt Way, Los Angeles, CA 90089, USA, ATR Computational Neuroscience Laboratory, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
- Corresponding author
- Peters, Jan
- ORCID
- Peters, Jan
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2008-02
- Volume
- 27
- Issue
- 2
- Pages
- 197-212
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W2168921921
- Cited by
- 162
- References
- 49
- Citations by year
- 2026: 1 · 2024: 7 · 2023: 4 · 2022: 7 · 2021: 7 · 2019: 3 · 2018: 10 · 2017: 7 · 2016: 12 · 2015: 9 · 2014: 17 · 2013: 8 · 2012: 11
- FWCI
- 30.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The International Journal of Robotics Research (publishedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robotic Mechanisms and Dynamics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
- Keywords
- operational space control, redundant manipulators, redundancy resolution, constrained optimal control, reinforcement learning, resolved motion rate control, humanoid robots, robotic arm control
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
