The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Reinforcement Learning in Robotics: Applications and Real-World Challenges
Open accessAbstract
In robotics, the ultimate goal of reinforcement learning is to endow robots with the ability to learn, improve, adapt and reproduce tasks with dynamically changing constraints based on exploration and autonomous learning. We give a summary of the state-of-the-art of reinforcement learning in the context of robotics, in terms of both algorithms and policy representations. Numerous challenges faced by the policy representation in robotics are identified. Three recent examples for the application of reinforcement learning to real-world robots are described: a pancake flipping task, a bipedal walking energy minimization task and an archery-based aiming task. In all examples, a state-of-the-art expectation-maximization-based reinforcement learning is used, and different policy representations are proposed and evaluated for each task. The proposed policy representations offer viable solutions to six rarely-addressed challenges in policy representations: correlations, adaptability, multi-resolution, globality, multi-dimensionality and convergence. Both the successes and the practical difficulties encountered in these examples are discussed. Based on insights from these particular cases, conclusions are drawn about the state-of-the-art and the future perspective directions for reinforcement learning in robotics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Corke, P., Roberts, J., Cunningham, J., & Hainsworth, D. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_50 · Cited by 3,543
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
Optimal feedback control as a theory of motor coordination
Todorov, E., & Jordan, M. I. (2002) · Nature Neuroscience · doi:10.1038/nn963 · Cited by 3,081 · cited 1×
“[30] a framework to learn coordination for simple rest-to-rest movements, by taking inspiration of the motor coordination, joint synergies and the importance of coupling in motor control [31–33].”
BackgroundThe CMA Evolution Strategy: A Comparing Review
Hansen, N. (2007) · Studies in Fuzziness and Soft Computing · doi:10.1007/3-540-32494-1_4 · Cited by 1,797
Rubinstein, R. Y., & Kroese, D. P. (2004) · Medical Entomology and Zoology · Cited by 1,074
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Policy Gradient Methods for Robotics
Peters, J., & Schaal, S. (2006) · 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2006.282564 · Cited by 498
Policy search for motor primitives in robotics
Kober, J., & Peters, J. (2011) · Machine Learning · doi:10.1007/s10994-010-5223-6 · Cited by 427
Hoffmann, H., Pastor, P., Park, D.-H., & Schaal, S. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152423 · Cited by 283
A Generalized Path Integral Control Approach to Reinforcement Learning
TheodorouEvangelos, BuchliJonas, & SchaalStefan (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953033 · Cited by 280
Kormushev, P., Calinon, S., & Caldwell, D. G. (2011) · Advanced Robotics · doi:10.1163/016918611x558261 · Cited by 280
Self-citationTsagarakis, N. G., Metta, G., Sandini, G., Vernon, D., Beira, R., Becchi, F., Righetti, L., Santos-Victor, J., Ijspeert, A. J., Carrozza, M. C., et al. (2007) · Advanced Robotics · doi:10.1163/156855307781389419 · Cited by 276
Self-citationDynamics systems vs. optimal control — a unifying view
Schaal, S., Mohajerian, P., & Ijspeert, A. (2007) · Progress in Brain Research · doi:10.1016/s0079-6123(06)65027-9 · Cited by 260
Moore, A. W., & Atkeson, C. G. (1995) · Machine Learning · doi:10.1023/a:1022656217772 · Cited by 259
Robot motor skill coordination with EM-based Reinforcement Learning
Kormushev, P., Calinon, S., & Caldwell, D. G. (2010) · 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2010.5649089 · Cited by 258
Self-citationLearning motor primitives for robotics
Kober, J., & Peters, J. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152577 · Cited by 220
Skill learning and task outcome prediction for manipulation
Pastor, P., Kalakrishnan, M., Chitta, S., Theodorou, E., & Schaal, S. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980200 · Cited by 202
Keyframe-based Learning from Demonstration: Method and Evaluation
Akgun, B., Cakmak, M., Jiang, K., & Thomaz, A. L. (2012) · International Journal of Social Robotics · doi:10.1007/s12369-012-0160-0 · Cited by 197
Path Integral Policy Improvement with Covariance Matrix AdaptationPreprint
Stulp, F., & Sigaud, O. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.4621 · Cited by 192
Simplified and effective motor control based on muscle synergies to exploit musculoskeletal dynamics
Berniker, M., Jarc, A., Bizzi, E., & Tresch, M. C. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0901512106 · Cited by 178
Trajectory formation for imitation with nonlinear dynamical systems
Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2002) · Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180) · doi:10.1109/iros.2001.976259 · Cited by 175
Reinforcement learning for imitating constrained reaching movements
Guenter, F., Hersch, M., Calinon, S., & Billard, A. (2007) · Advanced Robotics · doi:10.1163/156855307782148550 · Cited by 166
Self-citationCalinon, S., Sardellitti, I., & Caldwell, D. G. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5648931 · Cited by 157
Self-citationStatistical dynamical systems for skills acquisition in humanoids
Calinon, S., Li, Z., Alizadeh, T., Tsagarakis, N. G., & Caldwell, D. G. (2012) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2012.6651539 · Cited by 154
Self-citationSkill Acquisition in Sport: Research, Theory and Practice
(2004) · Routledge eBooks · doi:10.4324/9780203646564 · Cited by 150
(2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511489808 · Cited by 139
Apprenticeship learning for helicopter control
Coates, A., Abbeel, P., & Ng, A. Y. (2009) · Communications of the ACM · doi:10.1145/1538788.1538812 · Cited by 110
Nehaniv, C. L., & Dautenhahn, K. (2009) · Medical Entomology and Zoology · Cited by 105
Upper-body kinesthetic teaching of a free-standing humanoid robot
Kormushev, P. S., Nenchev, D. N., Calinon, S., & Caldwell, D. G. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979537 · Cited by 81
Self-citationMorimoto, J., & Atkeson, C. G. (2007) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2007.380654 · Cited by 79
Learning the skill of archery by a humanoid robot iCub
Kormushev, P. S., Calinon, S., Saegusa, R., & Giorgio Metta, B. (2010) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2010.5686841 · Cited by 72
Self-citationReinforcement learning of full-body humanoid motor skills
Stulp, F., Buchli, J., Theodorou, E. A., & Stefan Schaal, A. (2010) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2010.5686320 · Cited by 54
Reinforcement learning with via-point representation
Miyamoto, H., Morimoto, J., Doya, K., & Kawato, M. (2004) · Neural Networks · doi:10.1016/j.neunet.2003.11.004 · Cited by 37
Adaptive-resolution reinforcement learning with polynomial exploration in deterministic domains
Bernstein, A., & Shimkin, N. (2010) · Machine Learning · doi:10.1007/s10994-010-5186-7 · Cited by 33
Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning
Peters, J., & Schaal, S. (2007) · Cited by 27
Learning at the level of synergies for a robot weightlifter
Rosenstein, M. T., Barto, A. G., & van Emmerik, R. E. A. (2006) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2006.03.002 · Cited by 15
The evolution of coordination during skill acquisition: The dynamical systems approach
Huys, R., Daffertshofer, A., & Beek, P. J. (2004) · Data Archiving and Networked Services (DANS) · Cited by 15
Reinforcement Learning for Motor Primitives
Kober, J. (2008) · Max Planck Institute for Plasma Physics · Cited by 10
Challenges for the policy representation when applying reinforcement learning in robotics
Kormushev, P. S., Calinon, S., Caldwell, D. G., & Uğurlu, B. (2012) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252758 · Cited by 9
Self-citationA reinforcement learning scheme for acquisition of via-point representation of human motion
Wada, Y., & Sumita, K. (2005) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2004.1380089 · Cited by 8
Cited by
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
A Gentle Introduction to Reinforcement Learning and its Application in Different Fields
Naeem, M., Rizvi, S. T. H., & Coronato, A. (2020) · IEEE Access · doi:10.1109/access.2020.3038605 · Cited by 257
Target localization using multi-agent deep reinforcement learning with proximal policy optimization
Alagha, A., Singh, S., Mizouni, R., Bentahar, J., & Otrok, H. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.06.015 · Cited by 72
Huang, S. H., Pan, J., Mulcaire, G., & Abbeel, P. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353475 · Cited by 39
Robot Online Learning Through Digital Twin Experiments: A Weightlifting Project
Verner, I., Cuperman, D., Fang, A., Reitman, M., Romm, T., & Balikin, G. (2018) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-319-64352-6_29 · Cited by 34
Metadata
- Authors
- Kormushev, P., Calinon, S., & Caldwell, D.
- Venue
- Robotics
- Publisher
- MDPI AG
- Published
- 2013-07-05
- Volume
- 2
- Issue
- 3
- Pages
- 122-148
- ISSN
- 2218-6581 (ISSN-L)
- OpenAlex
- W2124267516
- Cited by
- 258
- References
- 43
- FWCI
- 11.41
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.