Sim-to-real: Six-legged Robot Control with Deep Reinforcement Learning and Curriculum Learning
Abstract
Six-Iegged robots have higher stability and balance, which helps them face more complex terrain conditions, such as sand, swamp, mine and so forth. Therefore, it is necessary to study the gait planning of six-legged robot to adapt to complex terrain. In order to control six-legged robots to adapt to different terrains, we adopt the method of deep reinforcement learning (DRL) to plan the gait of six-legged robots. The main idea is training the robot through Actor-Critic network with proximal policy optimization (PPO), in which outputs are step length, step height and orientation of the robot. This is an end-to-end approach, which tries to make the robot learn by itself and finally achieve its safe arrival to the target point through complex terrains. In order to train a good model for our robots, simplified environment is adopted to accelerate the training process. We also use curriculum learning to speed up and optimize the training. Then, we verify the reliability of the method in simulation platform and finally transfer the learned model to real robot. Our experiment shows the effectiveness of deep reinforcement learning for locomotion of six-legged robots, the acceleration of the training process by means of curriculum learning, and the improvement of the training effect.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1× · see citing works
“The curriculum learning was firstly proposed by Bengio in 2009[13].”
BackgroundDomain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1× · see citing works
“Previous studies on sim-to-real have been carried out [20,21,22].”
BackgroundHybrid Position/Force Control of Manipulators
Raibert, M. H., & Craig, J. J. (1981) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3139652 · Cited by 3,023 · see citing works
Learning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482 · see citing works
Capture Point: A Step toward Humanoid Push Recovery
Pratt, J. E., Carff, J., Drakunov, S., et al. (2006) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2006.321385 · Cited by 1,132 · see citing works
peng (2017) · doi:10.1145/3072959.3073602 · Cited by 502 · see citing works
Active learning of inverse models with intrinsically motivated goal exploration in robots
Baranès, A., & Oudeyer, P. (2012) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 466 · see citing works
Design principles for highly efficient quadrupeds and implementation on the MIT Cheetah robot
Seok, S., Wang, A. Z., Chuah, M. Y., Otten, D. M., Lang, J. H., & Kim, S. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6631038 · Cited by 457 · see citing works
Limit Cycles in a Passive Compass Gait Biped and Passivity-Mimicking Control Laws
Goswami, A., Espiau, B., & Keramane, A. (1997) · Autonomous Robots · doi:10.1023/a:1008844026298 · Cited by 431 · see citing works
Dynamic Locomotion Through Online Nonlinear Motion Optimization for Quadrupedal Robots
Bellicoso, C. D., Jenelten, F., Gehring, C., & Hutter, M. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2794620 · Cited by 206 · see citing works
Fast, robust quadruped locomotion over challenging terrain
Kalakrishnan, M., Buchli, J., Pástor, P., Mistry, M., & Schaal, S. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509805 · Cited by 168 · see citing works
Trajectory Optimization Through Contacts and Automatic Gait Discovery for Quadrupeds
Neunert, M., Farshidian, F., Winkler, A., et al. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2665685 · Cited by 132 · see citing works
Spring loaded inverted pendulum running: A plant model.
Schwind, W. J., & Koditschek, D. E. (1998) · Deep Blue (University of Michigan) · Cited by 117 · see citing works
Learning and Transfer of Modulated Locomotor ControllersPreprint
Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05182 · Cited by 113 · see citing works
Accuracy-based Curriculum Learning in Deep Reinforcement LearningPreprint
Fournier, P., Sigaud, O., Chétouani, M., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09614 · Cited by 30 · see citing works
Curriculum Learning in Reinforcement Learning
Narvekar, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/757 · Cited by 30 · see citing works
Zhang, F., Leitner, J., Milford, M. J., & Corke, P. (2017) · arXiv (Cornell University) · Cited by 21 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · see citing works
- Authors
- Qin, B., Gao, Y., & Bai, Y.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Venue
- 2019 4th International Conference on Robotics and Automation Engineering (ICRAE)
- Publisher
- IEEE
- Published
- 2019-11
- Pages
- 1-5
- OpenAlex
- W3013699912
- Cited by
- 18
- References
- 18
- Citations by year
- 2026: 2 · 2024: 2 · 2023: 5 · 2022: 4 · 2021: 3 · 2020: 2
- FWCI
- 0.86
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- International Conference on Robotics and Automation Engineering (ICRAE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
- Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
- Keywords
- hexapod robot, hexapod locomotion, gait planning, deep reinforcement learning, proximal policy optimization, curriculum learning, sim-to-real transfer, complex terrain, actor-critic networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
