The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Policy gradient reinforcement learning for fast quadrupedal locomotion
Abstract
This paper presents a machine learning approach to optimizing a quadrupedal trot gait for forward speed. Given a parameterized walk designed for a specific robot, we propose using a form of policy gradient reinforcement learning to automatically search the set of possible parameters with the goal of finding the fastest possible walk. We implement and test our approach on a commercially available quadrupedal robot platform, namely the Sony Aibo robot. After about three hours of learning, all on the physical robots and with no human intervention other than to change the batteries, the robots achieved a gait faster than any previously known gait known for the Aibo, significantly outperforming a variety of existing hand-coded and learned solutions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Numerical recipes in C: the art of scientific computing
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5638 · Cited by 17,933
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
(1999) · Cited by 5,147
Numerical Recipes--The Art of Scientific Computing.
Fritsch, F. N., Press, W. H., Flannery, B. P., Teukolsky, S. A., & Vetterling, W. T. (1988) · Mathematics of Computation · doi:10.2307/2007938 · Cited by 4,845
Numerical Recipes in C: The Art of Scientific Computing
Seiler, M. C., & Seiler, F. A. (1989) · Risk Analysis · doi:10.1111/j.1539-6924.1989.tb01007.x · Cited by 1,346
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Cited by 525
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Journal of Artificial Intelligence Research · doi:10.1613/jair.806 · Cited by 343
Autonomous Helicopter Flight via Reinforcement Learning
Kim, H. J., Jordan, M. I., Sastry, S., & Ng, A. Y. (2003) · neural information processing systems · Cited by 287
Autonomous helicopter control using reinforcement learning policy search methods
Bagnell, J. A., & Schneider, J. G. (2002) · Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164) · doi:10.1109/robot.2001.932842 · Cited by 276
Evolving robust gaits with AIBO
Hornby, G. S., Takamura, S., Yokono, J., Hanagata, O., Yamamoto, T., & Fujita, M. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2000.846489 · Cited by 145
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989385 · Cited by 1,424
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Imitation Learning: A Survey of Learning Methods
Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2018) · ACM Computing Surveys · doi:10.1145/3054912 · Cited by 1,060
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Robots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Natural actor–critic algorithms
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009) · Automatica · doi:10.1016/j.automatica.2009.07.008 · Cited by 559
Cognitive Mapping and Planning for Visual Navigation
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.769 · Cited by 500
Policy Gradient Methods for Robotics
Peters, J., & Schaal, S. (2006) · 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2006.282564 · Cited by 498
A Survey of Multi-Objective Sequential Decision-Making
Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3987 · Cited by 461
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Zhang, T., Kahn, G., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487175 · Cited by 452
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodLearning to Walk Via Deep Reinforcement Learning
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2019) · Robotics: Science and Systems XV · doi:10.15607/rss.2019.xv.011 · Cited by 430
Deep spatial autoencoders for visuomotor learning
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487173 · Cited by 430
Spröwitz, A., Tuleu, A., Vespignani, M., Ajallooeian, M., Badri, E., & Ijspeert, A. J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913489205 · Cited by 424
How Do You Learn to Walk? Thousands of Steps and Dozens of Falls per Day
Adolph, K. E., Cole, W. G., Komati, M., Garciaguirre, J. S., Badaly, D., Lingeman, J. M., Chan, G. L. Y., & Sotsky, R. B. (2012) · Psychological Science · doi:10.1177/0956797612446346 · Cited by 419
Tapus, A., Ţăpuş, C., & Matarić, M. J. (2008) · Intelligent Service Robotics · doi:10.1007/s11370-008-0017-4 · Cited by 402
“For applications like the one described in Kohl and Stone [22], where the authors try to maximize the speed of a robot, the reward function is clearly determined; it is the speed of the robot.”
MethodHigh speed obstacle avoidance using monocular vision and reinforcement learning
Michels, J., Saxena, A., & Ng, A. Y. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102426 · Cited by 394
Learning Agile Robotic Locomotion Skills by Imitating Animals
Bin Peng, X., Coumans, E., Zhang, T., Lee, T.-W., Tan, J., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.064 · Cited by 357
Development of quadruped walking robots: A review
Biswal, P., & Mohanty, P. K. (2021) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2020.11.005 · Cited by 354
IQA: Visual Question Answering in Interactive Environments
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., & Farhadi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00430 · Cited by 352
Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach
Desjardins, C., & Chaib-draa, B. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2157145 · Cited by 344
Williams, J. D., Asadi, K., & Zweig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1062 · Cited by 324
Learning, planning, and control for quadruped locomotion over challenging terrain
Kalakrishnan, M., Buchli, J., Pastor, P., Mistry, M., & Schaal, S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364910388677 · Cited by 294
Aliramezani, M., Koch, C. R., & Shahbakhti, M. (2022) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2021.100967 · Cited by 288
The Transferability Approach: Crossing the Reality Gap in Evolutionary Robotics
Koos, S., Mouret, J.-B., & Doncieux, S. (2013) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2185849 · Cited by 272
Virtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326
“Reinforcement learning has been applied to a wide variety of robotics related tasks, such as computer games [17], robot locomotion [8, 12], and autonomous driving [1, 25].”
BackgroundLearning contact-rich manipulation skills with guided policy search
Levine, S., Wagener, N., & Abbeel, P. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7138994 · Cited by 267
Stochastic policy gradient reinforcement learning on a simple 3D biped
Tedrake, R., Zhang, T. W., & Seung, H. S. (2005) · 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566) · doi:10.1109/iros.2004.1389841 · Cited by 264
Quality and Diversity Optimization: A Unifying Modular Framework
Cully, A., & Demiris, Y. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2704781 · Cited by 258
Model-Free reinforcement learning with continuous action in practice
Degris, T., Pilarski, P. M., & Sutton, R. S. (2012) · 2012 American Control Conference (ACC) · doi:10.1109/acc.2012.6315022 · Cited by 239
Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
Li, Z., Cheng, X., Peng, X. B., Abbeel, P., Levine, S., Berseth, G., & Sreenath, K. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9560769 · Cited by 219
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200
“When Deep Neural Networks (DNNs) are employed as approximators, the approach is referred to as Deep Reinforcement Learning (DRL); it allows to develop RL controllers with less manual feature engineering than classic tools (radial basis functions, tile coding, etc.) [4, 5, 6].”
MethodA Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160
TAMER: Training an Agent Manually via Evaluative Reinforcement
Bradley Knox, W., & Stone, P. (2008) · 2008 7th IEEE International Conference on Development and Learning · doi:10.1109/devlrn.2008.4640845 · Cited by 117
Self-citationAVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Smith, L., Dhawan, N., Zhang, M., Abbeel, P., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.024 · Cited by 102
Multi-Agent Reinforcement Learning for Traffic Signal Control: A Cooperative Approach
Kolat, M., Kővári, B., Bécsi, T., & Aradi, S. (2023) · Sustainability · doi:10.3390/su15043479 · Cited by 94
“Although reinforcement learning has reached success in the past [18–21], previous approaches faced several limitations such as dimensional, memory, computational or sample complexity issues [22].”
BackgroundData-Efficient Generalization of Robot Skills with Contextual Policy Search
Kupcsik, A., Deisenroth, M., Peters, J., & Neumann, G. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8546 · Cited by 92
From motor control to team play in simulated humanoid football
Liu, S., Lever, G., Wang, Z., Merel, J., Eslami, S. M. A., Hennes, D., Czarnecki, W. M., Tassa, Y., Omidshafiei, S., Abdolmaleki, A., et al. (2022) · Science Robotics · doi:10.1126/scirobotics.abo0235 · Cited by 88
A Survey of Exploration Methods in Reinforcement Learning
Amin, S., Gomrokchi, M., Satija, H., van Hoof, H., & Precup, D. (2021) · arXiv · doi:10.48550/arxiv.2109.00157 · Cited by 41
A multi-agent deep reinforcement learning based energy management for behind-the-meter resources
Wilk, P., Wang, N., & Li, J. (2022) · The Electricity Journal · doi:10.1016/j.tej.2022.107129 · Cited by 9
Metadata
- Authors
- Kohl, N., & Stone, P.
- Venue
- IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004
- Publisher
- IEEE
- Published
- 2004
- Pages
- 2619-2624 Vol.3
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2139053308
- Cited by
- 564
- References
- 13
- FWCI
- 42.20
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.