Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots
Open accessAbstract
Developing robust walking controllers for bipedal robots is a challenging endeavor. Traditional model-based locomotion controllers require simplifying assumptions and careful modelling; any small errors can result in unstable control. To address these challenges for bipedal locomotion, we present a model-free reinforcement learning framework for training robust locomotion policies in simulation, which can then be transferred to a real bipedal Cassie robot. To facilitate sim-to-real transfer, domain randomization is used to encourage the policies to learn behaviors that are robust across variations in system dynamics. The learned policies enable Cassie to perform a set of diverse and dynamic behaviors, while also being more robust than traditional controllers and prior learning-based methods that use residual control. We demonstrate this on versatile walking behaviors such as tracking a target walking velocity, walking height, and turning yaw. (Video1)
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Robots still struggle to solve high-level tasks such as locomotion, autonomous driving and dexterous manipulation, even though low-level controllers and motion planning algorithms are robust enough to drive joints and end-effectors to desired configurations.
“Even though low-level controllers and motion planning algorithms are robust enough to drive the joints and end-effectors to reach their desired configuration, robots still struggle to solve high-level tasks, such as locomotion [3], autonomous driving [4] and dexterous manipulation [5].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [3]
Cites
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · see citing works
Self-citationZERO-MOMENT POINT — THIRTY FIVE YEARS OF ITS LIFE
VUKOBRATOVIC, M. K., & Borovać, B. (2004) · International Journal of Humanoid Robotics · doi:10.1142/s0219843604000083 · Cited by 2,051 · see citing works
Learning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Self-citationLearning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Science Robotics · doi:10.1126/scirobotics.abc5986 · Cited by 1,156 · see citing works
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 946 · see citing works
Self-citationLearning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000448343 · Cited by 752 · see citing works
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 676 · see citing works
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618 · see citing works
Self-citationpeng (2017) · doi:10.1145/3072959.3073602 · Cited by 502 · see citing works
Residual Reinforcement Learning for Robot Control
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., Ojea, J. A., Solowjow, E., & Levine, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794127 · Cited by 480 · see citing works
Self-citationKoolen, T., de Boer, T., Rebula, J. R., Goswami, A., & Pratt, J. E. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912452673 · Cited by 477 · see citing works
Whole-body motion planning with centroidal dynamics and full kinematics
Dai, H., Valenzuela, A., & Tedrake, R. (2014) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2014.7041375 · Cited by 437 · see citing works
Learning Agile Robotic Locomotion Skills by Imitating Animals
Bin Peng, X., Coumans, E., Zhang, T., Lee, T.-W., Tan, J., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.064 · Cited by 357 · see citing works
Self-citationPratt, J. E., Koolen, T., de Boer, T., et al. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912452762 · Cited by 263 · see citing works
Feedback Control of a Cassie Bipedal Robot: Walking, Standing, and Riding a Segway
Gong, Y., Hartley, R., Da, X., Hereid, A., Harib, O., Huang, J.-K., & Grizzle, J. (2019) · 2019 American Control Conference (ACC) · doi:10.23919/acc.2019.8814833 · Cited by 221 · cited 2× · see citing works
“Model-based methods require careful system identification to bridge the reality gap [16], [17].”
BackgroundMethodFeedback Control For Cassie With Deep Reinforcement Learning
Xie, Z., Berseth, G., Clary, P., HURST, J. W., & van de Panne, M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593722 · Cited by 196 · see citing works
Self-citationAn efficiently solvable quadratic program for stabilizing dynamic locomotion
Kuindersma, S., Permenter, F., & Tedrake, R. L. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 180 · see citing works
An efficiently solvable quadratic program for stabilizing dynamic locomotion
Kuindersma, S., Permenter, F., & Tedrake, R. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907230 · Cited by 173 · see citing works
3D Bipedal Robotic Walking: Models, Feedback Control, and Open Problems
Grizzle, J. W., Chevallereau, C., Ames, A. D., et al. (2010) · IFAC Proceedings Volumes · doi:10.3182/20100901-3-it-2016.00302 · Cited by 131 · see citing works
Locomotion skills for simulated quadrupeds
Coros, S., Karpathy, A., Jones, B., Revéret, L., & van de Panne, M. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964954 · Cited by 125 · see citing works
Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real
Xie, Z., Clary, P., Dao, J., Morais, P., HURST, J. W., & van de Panne, M. (2019) · Conference on Robot Learning · Cited by 114 · see citing works
Locomotion skills for simulated quadrupeds
Coros, S., Karpathy, A., Jones, B., et al. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964954 · Cited by 114 · see citing works
Sim-to-Real Transfer for Biped Locomotion
Yu, W., Kumar, V. C. V., Turk, G., & Liu, C. K. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968053 · Cited by 105 · see citing works
Imitation learning for locomotion and manipulation
Ratliff, N., Bagnell, J. A., & Srinivasa, S. S. (2007) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2007.4813899 · Cited by 104 · see citing works
From 2D Design of Underactuated Bipedal Gaits to 3D Implementation: Walking With Speed Tracking
Da, X., Harib, O., Hartley, R., et al. (2016) · IEEE Access · doi:10.1109/access.2016.2582731 · Cited by 101 · see citing works
Summary of Team IHMC's virtual robotics challenge entry
Koolen, T., Smith, J., Thomas, G. C., et al. (2013) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2013.7029992 · Cited by 87 · see citing works
3D walking based on online optimization
Feng, S., Xinjilefu, X., Huang, W., et al. (2013) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2013.7029950 · Cited by 86 · see citing works
Learning Fast Adaptation With Meta Strategy Optimization
Yu, W., Tan, J., Bai, Y., Coumans, E., & Ha, S. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974685 · Cited by 86 · see citing works
Gaits and gait transitions for legged robots
Haynes, G. C., & Rizzi, A. A. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2006.1641859 · Cited by 85 · see citing works
Morimoto, J., & Atkeson, C. G. (2007) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2007.380654 · Cited by 79 · see citing works
Learning Memory-Based Control for Human-Scale Bipedal Locomotion
Siekmann, J., Valluri, S., Dao, J., Bermillo, L., Duan, H., Fern, A., & HURST, J. W. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.031 · Cited by 78 · see citing works
Dynamic Walking on Randomly-Varying Discrete Terrain with One-step Preview
Nguyen, Q., Agrawal, A., Da, X., et al. (2017) · doi:10.15607/rss.2017.xiii.072 · Cited by 51 · see citing works
Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
Li, T., Lambert, N., Calandra, R., Meier, F., & Rai, A. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196642 · Cited by 44 · see citing works
Hereid, A., Harib, O., Hartley, R., et al. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968056 · Cited by 43 · see citing works
Castillo, G. A., Weng, B., Zhang, W., & Hereid, A. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197175 · Cited by 43 · see citing works
Coupling Reduced Order Models via Feedback Control for 3D Underactuated Bipedal Robotic Walking
Xiong, X., & Ames, A. D. (2018) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2018.8625066 · Cited by 40 · see citing works
Animated Cassie: A Dynamic Relatable Robotic Character
Li, Z., Cummings, C., & Sreenath, K. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340894 · Cited by 25 · see citing works
Self-citationImitation Learning for Locomotion and Manipulation
Ratliff, N., Bagnell, J. A., & Srinivasa, S. S. (2007) · doi:10.21236/ada528601 · Cited by 25 · see citing works
Castillo, G. A., Weng, B., Stewart, T. C., Zhang, W., & Hereid, A. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341569 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Embodied intelligence via learning and evolution
Gupta, A., Savarese, S., Ganguli, S., & Fei-Fei, L. (2021) · Nature Communications · doi:10.1038/s41467-021-25874-z · Cited by 352 · see citing works
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“…[46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and notation above provided, the designer should ensure that (a) the…”
BackgroundKadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Even though low-level controllers and motion planning algorithms are robust enough to drive the joints and end-effectors to reach their desired con-figuration, robots still struggle to solve high-level tasks, such as locomotion [3], autonomous driving [4] and dexterous manipulation [5].”
BackgroundKim, D., Lee, H., Cha, J., & Park, J. (2025) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2024.3505784 · Cited by 12 · see citing works
Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning
Kim, S., Sorokin, M., Lee, J., & Ha, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.021 · Cited by 9 · see citing works
- Authors
- Li, Z., Cheng, X., Peng, X. B., Abbeel, P., Levine, S., Berseth, G., & Sreenath, K.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2021 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2021-05-30
- Pages
- 2811-2817
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3206620955
- Cited by
- 219
- References
- 42
- Citations by year
- 2026: 23 · 2025: 53 · 2024: 56 · 2023: 53 · 2022: 27 · 2021: 7
- FWCI
- 67.71
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
- Keywords
- bipedalism, Cassie robot, reinforcement learning, sim-to-real transfer, domain randomization, walking control, model-free control, speed tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
