CoChat
Sign Up
Cited by 125Robotic Locomotion and ControlReinforcement Learning in RoboticsProsthetics and Rehabilitation Robotics

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

Open access
Bellegarda, Guillaume, Ijspeert, Auke·2022·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2022.3218167
Open this paper in CoChat — free accountOpen PDF

Abstract

In this letter, we present a method for integrating central pattern generators (CPGs), i.e. systems of coupled oscillators, into the deep reinforcement learning (DRL) framework to produce robust and omnidirectional quadruped locomotion. The agent learns to directly modulate the intrinsic oscillator setpoints (amplitude and frequency) and coordinate rhythmic behavior among different oscillators. This approach also allows the use of DRL to explore questions related to neuroscience, namely the role of descending pathways, interoscillator couplings, and sensory feedback in gait generation. We train our policies in simulation and perform a sim-to-real transfer to the Unitree A1 quadruped, where we observe robust behavior to disturbances unseen during training, most notably to a dynamically added 13.75 kg load representing 115% of the nominal quadruped mass. We test several different observation spaces based on proprioceptive sensing and show that our framework is deployable with no domain randomization and very little feedback, where along with the oscillator states, it is possible to provide only contact booleans in the observation space.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2022.3218167

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Thousands of transgenic fly lines have been generated that allow investigators to repeatedly target sparse sets of neurons for optogenetic activation and recording.

    “Additionally, thousands of transgenic fly lines have been generated that allow investigators to repeatedly target sparse sets of neurons 40 for optogenetic activation 41 and recording 42 .”

    From NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila (2024) · cites this paper as [41]

    The Related Work (p. 1) and Contribution (p. 2) discuss central pattern generators and robot locomotion, not transgenic fly lines. Searches for sparse neuronal targeting, optogenetic activation, and recording found no passage in our copy addressing the claim. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [1][1] A. J. Ijspeert, 'Central pattern generators for locomotion control in animals and robots: A review,' Neural Networks , vol. 21, no. 4, pp. 642-653, 2008, robotics and Neuroscience. -
  2. [2][2] Y. Fukuoka, H. Kimura, and A. H. Cohen, 'Adaptive dynamic walking of a quadruped robot on irregular terrain based on biological concepts,' The International Journal of Robotics Research , vol. 22, no. 3-4, pp. 187-202, 2003. -
  3. [3][3] D. Owaki, T. Kano, K. Nagasawa, A. Tero, and A. Ishiguro, 'Simple robot suggests physical interlimb communication is essential for quadruped walking,' Journal of The Royal Society Interface , vol. 10, no. 78, p. 20120669, 2013. -
  4. [4][4] D. Owaki and A. Ishiguro, 'A quadruped robot exhibiting spontaneous gait transitions from walking to trotting to galloping,' Scientific reports , vol. 7, no. 1, pp. 1-10, 2017. -
  5. [5][5] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, 'Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,' in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2018, pp. 1-9. -
  6. [6][6] D. Kim, J. Di Carlo, B. Katz, G. Bledt, and S. Kim, 'Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control,' arXiv preprint arXiv:1909.06586 , 2019. - · arXiv
  7. [7][7] M. Sombolestan, Y. Chen, and Q. Nguyen, 'Adaptive force-based control for legged robots,' in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021, pp. 7440-7447. -
  8. [8][8] C. D. Bellicoso, F. Jenelten, C. Gehring, and M. Hutter, 'Dynamic locomotion through online nonlinear motion optimization for quadrupedal robots,' IEEE Robotics and Automation Letters , vol. 3, no. 3, pp. 2261-2268, 2018. -
  9. [10][10] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, 'Learning agile and dynamic motor skills for legged robots,' Science Robotics , vol. 4, no. 26, 2019. -
  10. [11][11] A. Kumar, Z. Fu, D. Pathak, and J. Malik, 'Rma: Rapid motor adaptation for legged robots,' arXiv preprint arXiv:2107.04034 , 2021. - · arXiv
  11. [13][13] A. Spr¨ owitz, A. Tuleu, M. Vespignani, M. Ajallooeian, E. Badri, and A. J. Ijspeert, 'Towards dynamic trot gait locomotion: Design, control, and experiments with cheetah-cub, a compliant quadruped robot,' The International Journal of Robotics Research , vol. 32, no. 8, pp. 932-950, 2013. -
  12. [14][14] L. Righetti and A. J. Ijspeert, 'Pattern generators with sensory feedback for the control of quadruped locomotion,' in IEEE International Conference on Robotics and Automation , 2008, pp. 819-824. -
  13. [15][15] V. Barasuol, J. Buchli, C. Semini, M. Frigerio, E. R. De Pieri, and D. G. Caldwell, 'A reactive controller framework for quadrupedal locomotion on challenging terrain,' in 2013 IEEE International Conference on Robotics and Automation . IEEE, 2013, pp. 2554-2561. -
  14. [16][16] M. Ajallooeian, S. Pouya, A. Sproewitz, and A. J. Ijspeert, 'Central pattern generators augmented with virtual model control for quadruped rough terrain locomotion,' in 2013 IEEE International Conference on Robotics and Automation , 2013, pp. 3321-3328. -
  15. [17][17] M. Ajallooeian, S. Gay, A. Tuleu, A. Spr¨ owitz, and A. J. Ijspeert, 'Modular control of limit cycle locomotion over unperceived rough terrain,' in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2013, pp. 3390-3397. -
  16. [18][18] D. J. Hyun, S. Seok, J. Lee, and S. Kim, 'High speed trot-running: Implementation of a hierarchical controller using proprioceptive impedance control on the mit cheetah,' The International Journal of Robotics Research , vol. 33, no. 11, pp. 1417-1445, 2014. -
  17. [19][19] S. Gay, J. Santos-Victor, and A. Ijspeert, 'Learning robot gait stability using neural networks as sensory feedback function for central pattern generators,' in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2013, pp. 194-201. -
  18. [20][20] M. H. Raibert, Legged robots that balance . MIT press, 1986. -
  19. [22][22] Z. Fu, A. Kumar, J. Malik, and D. Pathak, 'Minimizing energy consumption leads to the emergence of gaits in legged robots,' in 5th Annual Conference on Robot Learning , 2021. -
  20. [23][23] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang, 'Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,' arXiv preprint arXiv:2107.03996 , 2021. - · arXiv
  21. [25][25] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, 'Learning to walk in minutes using massively parallel deep reinforcement learning,' in 5th Annual Conference on Robot Learning , 2021. -
  22. [26][26] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang, 'Visual-locomotion: Learning to walk on complex terrains with vision,' in 5th Annual Conference on Robot Learning , 2021. -
  23. [27][27] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. bae Kim, and P. Agrawal, 'Learning to jump from pixels,' in 5th Annual Conference on Robot Learning , 2021. -
  24. [28][28] S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath, 'Learning torque control for quadrupedal locomotion,' arXiv preprint arXiv:2203.05194 , 2022. - · arXiv
  25. [29][29] G. Bellegarda and K. Byl, 'Training in task space to speed up and guide reinforcement learning,' in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2019, pp. 2693-2699. -
  26. [30][30] G. Bellegarda and Q. Nguyen, 'Robust quadruped jumping via deep reinforcement learning,' arXiv preprint arXiv:2011.07089 , 2020. - · arXiv
  27. [31][31] G. Bellegarda and Q. Nguyen, 'Robust high-speed running for quadruped robots via deep reinforcement learning,' arXiv preprint arXiv:2103.06484 , 2021. - · arXiv
  28. [32][32] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke, 'Policies modulating trajectory generators,' in Conference on Robot Learning . PMLR, 2018, pp. 916-926. -
  29. [33][33] Y. Shao, Y. Jin, X. Liu, W. He, H. Wang, and W. Yang, 'Learning free gait transition for quadruped robots via phase-guided controller,' arXiv preprint arXiv:2201.00206 , 2022. - · arXiv
  30. [34][34] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots, 'Fast and efficient locomotion via learned gait transitions,' in 5th Annual Conference on Robot Learning , 2021. -
  31. [35][35] C. Li, R. Lowe, and T. Ziemke, 'Humanoids learning to walk: A natural cpg-actor-critic architecture,' Frontiers in Neurorobotics , vol. 7, p. 5, 2013. -
  32. [36][36] M. Kasaei, M. Abreu, N. Lau, A. Pereira, and L. P. Reis, 'A cpg-based agile and versatile locomotion framework using proximal symmetry loss,' arXiv preprint arXiv:2103.00928 , 2021. - · arXiv
  33. [37][37] H. Shi, B. Zhou, H. Zeng, F. Wang, Y. Dong, J. Li, K. Wang, H. Tian, and M. Q.-H. Meng, 'Reinforcement learning with evolutionary trajectory generator: A general approach for quadrupedal locomotion,' arXiv preprint arXiv:2109.06409 , 2021. - · arXiv
  34. [38][38] R. Thandiackal, K. Melo, L. Paez, J. Herault, T. Kano, K. Akiyama, F. Boyer, D. Ryczko, A. Ishiguro, and A. J. Ijspeert, 'Emergence of robust self-organized undulatory swimming based on local hydrodynamic force sensing,' Science Robotics , vol. 6, no. 57, 2021. -
  35. [39][39] R. S. Sutton and A. G. Barto, Reinforcement learning - an introduction , ser. Adaptive computation and machine learning. MIT Press, 1998. -
  36. [40][40] A. J. Ijspeert, A. Crespi, D. Ryczko, and J.-M. Cabelguen, 'From swimming to walking with a salamander robot driven by a spinal cord model,' Science , vol. 315, no. 5817, pp. 1416-1420, 2007. -
  37. [41][41] N. Hansen, 'The cma evolution strategy: A tutorial,' arXiv preprint arXiv:1604.00772 , 2016. - · arXiv
  38. [42][42] D. Owaki, M. Goda, S. Miyazawa, and A. Ishiguro, 'A minimal model describing hexapedal interlimb coordination: the tegotae-based approach,' Frontiers in neurorobotics , vol. 11, p. 29, 2017. -
  39. [43][43] O. Ekeberg and K. Pearson, 'Computer simulation of stepping in the hind legs of the cat: an examination of mechanisms regulating the stance-to-swing transition,' Journal of neurophysiology , vol. 94, no. 6, pp. 4256-4268, 2005. -
  40. [44][44] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa et al. , 'Isaac gym: High performance gpu-based physics simulation for robot learning,' arXiv preprint arXiv:2108.10470 , 2021. - · arXiv

Cited by

Sorted by citations

Authors
Bellegarda, G., & Ijspeert, A.
Affiliations
École Polytechnique Fédérale de Lausanne (CH)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-10
Volume
7
Issue
4
Pages
12547-12554
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2022.3218167
OpenAlex
W4309323738
Cited by
125
References
3
Citations by year
2026: 32 · 2025: 40 · 2024: 38 · 2023: 15
FWCI
6.76
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (197237)
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
Keywords
central pattern generator, coupled oscillators, quadrupedal locomotion, deep reinforcement learning, sim-to-real transfer, proprioception, sensory feedback, gait generation, descending pathways, load disturbance robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review