CoChat
Sign Up
PreprintCited by 84Robotic Locomotion and ControlReinforcement Learning in RoboticsRobotics and Sensor-Based Localization

Legged Locomotion in Challenging Terrains using Egocentric Vision

Open access
Agarwal, Ananye, Kumar, Ashish, Malik, Jitendra, Pathak, Deepak·2022·arXiv
DOI 10.48550/arxiv.2211.07638arXiv 2211.07638
Open this paper in CoChat — free accountOpen PDF

Abstract

Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not seen elsewhere. The egocentric camera requires the policy to remember past information to estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases - first, we train a policy using reinforcement learning with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being robust to perturbations like pushes, slippery surfaces, and rocky terrain. Videos are at https://vision-locomotion.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.07638

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 62

  1. [1][1] J. M. Loomis, J. A. Da Silva, N. Fujita, and S. S. Fukusima. Visual space perception and visually directed action. Journal of experimental psychology: Human Perception and Performance , 18 (4):906, 1992. -
  2. [4][4] J. S. Matthis and B. R. Fajen. Visual control of foot placement when walking over complex terrain. Journal of experimental psychology: human perception and performance , 40(1):106, 2014. -
  3. [5][5] A. E. Patla. Understanding the roles of vision in the control of human locomotion. Gait & posture , 5(1):54-69, 1997. -
  4. [6][6] A. A. Mohagheghi, R. Moraes, and A. E. Patla. The effects of distant and on-line visual information on the control of approach phase and step over an obstacle during locomotion. Experimental brain research , 155(4):459-468, 2004. -
  5. [9][9] D. Kim, D. Carballo, J. Di Carlo, B. Katz, G. Bledt, B. Lim, and S. Kim. Vision aided dynamic exploration of unstructured terrain with a small-scale quadruped robot. In ICRA , 2020. -
  6. [11][11] P. Fankhauser, M. Bloesch, C. Gehring, M. Hutter, and R. Siegwart. Robot-centric elevation mapping with uncertainty estimates. In Mobile Service Robotics , pages 433-440. World Scientific, 2014. -
  7. [12][12] T. Miki, L. Wellhausen, R. Grandia, F. Jenelten, T. Homberger, and M. Hutter. Elevation mapping for locomotion and navigation using gpu. arXiv preprint arXiv:2204.12876 , 2022. - · arXiv
  8. [14][14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06347 , 2017. - · arXiv
  9. [15][15] P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10):1550-1560, 1990. -
  10. [16][16] Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak. Coupling vision and proprioception for navigation of legged robots. arXiv preprint arXiv:2112.02094 , 2021. - · arXiv
  11. [17][17] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning , pages 91-100. PMLR, 2022. -
  12. [18][18] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , 2011. -
  13. [20][20] M. H. Raibert. Hopping in legged systems-modeling and simulation for the two-dimensional one-legged case. IEEE Transactions on Systems, Man, and Cybernetics , 1984. -
  14. [21][21] H. Geyer, A. Seyfarth, and R. Blickhan. Positive force feedback in bouncing gaits? Proceedings of the Royal Society of London. Series B: Biological Sciences , 2003. -
  15. [22][22] K. Yin, K. Loken, and M. Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics , 2007. -
  16. [24][24] A. M. Johnson, T. Libby, E. Chang-Siu, M. Tomizuka, R. J. Full, and D. E. Koditschek. Tail assisted dynamic self righting. In Adaptive Mobile Robotics . World Scientific, 2012. -
  17. [26][26] A. D. Ames, K. Galloway, K. Sreenath, and J. W. Grizzle. Rapidly exponentially stabilizing control lyapunov functions and hybrid zero dynamics. IEEE Transactions on Automatic Control , 2014. -
  18. [28][28] M. Barragan, N. Flowers, and A. M. Johnson. MiniRHex: A small, open-source, fully programmable walking hexapod. In RSS Workshop , 2018. -
  19. [29][29] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. In IROS , 2018. -
  20. [30][30] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, et al. Anymal-a highly mobile and dynamic quadrupedal robot. In IROS , 2016. -
  21. [31][31] C. S. Imai, M. Zhang, Y. Zhang, M. Kierebinski, R. Yang, Y. Qin, and X. Wang. Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. arXiv:2109.14549 , 2021. - · arXiv
  22. [32][32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR , 2016. -
  23. [33][33] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML , 2016. -
  24. [34][34] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML , 2018. -
  25. [36][36] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS , 2017. -
  26. [37][37] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA , 2018. -
  27. [38][38] Z. Xie, X. Da, M. van de Panne, B. Babich, and A. Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. In ICRA , 2021. -
  28. [39][39] O. Nachum, M. Ahn, H. Ponte, S. S. Gu, and V. Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. In CoRL , 2020. -
  29. [41][41] J. Hanna and P. Stone. Grounded action transformation for robot learning in simulation. In AAAI , 2017. -
  30. [43][43] W. Yu, C. K. Liu, and G. Turk. Policy transfer with strategy optimization. In ICLR , 2018. -
  31. [45][45] W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. In ICLR , 2019. -
  32. [46][46] W. Yu, V. C. V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In IROS , 2019. -
  33. [48][48] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and J. Tan. Rapidly adaptable legged robots via evolutionary meta-learning. In IROS , 2020. -
  34. [49][49] I. Clavera, A. Nagabandi, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In ICLR , 2019. -
  35. [50][50] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emergence of gaits in legged robots. In CoRL , 2021. -
  36. [51][51] L. Smith, J. C. Kew, X. B. Peng, S. Ha, J. Tan, and S. Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In ICRA , 2022. -
  37. [52][52] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. In ICLR , 2022. -
  38. [53][53] P. Fankhauser, M. Bloesch, and M. Hutter. Probabilistic terrain mapping for mobile robots with uncertain localization. IEEE Robotics and Automation Letters , 3(4):3019-3026, 2018. -
  39. [55][55] I.-S. Kweon, M. Hebert, E. Krotkov, and T. Kanade. Terrain mapping for a roving planetary explorer. In IEEE International Conference on Robotics and Automation , pages 997-1002. IEEE, 1989. -
  40. [56][56] I.-S. Kweon and T. Kanade. High-resolution terrain map from multiple sensor data. IEEE Transactions on Pattern Analysis and Machine Intelligence , 14(2):278-292, 1992. -
  41. [57][57] A. Kleiner and C. Dornhege. Real-time localization and elevation mapping within urban search and rescue scenarios. Journal of Field Robotics , 24(8-9):723-745, 2007. -
  42. [59][59] M. Wermelinger, P. Fankhauser, R. Diethelm, P. Kr¨ usi, R. Siegwart, and M. Hutter. Navigation planning for legged robots in challenging terrain. In IROS , 2016. -
  43. [60][60] A. Chilian and H. Hirschm¨ uller. Stereo camera based navigation of mobile robots on rough terrain. In IROS , 2009. -
  44. [61][61] C. Mastalli, I. Havoutis, A. W. Winkler, D. G. Caldwell, and C. Semini. On-line and on-board planning and perception for quadrupedal locomotion. In 2015 IEEE International Conference on Technologies for Practical Robot Applications , 2015. -
  45. [62][62] P. Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter. Robust rough-terrain locomotion with a quadrupedal robot. In ICRA , 2018. -
  46. [63][63] A. Agrawal, S. Chen, A. Rai, and K. Sreenath. Vision-aided dynamic quadrupedal locomotion on discrete terrain using motion libraries. arXiv preprint arXiv:2110.00891 , 2021. - · arXiv
  47. [64][64] J. Z. Kolter, M. P. Rodgers, and A. Y. Ng. A control architecture for quadruped locomotion over rough terrain. In ICRA , 2008. -
  48. [65][65] M. Kalakrishnan, J. Buchli, P. Pastor, and S. Schaal. Learning locomotion over rough terrain using terrain templates. In IROS , 2009. -
  49. [66][66] L. Wellhausen and M. Hutter. Rough terrain navigation for legged robots using reachability planning and template learning. In IROS , 2021. -
  50. [67][67] C. Mastalli, M. Focchi, I. Havoutis, A. Radulescu, S. Calinon, J. Buchli, D. G. Caldwell, and C. Semini. Trajectory and foothold optimization using low-dimensional models for rough terrain locomotion. In ICRA , 2017. -
  51. [69][69] B. Yang, L. Wellhausen, T. Miki, M. Liu, and M. Hutter. Real-time optimal navigation planning using learned motion costs. In ICRA , 2021. -
  52. [72][72] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In International Conference on Robotics and Automation (ICRA) , 2021. -
  53. [73][73] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters , 5 (2):3699-3706, 2020. -
  54. [74][74] X. B. Peng, G. Berseth, and M. Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG) , 35(4):1-12, 2016. -
  55. [75][75] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG) , 36(4): 1-13, 2017. -
  56. [76][76] Z. Xie, H. Y. Ling, N. H. Kim, and M. van de Panne. Allsteps: Curriculum-driven learning of stepping stone skills. In Computer Graphics Forum . Wiley Online Library, 2020. -
  57. [77][77] D. Belter, P. Łabcki, and P. Skrzypczy´ nski. Estimating terrain elevation maps from sparse and uncertain multi-sensor data. In 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO) , pages 715-722, 2012. doi:10.1109/ROBIO.2012.6491052. - · DOI
  58. [78][78] D. Jain, A. Iscen, and K. Caluwaerts. From pixels to legs: Hierarchical learning of quadruped locomotion. arXiv preprint arXiv:2011.11722 , 2020. - · arXiv
  59. [79][79] A. Escontrela, G. Yu, P. Xu, A. Iscen, and J. Tan. Zero-shot terrain generalization for visual locomotion policies. arXiv preprint arXiv:2011.05513 , 2020. - · arXiv
  60. [80][80] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual-locomotion: Learning to walk on complex terrains with vision. In 5th Annual Conference on Robot Learning , 2021. -
  61. [81][81] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal. Learning to jump from pixels. arXiv preprint arXiv:2110.15344 , 2021. - · arXiv
  62. [82][82] S. P. Singh and R. C. Yee. An upper bound on the loss from approximate optimal-value functions. Machine Learning , 16(3):227-233, 1994. -

Cited by

Sorted by citations

Related

Authors
Agarwal, A., Kumar, A., Malik, J., & Pathak, D.
Affiliations
  • Carnegie Mellon University (US)
  • University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.07638
arXiv
2211.07638
OpenAlex
W4309137642
Cited by
82
References
0
Citations by year
2025: 12 · 2024: 50 · 2023: 20
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Defense Advanced Research Projects Agency
  • Office of Naval Research (N00014-22-1-2096, N00014-22, N00014)
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
Keywords
legged locomotion, egocentric vision, quadruped robot, terrain traversal, depth camera, reinforcement learning, knowledge distillation, supervised learning, gait pattern, real-time control, sim-to-real transfer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review