CoChat
Sign Up
Cited by 50Robot Manipulation and LearningReinforcement Learning in RoboticsAI-based Problem Solving and Planning

Bottom-Up Skill Discovery From Unsegmented Demonstrations for Long-Horizon Robot Manipulation

Open access
Zhu, Yifeng, Stone, Peter, Zhu, Yuke·2022·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2022.3146589
Open this paper in CoChat — free accountOpen PDF

Abstract

We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations and use these skills to synthesize prolonged robot behaviors. Our method starts with constructing a hierarchical task structure from each demonstration through agglomerative clustering. From the task structures of multi-task demonstrations, we identify skills based on the recurring patterns and train goal-conditioned sensorimotor policies with hierarchical imitation learning. Finally, we train a meta controller to compose these skills to solve long-horizon manipulation tasks. The entire model can be trained on a small set of human demonstrations collected within 30 minutes without further annotations, making it amendable to real-world deployment. We systematically evaluated our method in simulation environments and on a real robot. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks. Furthermore, skills discovered from multi-task demonstrations boost the average task success by 8% compared to those discovered from individual tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2022.3146589

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • On Hammer Cleanup, BC-RNN achieves 59.3% on 10 source demos, 100.0% on generated 1000 D0 demos, and 62.7% on D1 variant, while BUDS achieved 68.6% on D0.

    “On Hammer Cleanup, BUDS achieved 68.6% ( D 0 ), while BC-RNN achieves 59.3% on our 10 source demos, 100.0% on our generated 1000 D 0 demos, and 62.7% on the D 1 variant where both the hammer and drawer move substantially.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [53]

  • On Kitchen, BC-RNN achieves 54.7% on 10 source demos, 100.0% on generated D0 data, and 76.0% on D1 variant, while BUDS achieved 72.0% on D0.

    “On Kitchen, BUDS achieved 72.0% ( D 0 ), while BC-RNN achieves 54.7% on our 10 source demos, 100.0% on our generated D 0 data, and 76.0% on the D 1 variant, where all objects move in wider regions.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [53]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1][1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, 'Hindsight experience replay,' in NIPS , 2017. -
  2. [2][2] A. Bagaria and G. Konidaris, 'Option discovery using deep skill chaining,' in ICLR , 2019. -
  3. [4][4] R. A. Brooks, 'Intelligence without representation,' Artificial Intelligence , vol. 47, no. 1-3, pp. 139-159, 1991. -
  4. [5][5] V. Chu, R. A. Gutierrez, S. Chernova, A. L. Thomaz, V. Chu, R. A. Gutierrez, S. Chernova, and A. L. Thomaz, 'Real-time multisensory affordance-based control for adaptive object manipulation,' in ICRA , IEEE, 2019. -
  5. [6][6] T. G. Dietterich, 'The MAXQ method for hierarchical reinforcement learning.,' in ICML , vol. 98, 1998. -
  6. [7][7] Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, 'One-shot imitation learning,' arXiv:1703.07326 , 2017. - · arXiv
  7. [8][8] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, 'Diversity is all you need: Learning skills without a reward function,' arXiv:1802.06070 , 2018. - · arXiv
  8. [9][9] A. Farag, L. Lu, H. R. Roth, J. Liu, E. Turkbey, and R. M. Summers, 'A bottomup approach for pancreas segmentation using cascaded superpixels and (deep) image patch labeling,' IEEE Transactions on Image Processing , vol. 26, no. 1, pp. 386-399, 2016. -
  9. [10][10] R. Fox, S. Krishnan, I. Stoica, and K. Goldberg, 'Multi-level discovery of deep options,' arXiv:1703.08294 , 2017. - · arXiv
  10. [11][11] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´ erez, 'Integrated task and motion planning,' Annual review of control, robotics, and autonomous systems , vol. 4, pp. 265-293, 2021. -
  11. [12][12] J. J. Gibson and L. Carmichael, The senses considered as perceptual systems , 1. Houghton Mifflin Boston, 1966, vol. 2. -
  12. [13][13] K. Gregor, D. J. Rezende, and D. Wierstra, 'Variational intrinsic control,' arXiv:1611.07507 , 2016. - · arXiv
  13. [14][14] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman, 'Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,' in CoRL , 2020, pp. 1025-1037. -
  14. [15][15] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller, 'Learning an embedding space for transferable robot skills,' in ICLR , 2018. -
  15. [17][17] G. E. Hovland, P. Sikka, and B. J. McCarragher, 'Skill acquisition from human demonstration using a hidden markov model,' in ICRA , vol. 3, 1996. -
  16. [18][18] L. P. Kaelbling and T. Lozano-P´ erez, 'Hierarchical task and motion planning in the now,' in ICRA , IEEE, 2011. -
  17. [19][19] --, 'Integrated task and motion planning in belief space,' IJRR , vol. 32, no. 910, pp. 1194-1227, 2013. -
  18. [20][20] O. Khatib, 'A unified approach for motion and force control of robot manipulators: The operational space formulation,' IEEE Journal on Robotics and Automation , vol. 3, no. 1, pp. 43-53, 1987. -
  19. [21][21] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' arXiv:1312.6114 , 2013. - · arXiv
  20. [22][22] T. Kipf, Y. Li, H. Dai, V. Zambaldi, E. Grefenstette, P. Kohli, and P. Battaglia, 'Compositional imitation learning: Explaining and executing one task at a time,' arXiv:1812.01483 , 2018. - · arXiv
  21. [23][23] G. Konidaris and A. Barto, 'Skill discovery in continuous reinforcement learning domains using skill chaining,' NIPS , vol. 22, pp. 1015-1023, 2009. -
  22. [24][24] G. D. Konidaris, S. Kuindersma, A. G. Barto, and R. A. Grupen, 'Constructing skill trees for reinforcement learning agents from demonstration trajectories.,' in NIPS , 2010. -
  23. [26][26] V. C. Kumar, S. Ha, and C. K. Liu, 'Expanding motor skills using relay networks,' in CoRL , 2018. -
  24. [27][27] T. Lan, Y. Zhu, A. R. Zamir, and S. Savarese, 'Action recognition by hierarchical mid-level action elements,' in ICCV , 2015, pp. 4552-4560. -
  25. [28][28] H. Law and J. Deng, 'Cornernet: Detecting objects as paired keypoints,' in ECCV , 2018. -
  26. [29][29] H. Le, N. Jiang, A. Agarwal, M. Dudı´ ek, Y. Yue, and H. Daum´ e, 'Hierarchical imitation and reinforcement learning,' in ICML , 2018, pp. 2917-2926. -
  27. [30][30] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, 'Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,' IEEE Transactions on Robotics , 2020. -
  28. [31][31] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox, 'Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data,' in ICRA , IEEE, 2020. -
  29. [35][35] S. Niekum, S. Osentoski, C. G. Atkeson, and A. G. Barto, 'Online bayesian changepoint detection for articulated motion models,' in ICRA , IEEE, 2015. -
  30. [36][36] S. Niekum, S. Osentoski, G. Konidaris, and A. G. Barto, 'Learning and generalization of complex tasks from unstructured demonstrations,' in IROS , IEEE, 2012. -
  31. [37][37] S. Pirk, K. Hausman, A. Toshev, and M. Khansari, 'Modeling long-horizon tasks as sequential interaction landscapes,' arXiv:2006.04843 , 2020. - · arXiv
  32. [39][39] M. S. Sarfraz, N. Murray, V. Sharma, A. Diba, L. Van Gool, and R. Stiefelhagen, 'Temporally-weighted hierarchical clustering for unsupervised action segmentation,' arXiv:2103.11264 , 2021. - · arXiv
  33. [40][40] S. Sarfraz, V. Sharma, and R. Stiefelhagen, 'Efficient parameter-free clustering using first neighbor relations,' in CVPR , 2019. -
  34. [41][41] T. Schaul, D. Horgan, K. Gregor, and D. Silver, 'Universal value function approximators,' in ICML , 2015. -
  35. [42][42] T. Shankar and A. Gupta, 'Learning robot skills with temporal variational inference,' in ICML , 2020, pp. 8624-8633. -
  36. [43][43] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman, 'Dynamics-aware unsupervised discovery of skills,' in ICLR , 2019. -
  37. [44][44] K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner, 'Taco: Learning task decomposition via temporal alignment for control,' in ICML , 2018, pp. 46544663. -
  38. [46][46] Z. Su, O. Kroemer, G. E. Loeb, G. S. Sukhatme, and S. Schaal, 'Learning manipulation graphs from demonstrations using multimodal sensory signals,' in ICRA , IEEE, 2018. -
  39. [48][48] D. Tanneberg, K. Ploeger, E. Rueckert, and J. Peters, 'Skid raw: Skill discovery from raw trajectories,' RA-L , vol. 6, no. 3, pp. 4696-4703, 2021. -
  40. [49][49] A. Tung, J. Wong, A. Mandlekar, R. Martı´ en-Martı´ en, Y. Zhu, L. Fei-Fei, and S. Savarese, 'Learning multi-arm manipulation through collaborative teleoperation,' in ICRA , 2021. -
  41. [50][50] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu, 'Feudal networks for hierarchical reinforcement learning,' in ICML , 2017. -
  42. [51][51] U. Von Luxburg, 'A tutorial on spectral clustering,' Statistics and computing , vol. 17, no. 4, pp. 395-416, 2007. -
  43. [52][52] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, 'Neural task programming: Learning to generalize across hierarchical tasks,' in ICRA , IEEE, 2018. -
  44. [53][53] T. Yu, P. Abbeel, S. Levine, and C. Finn, 'One-shot hierarchical imitation learning of compound visuomotor tasks,' arXiv:1810.11043 , 2018. - · arXiv
  45. [54][54] J. M. Zacks and B. Tversky, 'Event structure in perception and conception,' Psychological bulletin , vol. 127, no. 1, p. 3, 2001. -
  46. [55][55] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, 'Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,' in ICRA , 2018. -
  47. [57][57] Y. Zhu, J. Wong, A. Mandlekar, and R. Martı´ en-Martı´ en, 'Robosuite: A modular simulation framework and benchmark for robot learning,' arXiv:2009.12293 , 2020. · arXiv

Cited by

Sorted by citations

Authors
Zhu, Y., Stone, P., & Zhu, Y.
Affiliations
The University of Texas at Austin (US)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-04
Volume
7
Issue
2
Pages
4126-4133
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2022.3146589
OpenAlex
W4225873172
Cited by
50
References
6
Citations by year
2026: 7 · 2025: 27 · 2024: 9 · 2023: 3 · 2022: 4
FWCI
4.33
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NSF
  • MLL Research Award
  • FLI
  • ARO
  • Defense Advanced Research Projects Agency
  • Lockheed Martin
  • GM
  • Robert Bosch (Australia) Pty
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
skill discovery, hierarchical task structure, agglomerative clustering, meta-controller, skill composition, human demonstrations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review