CoChat
Sign Up
Cited by 70Robot Manipulation and LearningReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Open access
Wen, Bowen, Lian, Wenzhao, Bekris, Kostas, Schaal, Stefan·2022·Robotics: Science and Systems XVIII
DOI 10.15607/rss.2022.xviii.044
Open this paper in CoChat — free account

Abstract

Promising results have been achieved recently in category-level manipulation that generalizes across object instances.Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task.Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place.This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking.The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video.The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation.During execution, the manipulation horizon is decomposed into longrange, collision-free motion and last-inch manipulation.For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control.CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame.The frame is automatically selected along the manipulation horizon by a local attention mechanism.This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming.Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in highprecision assembly, which involve learning complex, long-horizon policies.The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations.The supplementary video is available at https://www.youtube.com/watch?v=WAr8ZY3mYyw

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15607/rss.2022.xviii.044

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 53

  1. W. Lian, T. Kelch, D. Holz, A. Norton, and S. Schaal, "Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks," in IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, pp. 1046-1053.
  2. M. Kyrarini, M. A. Haseeb, D. Ristić-Durrant, and A. Gräser, "Robot learning of industrial assembly task via human demonstrations," Au- tonomous Robots, vol. 43, no. 1, pp. 239-257, 2019. · DOI
  3. P. R. Florence, L. Manuelli, and R. Tedrake, "Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation," CoRL, 2018.
  4. W. Gao and R. Tedrake, "kPAM 2.0: Feedback Control for Category- Level Robotic Manipulation," IEEE Robotics and Automation Letter (RA-L), 2020.
  5. S. Yang, W. Zhang, R. Song, J. Cheng, and Y. Li, "Learning Multi- Object Dense Descriptor for Autonomous Goal-Conditioned Grasping," IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 4109-4116, 2021.
  6. C.-Y. Chai, K.-F. Hsu, and S.-L. Tsao, "Multi-step pick-and-place tasks using object-centric dense correspondences," in 2019 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 4004-4011.
  7. Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese, "KETO: Learning keypoint representations for tool manipulation," in 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 7278-7285.
  8. M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz, "S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency," in Conference on Robot Learning (CoRL), 2020.
  9. D. B. Grimes, R. Chalodhorn, and R. P. Rao, "Dynamic Imitation in a Humanoid Robot through Nonparametric Probabilistic Inference." in Robotics: science and systems. Cambridge, MA, 2006, pp. 199-206.
  10. P. Englert, A. Paraschos, M. P. Deisenroth, and J. Peters, "Probabilistic model-based imitation learning," Adaptive Behavior, vol. 21, no. 5, pp. 388-403, 2013.
  11. A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine, "Combining self-supervised learning and imitation for vision-based rope manipulation," in 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 2017, pp. 2146-2153.
  12. Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, "One-shot imitation learning," NIPS, 2017.
  13. "Third-person imitation learning, author=Stadie, Bradly C and Abbeel, Pieter and Sutskever, Ilya," ICLR, 2017.
  14. C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, "One-shot visual imitation learning via meta-learning," in Conference on Robot Learning. PMLR, 2017, pp. 357-368.
  15. S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert, "Learning movement primitives," in Robotics research. the eleventh international symposium. Springer, 2005, pp. 561-572. · DOI
  16. S. Calinon and A. Billard, "Statistical learning by imitation of competing constraints in joint space and task space," Advanced Robotics, vol. 23, no. 15, pp. 2059-2076, 2009. · DOI
  17. P. Florence, L. Manuelli, and R. Tedrake, "Self-supervised correspon- dence in visuomotor policy learning," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 492-499, 2019.
  18. S. Paradis, M. Hwang, B. Thananjeyan, J. Ichnowski, D. Seita, D. Fer, T. Low, J. E. Gonzalez, and K. Goldberg, "Intermittent visual servoing: Efficiently learning policies robust to instrument changes for high- precision surgical manipulation," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 7166-7173.
  19. E. Johns, "Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration," ICRA, 2021.
  20. B. Wu, F. Xu, Z. He, A. Gupta, and P. K. Allen, "SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks," in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 9720-9727.
  21. D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, "Neural task graphs: Generalizing to unseen tasks from a single video demonstration," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8565-8574.
  22. J. Liang, B. Wen, K. Bekris, and A. Boularias, "Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstra- tions," ICRA, 2022.
  23. M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, "Graph- structured visual imitation," in Conference on Robot Learning. PMLR, 2020, pp. 979-989.
  24. D. Kappler, F. Meier, J. Issac, J. Mainprice, C. G. Cifuentes, M. Wüthrich, V. Berenz, S. Schaal, N. Ratliff, and J. Bohg, "Real- time perception meets reactive motion generation," IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 1864-1871, 2018.
  25. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," The Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334-1373, 2016.
  26. S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, "Learning hand-eye coordination for robotic grasping with deep learning and large- scale data collection," The International Journal of Robotics Research, vol. 37, no. 4-5, pp. 421-436, 2018. · DOI · Paper page
  27. U. Viereck, A. Pas, K. Saenko, and R. Platt, "Learning a visuomotor controller for real world robotic grasping using simulated depth images," in Conference on Robot Learning. PMLR, 2017, pp. 291-300.
  28. O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray et al., "Learning dexterous in-hand manipulation," The International Journal of Robotics Research, vol. 39, no. 1, pp. 3-20, 2020. · DOI · Paper page
  29. F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine, "Visual foresight: Model-based deep reinforcement learning for vision-based robotic control," arXiv preprint arXiv:1812.00568, 2018. · arXiv
  30. L. Manuelli, Y. Li, P. Florence, and R. Tedrake, "Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning," CoRL, 2020.
  31. A. Byravan, F. Leeb, F. Meier, and D. Fox, "Se3-pose-nets: Structured deep dynamics models for visuomotor control," in 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 3339-3346.
  32. S. Umeyama, "Least-squares estimation of transformation parameters be- tween two point patterns," IEEE Computer Architecture Letters, vol. 13, no. 04, pp. 376-380, 1991.
  33. B. Wen and K. E. Bekris, "BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models," in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021.
  34. H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, "Normalized object coordinate space for category-level 6d object pose and size estimation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2642-2651.
  35. B. Wen, W. Lian, K. Bekris, and S. Schaal, "CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation," arXiv preprint arXiv:2109.09163, 2021. · arXiv
  36. C. R. Qi, H. Su, K. Mo, and L. J. Guibas, "PointNet: Deep learning on point sets for 3d classification and segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652-660.
  37. B. Wen, C. Mitash, B. Ren, and K. E. Bekris, "se (3)-tracknet: Data- driven 6D pose tracking by calibrating image residuals in synthetic domains," in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 10 367-10 373.
  38. X. Deng, A. Mousavian, Y. Xiang, F. Xia, T. Bretl, and D. Fox, "PoseRBPF: A Rao-Blackwellized Particle Filter for 6-D Object Pose Tracking," IEEE Transactions on Robotics, 2021.
  39. M. Ester, H.-P. Kriegel, J. Sander, X. Xu et al., "A density-based algorithm for discovering clusters in large spatial databases with noise." in kdd, vol. 96, no. 34, 1996, pp. 226-231.
  40. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834-848, 2017.
  41. K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
  42. Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, "Randla-net: Efficient semantic segmentation of large- scale point clouds," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 108-11 117.
  43. C. Mitash, B. Wen, K. Bekris, and A. Boularias, "Scene-level pose esti- mation for multiple instances of densely packed objects," in Conference on Robot Learning. PMLR, 2020, pp. 1133-1145.
  44. M. Gualtieri and R. Platt, "Robotic Pick-and-Place With Uncertain Object Instance Segmentation and Shape Completion," IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1753-1760, 2021.
  45. R. Malladi, J. A. Sethian, and B. C. Vemuri, "Shape modeling with front propagation: A level set approach," IEEE transactions on pattern analysis and machine intelligence, vol. 17, no. 2, pp. 158-175, 1995.
  46. A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt, "Grasp pose detection in point clouds," The International Journal of Robotics Research, vol. 36, no. 13-14, pp. 1455-1473, 2017. · DOI
  47. S. Karaman and E. Frazzoli, "Sampling-based algorithms for optimal motion planning," The international journal of robotics research, vol. 30, no. 7, pp. 846-894, 2011. · DOI
  48. M. A. Fischler and R. C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, 1981. · DOI · Paper page
  49. J. Jin, L. Petrich, Z. Zhang, M. Dehghan, and M. Jagersand, "Visual geometric skill inference by watching human demonstration," in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 8985-8991.
  50. H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, "Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos," IROS, 2021.
  51. J. Tremblay, T. To, A. Molchanov, S. Tyree, J. Kautz, and S. Birchfield, "Synthetically trained neural networks for learning human-readable plans from real-world demonstrations," in 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 5659-5666.
  52. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, pp. 8026-8037, 2019.
  53. A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann, "Neural Descriptor Fields: SE (3)- Equivariant Object Representations for Manipulation," arXiv preprint arXiv:2112.05124, 2021. · arXiv

Cited by

Sorted by citations

Authors
Wen, B., Lian, W., Bekris, K., & Schaal, S.
Affiliations
Rutgers, The State University of New Jersey (US)
ORCID
Venue
Robotics: Science and Systems XVIII
Publisher
Robotics: Science and Systems Foundation
Published
2022-06-27
DOI
10.15607/rss.2022.xviii.044
OpenAlex
W4221167977
Cited by
70
References
3
Citations by year
2026: 11 · 2025: 28 · 2024: 18 · 2023: 11 · 2022: 2
FWCI
22.52
Type
conference paper
Language
en
Open access
Locations
  • Robotics: Science and Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
behavior cloning, object-centric representation, long-horizon manipulation, high-precision assembly, feedback control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review