Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data
Open accessAbstract
Robotic manipulation of deformable 1D objects such as ropes, cables, and hoses is challenging due to the lack of high-fidelity analytic models and large configuration spaces. Furthermore, learning end-to-end manipulation policies directly from images and physical interaction requires significant time on a robot and can fail to generalize across tasks. We address these challenges using interpretable deep visual representations for rope, extending recent work on dense object descriptors for robot manipulation. This facilitates the design of interpretable and transferable geometric policies built on top of the learned representations, decoupling visual reasoning and control. We present an approach that learns point-pair correspondences between initial and goal rope configurations, which implicitly encodes geometric structure, entirely in simulation from synthetic depth images. We demonstrate that the learned representation — dense depth object descriptors (DDODs) — can be used to manipulate a real rope into a variety of different arrangements either by learning from demonstrations or using interpretable geometric policies. In 50 trials of a knot-tying task with the ABB YuMi Robot, the system achieves a 66% knot-tying success rate from previously unseen configurations. See https://tinyurl.com/rope-learning for supplementary material and videos.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sundaresan et al. achieved 66% real-world success in tying the overhand-knot by improving the descriptor perception system within the LfO framework.
“Sundaresan et al. (2020) [250] achieve 66% real-world success in tying the overhand-knot by improving the descriptor perception system—it can tell the exact location of the individual part of the rope in the image—within the LfO framework.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [250]
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
Siamese Neural Networks for One-shot Image Recognition
Koch, G. E., Zemel, R. S., & Salakhutdinov, R. (2015) · Cited by 3,712 · see citing works
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. Y. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.058 · Cited by 1,299 · see citing works
Self-citationDynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951 · see citing works
Trotter, H. F., Crowell, R. H., & Fox, R. H. (1964) · American Mathematical Monthly · doi:10.2307/2311437 · Cited by 733 · see citing works
Kuniyoshi, Y., Inaba, M., & Inoue, H. (1994) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.338535 · Cited by 683 · see citing works
Canny edge detection enhancement by scale multiplication
Bao, P. G., Zhang, L., & Wu, X. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.173 · Cited by 608 · see citing works
Realtime and Robust Hand Tracking from Depth
Qian, C., Sun, X. N., Wei, Y., Tang, X., & Sun, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.145 · Cited by 474 · see citing works
Perlin, K. (2002) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/566570.566636 · Cited by 465 · see citing works
Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., & Abbeel, P. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509439 · Cited by 418 · see citing works
Mahler, J., Pokorny, F. T., Hou, B., Roderick, M., Laskey, M., Aubry, M., Kohlhoff, K., Kröger, T., Kuffner, J., & Goldberg, K. Y. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487342 · Cited by 361 · see citing works
Self-citationMahler, J., Matl, M., Liu, X., Li, A., Gealy, D. V., & Goldberg, K. Y. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460887 · Cited by 326 · see citing works
Self-citationObject recognition and full pose registration from a single image for robotic manipulation
Collet, A., Berenson, D., Srinivasa, S. S., & Ferguson, D. (2009) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2009.5152739 · Cited by 277 · see citing works
Combining self-supervised learning and imitation for vision-based rope manipulation
Nair, A., Chen, D., Agrawal, P., Isola, P., Abbeel, P., Malik, J., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989247 · Cited by 260 · see citing works
Perlin, K. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566636 · Cited by 259 · see citing works
van den Berg, J., Miller, S., Duckworth, D., Hu, H., Wan, A., Fu, X., Goldberg, K., & Abbeel, P. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509621 · Cited by 233 · see citing works
Self-citationLearning Compact Geometric Features
Khoury, M., Zhou, Q.-Y., & Koltun, V. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.26 · Cited by 218 · see citing works
Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, F., Shelhamer, E., Malik, J., Efros, A. A., & Darrell, T. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00278 · Cited by 188 · see citing works
A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks
Mayer, H. G., Gomez, F. J., Wierstra, D., Nagy, I., Knoll, A., & Schmidhuber, J. (2008) · Advanced Robotics · doi:10.1163/156855308x360604 · Cited by 183 · see citing works
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic ManipulationPreprint
Florence, P., Manuelli, L., & Tedrake, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08756 · Cited by 181 · see citing works
Self-Supervised Visual Descriptor Learning for Dense Correspondence
Schmidt, T., Newcombe, R., & Fox, D. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2634089 · Cited by 146 · see citing works
Thananjeyan, B., Garg, A., Krishnan, S., Chen, C., Miller, L., & Goldberg, K. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989275 · Cited by 144 · see citing works
Self-citationPoint inversion and projection for NURBS curve and surface: Control polygon approach
Ma, Y., & Hewitt, W. T. (2003) · Computer Aided Geometric Design · doi:10.1016/s0167-8396(03)00021-9 · Cited by 138 · see citing works
Learning from Demonstrations Through the Use of Non-rigid Registration
Schulman, J., Ho, J., Lee, C., & Abbeel, P. (2016) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-28872-7_20 · Cited by 133 · cited 1× · see citing works
“There is also extensive literature on deformable manipulation [18, 25, 27, 36, 40].”
BackgroundKnotting/unknotting manipulation of deformable linear objects
Wakamatsu, H., Arai, E., & Hirai, S. (2006) · The International Journal of Robotics Research · doi:10.1177/0278364906064819 · Cited by 132 · see citing works
Learning Robotic Manipulation through Visual Planning and Acting
Wang, A., Kurutach, T., Liu, K., Abbeel, P., & Tamar, A. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.074 · Cited by 96 · see citing works
Schulman, J., Gupta, A., Venkatesan, S., Tayson-Frederick, M., & Abbeel, P. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696945 · Cited by 84 · see citing works
Thananjeyan, B., Balakrishna, A., Rosolia, U., Li, F., McAllister, R., Gonzalez, J. E., Levine, S., Borrelli, F., & Goldberg, K. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2976272 · Cited by 83 · see citing works
Self-citationA case study of flexible object manipulation
Hopcroft, J. E., Kearney, J. K., & Krafft, D. B. (1991) · The International Journal of Robotics Research · doi:10.1177/027836499101000105 · Cited by 82 · see citing works
Grasp2Vec: Learning Object Representations from Self-Supervised GraspingPreprint
Jang, E., Devin, C., Vanhoucke, V., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06964 · Cited by 82 · see citing works
(2003) · doi:10.1109/robot.2003.1242193 · Cited by 73 · see citing works
Task-oriented grasping with semantic and geometric scene understanding
Detry, R. J., Papon, J. A., & Matthies, L. H. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206162 · Cited by 69 · see citing works
Linear Push Policies to Increase Grasp Access for Robot Bin Picking
Danielczuk, M., Mahler, J., Correa, C., et al. (2018) · 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/coase.2018.8560406 · Cited by 63 · see citing works
Picking up a soft 3D object by “feeling” the grip
Lin, H., Guo, F., Wang, F., et al. (2015) · Int. J. Robotics Res. · doi:10.1177/0278364914564232 · Cited by 61 · see citing works
Online Deformable Object Tracking Based on Structure-Aware Hyper-Graph
Du, D., Qi, H., Li, W., Wen, L., Huang, Q., & Lyu, S. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2570556 · Cited by 55 · see citing works
An introduction to knot theory
(1998) · Choice Reviews Online · doi:10.5860/choice.35-5134 · Cited by 46 · see citing works
Grasping deformable planar objects: Squeeze, stick/slip analysis, and energy-based optimalities
Jia, Y.-B., Guo, F., & Lin, H. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913512170 · Cited by 44 · see citing works
Deep Transfer Learning of Pick Points on Fabric for Robot Bed-MakingPreprint
Seita, D., Jamali, N., Laskey, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09810 · Cited by 23 · see citing works
Self-citationInteractive computational imaging for deformable object analysis
Dansereau, D. G., Singh, S. P. N., & Leitner, J. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487697 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Modeling, learning, perception, and control methods for deformable object manipulation
Yin, H., Varava, A., & Kragic, D. (2021) · Science Robotics · doi:10.1126/scirobotics.abd8803 · Cited by 299 · see citing works
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works
Seita, D., Florence, P., Tompson, J., Coumans, E., Sindhwani, V., Goldberg, K., & Zeng, A. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561391 · Cited by 130 · see citing works
Self-citationUntangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies
Sundaresan*, P., Grannen*, J., Thananjeyan, B., Balakrishna, A., Ichnowski, J., Novoseller, E., Hwang, M., Laskey, M., Gonzalez, J., & Goldberg, K. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.013 · Cited by 26 · see citing works
Self-citationKadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Sundaresan et al. (2020) [250] achieve 66% real-world success in tying the overhand-knot by improving the descriptor perception system—it can tell the exact location of the individual part of the rope in the image—within the LfO framework.”
BackgroundMethodDisentangling Dense Multi-Cable Knots
Viswanath, V., Grannen, J., Sundaresan, P., Thananjeyan, B., Balakrishna, A., Novoseller, E., Ichnowski, J., Laskey, M., Gonzalez, J. E., & Goldberg, K. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636397 · Cited by 21 · see citing works
Self-citation
- Authors
- Sundaresan, P., Grannen, J., Thananjeyan, B., Balakrishna, A., Laskey, M., Stone, K., Gonzalez, J. E., & Goldberg, K.
- Affiliations
- University of California, Berkeley (US)
- Toyota Research Institute (US)
- Venue
- 2020 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 9411-9418
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3090004508
- Cited by
- 107
- References
- 41
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 20 · 2023: 26 · 2022: 18 · 2021: 24 · 2020: 5
- FWCI
- 27.38
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Keywords
- rope manipulation, deformable object manipulation, knot tying, imitation learning, robotic manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
