In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
Open accessAbstract
In this study, we report the successful execution of in-air knotting of rope using a dual-arm two-finger robot based on deep learning. Owing to its flexibility, the state of the rope was in constant flux during the operation of the robot. This required the robot control system to dynamically correspond to the state of the object at all times. However, a manual description of appropriate robot motions corresponding to all object states is difficult to be prepared in advance. To resolve this issue, we constructed a model that instructed the robot to perform bowknots and overhand knots based on two deep neural networks trained using the data gathered from its sensorimotor, including visual and proximity sensors. The resultant model was verified to be capable of predicting the appropriate robot motions based on the sensory information available online. In addition, we designed certain task motions based on the Ian knot method using the dual-arm two-fingers robot. The designed knotting motions do not require a dedicated workbench or robot hand, thereby enhancing the versatility of the proposed method. Finally, experiments were performed to estimate the knotting performance of the real robot while executing overhand knots and bowknots on rope and its success rate. The experimental results established the effectiveness and high performance of the proposed method.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most knot tying systems act either on the table or in the air.
“Most KT systems act either on the table [243,244,275,282,286,287] or in the air [52,91,105,239,284,285] as shown in Figure 6.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [239]
Suzuki et al. leveraged the multi-modal deep BC method to achieve a 95% success rate on in-air dual-arm bow-knots and overhand-knot tying in real-world trials.
“Suzuki et al. (2021) [239] leverage the multi-modal deep BC method to achieve a 95% success rate on in-air dual-arm bow-knots and overhand-knot tying in real-world trials.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [239]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Recurrent nets that time and count
Gers, F. A., & Schmidhuber, J. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.861302 · Cited by 656 · see citing works
Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., & Abbeel, P. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509439 · Cited by 418 · see citing works
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2852779 · Cited by 365 · see citing works
Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349 · see citing works
A geometric approach to robotic laundry folding
Miller, S., van den Berg, J., Fritz, M., Darrell, T., Goldberg, K., & Abbeel, P. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911430417 · Cited by 238 · see citing works
Repeatable Folding Task by Humanoid Robot Worker Using Deep Learning
Yang, P.-C., Sasaki, K., Suzuki, K., Kase, K., Sugano, S., & Ogata, T. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2633383 · Cited by 220 · see citing works
Self-citationLearning to Manipulate Deformable Objects without Demonstrations
Wu, Y., Yan, W., Kurutach, T., Pinto, L., & Abbeel, P. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.065 · Cited by 181 · see citing works
Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation
Tsurumine, Y., Cui, Y., Uchibe, E., & Matsubara, T. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.11.004 · Cited by 171 · see citing works
Knotting/unknotting manipulation of deformable linear objects
Wakamatsu, H., Arai, E., & Hirai, S. (2006) · The International Journal of Robotics Research · doi:10.1177/0278364906064819 · Cited by 132 · see citing works
Survey on model-based manipulation planning of deformable objects
Jiménez, P. (2011) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2011.08.002 · Cited by 129 · see citing works
Dynamic Cloth Manipulation with Deep Reinforcement Learning
Jangir, R., Alenya, G., & Torras, C. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196659 · Cited by 114 · see citing works
Learning Robotic Manipulation through Visual Planning and Acting
Wang, A., Kurutach, T., Liu, K., Abbeel, P., & Tamar, A. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.074 · Cited by 96 · see citing works
Hand eye coordination in rope handling
INABA, M., & INOUE, H. (1985) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.3.538 · Cited by 95 · see citing works
Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making
Seita, D., Jamali, N., Laskey, M., Tanwani, A. K., Berenstein, R., Baskaran, P., Iba, S., Canny, J., & Goldberg, K. Y. (2022) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-95459-8_17 · Cited by 88 · see citing works
Iterative path optimisation for personalised dressing assistance using vision and force information
Gao, Y., Chang, H. J., & Demiris, Y. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759647 · Cited by 78 · see citing works
VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
Hoque, R., Seita, D., Balakrishna, A., Ganapathi, A., Tanwani, A., Jamali, N., Yamane, K., Iba, S., & Goldberg, K. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.034 · Cited by 74 · see citing works
(2003) · doi:10.1109/robot.2003.1242193 · Cited by 73 · see citing works
Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
Ganapathi, A., Sundaresan, P., Thananjeyan, B., Balakrishna, A., Seita, D., Grannen, J., Hwang, M., Hoque, R., Gonzalez, J. E., Jamali, N., et al. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561980 · Cited by 69 · see citing works
Imitation Learning Based on Bilateral Control for Human–Robot Cooperation
Sasagawa, A., Fujimoto, K., Sakaino, S., & Tsuji, T. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3011353 · Cited by 63 · see citing works
EMD Net: An Encode–Manipulate–Decode Network for Cloth Manipulation
Tanaka, D., Arnold, S., & Yamazaki, K. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2800122 · Cited by 60 · see citing works
Folding paper with anthropomorphic robot hands using real-time physics-based modeling
Elbrechter, C., Haschke, R., & Ritter, H. J. (2012) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2012.6651522 · Cited by 50 · see citing works
Reinforcement learning of a motor skill for wearing a T-shirt using topology coordinates
Matsubara, T., Shinohara, D., & Kidode, M. (2013) · Advanced Robotics · doi:10.1080/01691864.2013.777012 · Cited by 41 · cited 1× · see citing works
“On the contrary, in machine learning-based approaches, especially in those based on deep neural networks (DNNs), the sensorimotor experience of the robot can be learned [6][11][12] or reinforcement learning using real robots or in a sim-to-real manner can be used [13][14][15][16][17].”
BackgroundCanal, G., Pignat, E., Alenyà, G., Calinon, S., & Carme Torras, A. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460606 · Cited by 33 · see citing works
Dynamic High-Speed Knotting of a Rope by a Manipulator
Yamakawa, Y., Namiki, A., & Ishikawa, M. (2013) · International Journal of Advanced Robotic Systems · doi:10.5772/56783 · Cited by 31 · see citing works
Put-in-Box Task Generated from Multiple Discrete Tasks by aHumanoid Robot Using Deep Learning
Kase, K., Suzuki, K., Yang, P.-C., et al. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460623 · Cited by 30 · see citing works
Self-citationCaldwell, T. M., Coleman, D., & Correll, N. (2014) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2014.6942666 · Cited by 26 · see citing works
Dynamic Manipulation of Flexible Objects with Torque Sequence Using a Deep Neural Network
Kawaharazuka, K., Ogawa, T., Tamura, J., & Nabeshima, C. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793513 · Cited by 24 · see citing works
In-air Knotting of Rope by a Dual-arm Multi-finger Robot
Kudoh, S., Gomi, T., Katano, R., Tomizawa, T., & Suehiro, T. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354262 · Cited by 23 · see citing works
Suzuki, K., Hiroki, M. O. R. I., & Ogata, T. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2853651 · Cited by 23 · see citing works
Self-citationDeep Transfer Learning of Pick Points on Fabric for Robot Bed-MakingPreprint
Seita, D., Jamali, N., Laskey, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09810 · Cited by 23 · see citing works
Suzuki, K., Mori, H., & Ogata, T. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3063702 · Cited by 21 · see citing works
Self-citationVisuoSpatial Foresight for Multi-Step, Multi-Task Fabric ManipulationPreprint
Hoque, R., Seita, D., Balakrishna, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09044 · Cited by 20 · see citing works
Manipulation planning for unraveling linear objects
Wakamatsu, H., Tsumaya, A., Arai, E., & Hirai, S. (2006) · Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. · doi:10.1109/robot.2006.1642075 · Cited by 19 · see citing works
Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real FabricsPreprint
Ganapathi, A., Sundaresan, P., Thananjeyan, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12698 · Cited by 15 · see citing works
Realization of Five Types of Tabletop Knotting with Dual-Arm Robot
Katano, R., Gomi, T., Tomizawa, T., et al. (2015) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.33.505 · Cited by 11 · see citing works
Put-In-Box task generated from multiple discrete tasks by humanoid robot using deep learning
Kase, K., Suzuki, K., Yang, P.-C., et al. (2017) · The Proceedings of JSME annual Conference on Robotics and Mechatronics (Robomec) · doi:10.1299/jsmermd.2017.1p2-n07 · Cited by 9 · see citing works
Self-citationDevelopment of a Basic Educational Kit for Robotic System with Deep Neural Networks
Kanamura, M., Suzuki, K., Suga, Y., & Ogata, T. (2021) · Sensors · doi:10.3390/s21113804 · Cited by 9 · see citing works
Self-citationRope Structure Recognition for Manipulation Using Topological Model and Knot Invariant
Matsuno, T., TAMAKI, D., Arai, F., & Fukuda, T. (2005) · Transactions of the Society of Instrument and Control Engineers · doi:10.9746/sicetr1965.41.366 · Cited by 6 · see citing works
Study on Tying of a Deformable Band-shaped Object by a Dual Arm Robot
Seo, A., Takizawa, M., Kudoh, S., & Suehiro, T. (2019) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2019.8700456 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [17][17] J. Matas, S. James, A. S. Davison, 'Sim-to-Real Reinforcement Learning for Deformable Object Manipulation,' in Proc. of Conf. on Robot Learning, 2018. -
- [19][19] Y. Yamakawa, A. Namiki, M. Ishikawa, M. Shimojo, 'Dynamic highspeed knotting of a rope by a manipulator,' International Journal of Advanced Robotic Systems, vol. 10, no. 361, pp. 1-12, 2013. -
- [25][25] T. Morita, J. Takamatsu, K. Ogawara, H. Kimura, K. Ikeuchi, 'Knot planning from observation,' in Proc. of IEEE Int. Conf. on Robotics and Automation, pp. 3887-3892, 2003. -
- [29][29] J. Grannen, P. Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, V. Viswanath, M. Laskey, J. E. Gonzalez, K. Goldberg, 'Learning Robot Policies for Untangling Dense Knots in Linear Deformable Structures,' in Proc. of Conf. on Robot Learning, 2020. -
Cited by
Sorted by citations
AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild
Fang, H., Fang, H.-S., Wang, Y., Ren, J., Chen, J., Zhang, R., Wang, W., & Lu, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610799 · Cited by 38 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Most KT systems act either on the table [243,244,275,282,286,287] or in the air [52,91,105,239,284,285] as shown in Figure 6.”
BackgroundMethod
- Authors
- Suzuki, K., Kanamura, M., Suga, Y., Mori, H., & Ogata, T.
- Affiliations
- Waseda University (JP)
- Venue
- 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2021-09-27
- Pages
- 6724-6731
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W3137760825
- Cited by
- 31
- References
- 41
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 7 · 2023: 4 · 2022: 11 · 2021: 1
- FWCI
- 8.58
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Energy and Industrial Technology Development Organization
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- rope manipulation, dual-arm robot, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
