CoChat
Sign Up
Cited by 6Robot Manipulation and LearningReinforcement Learning in RoboticsSoft Robotics and Applications

Cloth manipulation planning on basis of mesh representations with incomplete domain knowledge and voxel-to-mesh estimation

Open access
Arnold, Solvi, Tanaka, Daisuke, Yamazaki, Kimitoshi·2023·Frontiers in Neurorobotics
DOI 10.3389/fnbot.2022.1045747PMC PMC9849740
Open this paper in CoChat — free accountOpen PDF

Abstract

Cloth manipulation is common in both housework and manufacturing. However, robotic cloth manipulation remains challenging, especially for less controlled and open-goal settings. We consider the problem of open-goal planning for robotic cloth manipulation, with focus on the roles of cloth representation and epistemic uncertainty. Core of our system is a neural network trained as a forward model of cloth behaviour under manipulation, with planning performed through backpropagation. We introduce a neural network-based routine for estimating mesh representations from voxel input, and perform planning in mesh format internally. We address the problem of planning with incomplete domain knowledge by introducing an explicit epistemic uncertainty penalty, using prediction divergence between two instances of the forward model network as a proxy of epistemic uncertainty. This allows us to avoid plans with high epistemic uncertainty during planning. Finally, we introduce logic for handling restriction of grasp points to a discrete set of candidates, in order to accommodate graspability constraints imposed by robotic hardware. We evaluate the system’s mesh estimation, prediction, and planning ability on simulated cloth for sequences of one to three manipulations. Comparative experiments confirm that planning on basis of estimated meshes improves accuracy compared to voxel-based planning, and that epistemic uncertainty avoidance improves performance under conditions of incomplete domain knowledge. Planning time cost is a few seconds. We additionally present qualitative results on robot hardware. Our results indicate that representation format and epistemic uncertainty are important factors to consider for open-goal cloth manipulation planning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fnbot.2022.1045747

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Mesh representation of the cloth can bring topological information into planning for grasping and flattening, and plays a significant role in keeping track of the state of the article, especially when there are occlusions in the visual input.

    “Mesh representation of the cloth can bring topological information into planning for grasping and flattening, and it plays a significant role in keeping track of the state of the article, especially when there are occlusions in the visual input [179,235,267].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [235]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Arnold S. Yamazaki K. (2019a). Fast and flexible multi-step cloth manipulation planning using an encode-manipulate-decode network (EM*D Net). Front. Neurorobot. 13: 22. 10.3389/fnbot.2019.00022 31214008 PMC6554328 · DOI
  2. Arnold S. Yamazaki K. (2019b). “ Neural network architectures for estimating mesh representations of cloth objects,” in Proceedings of the 37th annual conference of the Robotics Society of Japan (RSJ2019), Tokyo.
  3. Arnold S. Yamazaki K. (2019c). Unknown Unknowns in Neural Network Based Planning - Meta-Knowledge for Self-Deception Avoidance. Available online at: https://www.researchgate.net/publication/336361908_Unknown_Unknowns_in_Neural_Network-Based_Planning_-_Meta-Knowledge_for_Self-Deception_Avoidance
  4. Arnold S. Yamazaki K. (2022). “ Cloth manipulation with estimation of material properties,” in Proceedings of the 40th annual conference of the Robotics Society of Japan (RSJ2022), Tokyo. 10.7210/jrsj.40.375 · DOI
  5. Athalye A. Carlini N. Wagner D. (2018). Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1802.00420 (accessed November 23, 2022). · arXiv
  6. Bernstein J. Wang Y.-X. Azizzadenesheli K. Anandkumar A. (2018). signSGD: Compressed optimisation for non-convex problems. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1802.04434 (accessed November 23, 2022). · arXiv
  7. Choi K. J. Ko H. S. (2002). Stable but responsive cloth. ACM Trans. Graph. 21 604– 611. 10.1145/566654.566624 · DOI
  8. Clavera I. Fu Y. Abbeel P. (2020). “ Model-augmented actor-Critic: Backpropagating through Paths,” in Proceedings of the 2020 Eighth International Conference on Learning Representations (ICLR), Addis Ababa.
  9. Ebert F. Finn C. Dasari S. Xie A. Lee A. Levine S. (2018). Visual foresight: Model-based deep reinforcement learning for vision based robotic control. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1812.00568 (accessed November 23, 2022). · arXiv
  10. Erickson Z. Clever H. M. Turk G. Liu C. K. Kemp C. C. (2018). “ Deep Haptic Model Predictive Control for Robot-Assisted Dressing,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), (Brisbane), 1– 8. 10.1109/ICRA.2018.8460656 · DOI
  11. Goodfellow I. S. J. Szegedy C. (2014). Explaining and harnessing adversarial examples. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1412.6572 (accessed November 23, 2022). · arXiv
  12. Granitto P. M. Verdes P. F. Ceccatto H. A. (2005). Neural network ensembles: Evaluation of aggregation algorithms. Artif. Intell. 163 139– 162. 10.1016/j.artint.2004.09.006 · DOI
  13. Han T. Zhao X. Sun P. Pan J. (2018). Robust shape estimation for 3D deformable object manipulation. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1809.09802 (accessed November 23, 2022). · arXiv
  14. Hansen L. K. Salamon P. (1990). Neural network ensembles. IEEE Trans. Patt. Anal. Mach. Intell. 12 993– 1001. 10.1109/34.58871 · DOI · Paper page
  15. Henaff M. Whitney W. F. Lecun Y. (2017). Model-based planning in discrete action spaces. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1705.07177 (accessed November 23, 2022). · arXiv
  16. Hoque R. Seita D. Balakrishna A. Ganapathi A. Tanwani A. K. Jamali N. (2020). VisuoSpatial foresight for multi-step, multi-task fabric manipulation. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/2003.09044 (accessed November 23, 2022). · arXiv
  17. Igel C. Hüsken M. (2000). “ Improving the Rprop Learning Algorithm,” in Proceedings of the 2000 Second International Symposium on Neural Computation, NC’2000, Berlin, 115– 121.
  18. Janner M. Fu J. Zhang M. Levine S. (2019). When to trust your model: Model-based policy optimization. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1906.08253 (accessed November 23, 2022). · arXiv
  19. Kawaharazuka K. Miki A. Bando M. Okada K. Inaba M. (2022). Dynamic cloth manipulation considering variable stiffness and material change using deep predictive model with parametric Bias. Front. Neurorobot. 16: 890695. 10.3389/fnbot.2022.890695 35677831 PMC9168129 · DOI
  20. Kawaharazuka K. Ogawa T. Tamura J. Nabeshima C. (2019). Dynamic manipulation of flexible objects with torque sequence using a deep neural network. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1901.10142 (accessed November 23, 2022). · arXiv
  21. Kita K. Kanehiro F. Ueshiba T. Kita N. (2014). “ Strategy for folding clothes on the basis of deformable models. image analysis and recognition (ICIAR2014),” in Proceedings of the 11th International Conference on Image Analysis and Recognition (ICIAR 2014), Vilamoura. 10.1007/978-3-319-11755-3_49 · DOI
  22. Koishihara Y. Arnold S. Matsubara T. Y. K. (2017). “ Hanging Work of T-Shirt in Consideration of Deformability and Strechability,” in Proceedings of the IEEE International Conference on Information and Automation, Macau, 130– 135. 10.1109/ICInfA.2017.8078894 · DOI
  23. Kurakin A. Goodfellow I. Bengio S. (2016). Adversarial examples in the physical world. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1607.02533 (accessed November 23, 2022). · arXiv
  24. Lesort T. Díaz-Rodríguez N. Goudou J.-F. Filliat D. (2018). State representation learning for control: An overview. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1802.04181 (accessed November 23, 2022). 30268059 · arXiv
  25. Li Y. Lei Q. Cheng C. Zhang G. Wang W. Xu Z. (2019). “ A review: Machine learning on robotic grasping,” in Proceedings of the Eleventh International Conference on Machine Vision (ICMV 2018), Munich. 10.1117/12.2522945 · DOI
  26. Li Y. Wang Y. Case M. Chang S.-F. Allen P. K. (2014). “ Real-time pose estimation of deformable objects using a volumetric approach,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2014), Chicago, IL. 10.1109/IROS.2014.6942687 · DOI
  27. Li Y. Yue Y. Xu D. Grinspun E. Allen P. K. (2015). “ Folding deformable objects using predictive simulation and trajectory optimization,” in Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, Hamburg, 6000– 6006. 10.1109/IROS.2015.7354231 · DOI
  28. Lips T. De Gusseme V. L. Wyffels F. (2022). “ Learning Keypoints from Synthetic Data for Robotic Cloth Folding,” in Proceedings of the ICRA 2022 2nd workshop on representing and manipulating deformable objects, Philadelphia, PA.
  29. Narain R. Pfaff T. O’Brien J. F. (2013). “ Folding and crumpling adaptive sheets,” in Proceedings of ACM SIGGRAPH 2013, ACM Transactions on Graphics, Tokyo, 1– 8. 10.1145/2461912.2462010 · DOI
  30. Narain R. Samii A. O’Brien J. F. (2012). Adaptive Anisotropic Remeshing for Cloth Simulation. ACM Trans. Graph. 31 1– 10. 10.1145/2366145.2366171 · DOI
  31. Pereira M. Fan D. D. An G. N. Theodorou E. (2018). MPC-inspired neural network policies for sequential decision making. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1802.05803 (accessed November 23, 2022). · arXiv
  32. Petrík V. Kyrki V. (2019). “ Feedback-based Fabric Strip Folding,” in Proceedings of the 2019 IEEE/RSJ Int’l Conference on Intelligent Robots and Systems, Macau. 10.1109/IROS40897.2019.8967657 · DOI
  33. Qi C. R. Su H. Mo K. Guibas L. J. (2017). “ PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” in Proceedings of the Computer Vision and Pattern Recognition (CVPR2017), Honolulu, HI.
  34. Seita D. Ganapathi A. Hoque R. Hwang M. Cen E. Tanwani A. K. (2019). Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1910.04854 (accessed November 23, 2022). · arXiv
  35. Seita D. Jamali N. Laskey M. Berenstein R. Tanwani A. K. Iba P. B. S. (2018). Robot bed-making: Deep transfer learning using depth sensing of deformable fabric. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1809.09810 (accessed November 23, 2022). · arXiv
  36. Shi J. Tomasi C. (1994). “ Good features to track,” in Proceedings of the 9th IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, 593– 600.
  37. Sun L. Aragon-Camarasa G. Rogers S. Siebert J. P. (2015). “ Accurate garment surface analysis using an active stereo robot head with application to dual-arm flattening,” in Proceedings of the 2015 IEEE Int’l Conference on Robotics and Automation, Seattle, WA, 185– 192. 10.1109/ICRA.2015.7138998 · DOI
  38. Szegedy C. Zaremba W. Sutskever I. Bruna J. Erhan D. Goodfellow I. (2013). Intriguing properties of neural networks. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/1312.6199 (accessed December 18, 2015). · arXiv
  39. Tanaka D. Arnold S. Yamazaki K. (2018). EMD Net: An encode–manipulate–decode network for cloth manipulation. IEEE Robot. Autom. Lett. 3 1771– 1778. 10.1109/LRA.2018.2800122 · DOI
  40. Tanaka D. Arnold S. Yamazaki K. (2021). Disruption-resistant deformable object manipulation on basis of online shape estimation and prediction-driven trajectory correction. IEEE Robot. Autom. Lett. 6 3809– 3816. 10.1109/LRA.2021.3060679 · DOI
  41. Van den Berg J. Miller S. Goldberg K. Abbeel P. (2010). “ Gravity-based robotic cloth folding,” in Algorithmic Foundations of Robotics IX. Springer Tracts in Advanced Robotics, Vol. 68 eds Hsu D. Isler V. Latombe J. C. Lin M. C. (Berlin: Springer). 10.1007/978-3-642-17452-0_24 · DOI
  42. Wahlström N. Schön T. Deisenroth M. (2015). “ From pixels to torques: Policy learning with deep dynamical models,” in Proceedings of the Deep Learning Workshop at the 32nd International Conference on Machine Learning (ICML), Lille. 10.1016/j.ifacol.2015.12.271 · DOI
  43. Watter M. Springenberg J. T. Boedecker J. Riedmiller M. (2015). Embed to control: A locally linear latent dynamics model for control from raw images. Adv. Neural Inf. Process. Syst. 28 2746– 2754.
  44. Willimon B. Hickson S. Walker I. Birchfield S. (2012). “ An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using RGBD Data,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2012), Vilamoura. 10.1109/IROS.2012.6386213 · DOI
  45. Wu Y. Yan W. Kurutach T. Pinto L. Abbeel P. (2020). Learning to manipulate deformable objects without demonstrations,” in Proceedings of the Robotics: Science and Systems, Corvalis, OR. 10.15607/RSS.2020.XVI.065 · DOI
  46. Yan W. Vangipuram A. Abbeel P. Pinto L. (2020). Learning predictive representations for deformable objects using contrastive estimation. arXiv [ Preprint]. Available online at: https://arxiv.org/abs/2003.05436 (accessed November 23, 2022). · arXiv
  47. Yang P.-C. Sasaki K. Suzuki K. Kase K. Sugano S. Ogata T. (2017). Repeatable folding task by humanoid robot worker using deep learning. IEEE Robot. Autom. Lett. 2 397– 403. 10.1109/LRA.2016.2633383 · DOI
  48. Yuba H. Arnold S. Yamazaki K. (2017). Unfolding of a rectangular cloth from unarranged starting shapes by a Dual-Armed robot with a mechanism for managing recognition error and uncertainty. Adv. Robot. 3 544– 556. 10.1080/01691864.2017.1285722 · DOI

Cited by

Sorted by citations

Authors
Arnold, S., Tanaka, D., & Yamazaki, K.
Affiliations
Shinshu University (JP)
Corresponding author
Arnold, Solvi
ORCID
Venue
Frontiers in Neurorobotics
Publisher
Frontiers Media SA
Published
2023-01-05
Volume
16
Pages
1045747-1045747
ISSN
1662-5218 (ISSN-L)
DOI
10.3389/fnbot.2022.1045747
PMC
PMC9849740
OpenAlex
W4313558389
Cited by
6
References
2
Citations by year
2025: 1 · 2024: 1 · 2023: 3 · 2022: 1
FWCI
0.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Neurorobotics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • New Energy and Industrial Technology Development Organization
  • Japan Society for the Promotion of Science
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
Keywords
fabric manipulation, mesh representation, epistemic uncertainty, cloth simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review