The DOI matches its Crossref record, with a small difference in title or year; 2 of the 2 references it cites were found in a registry or an index.
Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects
Open accessAbstract
Manipulating deformable objects, such as ropes and clothing, is a long-standing challenge in robotics, because of their large degrees of freedom, complex non-linear dynamics, and self-occlusion in visual perception. The key difficulty is a suitable representation, rich enough to capture the object shape, dynamics for manipulation and yet simple enough to be estimated reliably from visual observations. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). G-DOOM approximates a deformable object as a sparse set of interacting keypoints, which are extracted automatically from images via unsupervised learning. It learns a graph neural network that captures abstractly the geometry and the interaction dynamics of the keypoints. To handle object self-occlusion, G-DOOM uses a recurrent neural network to track the keypoints over time and condition their interactions on the history. We then train the resulting recurrent graph dynamics model through contrastive learning in a high-fidelity simulator. For manipulation planning, G-DOOM reasons explicitly about the learned dynamics model through model-predictive control applied at each keypoint. Preliminary experiments of G-DOOM on a set of challenging rope and cloth manipulation tasks indicate strong performance, compared with state-of-the-art methods. Although trained in a simulator, G-DOOM transfers directly to a real robot for both rope and cloth manipulation11Demo video available online at https://youtu.be/oCfbNMx2sQI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It often requires training the model with a large amount of data (more than 100k observation-action pairs).
“It often requires training the model with a large amount of data (more than 100k observation–action pairs) [38,233,234].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [234]
The Visual Keypoint Extraction section describes training on a collected dataset but gives no number of observation–action pairs. Searches for “100k,” “100,000,” and training-dataset size found no passage saying that the model requires more than 100,000 pairs; the 1,000 random seeds in Evaluation Setups refer to evaluation, not training data. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Seita, D., Florence, P., Tompson, J., Coumans, E., Sindhwani, V., Goldberg, K., & Zeng, A. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561391 · Cited by 130
Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience
Lee, R., Ward, D., Cosgun, A., Dasagi, V., Corke, P., & Leitner, J. (2020) · arXiv · doi:10.48550/arxiv.2010.03209 · Cited by 47
Flagged first, then by citations
Research works printed but not linked: 35
- 10.1109/TNNLS.2020.2978386 · DOI · Paper page
- wu, 2019, Learning to manipulate deformable objects without demonstrations, ArXiv Preprint
- 10.1109/ROBOT.2001.932534 · DOI
- veli?kovi?, 2017, Graph attention networks, ArXiv Preprint
- zeng, 2020, Transporter networks: Rearranging the visual world for robotic manipulation, ArXiv Preprint
- 10.1007/978-3-030-58610-2_30 · DOI
- yan, 2020, Learning predictive representations for deformable objects using contrastive estimation, ArXiv Preprint
- 10.1109/ICRA.2011.5979606 · DOI
- 10.1109/IROS.2015.7354231 · DOI
- li, 2019, Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids, 7th International Conference on Learning Representations ICLR 2019, 2019
- lin, 2020, Softgym: Benchmarking deep reinforcement learning for deformable object manipulation, ArXiv Preprint
- lin, 2021, Learning visible connectivity dynamics for cloth smoothing, ArXiv Preprint
- ma, 2020, Contrastive variational model-based reinforcement learning for complex observations, ArXiv Preprint
- 10.1609/aaai.v34i04.5952 · DOI
- matas, 2018, Sim-to-real reinforcement learning for deformable object manipulation, Conference on Robot Learning, 734
- 10.1177/0278364911430417 · DOI
- mnih, 2016, Asynchronous methods for deep reinforcement learning, Proceedings of the 33nd International Conference on Machine Learning ICML
- moll, 2006, Path planning for deformable linear objects, IEEE Transactions on Robotics, 22, 625, 10.1109/TRO.2006.878933 · DOI
- 10.1109/37.463 · DOI
- hafner, 2019, Learning latent dynamics for planning from pixels, Proceedings of the 36th International Conference on Machine Learning ICML 2019 9–15 June 2019 Long Beach California USA volume 97 of Proceedings of Machine Learning Research, 2555
- 10.1109/ACHI.2009.31 · DOI
- 10.1109/ICRA48506.2021.9561980 · DOI
- 10.15607/RSS.2019.XV.039 · DOI
- vaswani, 2017, Attention is all you need, ArXiv Preprint
- 10.15607/RSS.2015.XI.032 · DOI
- kulkarni, 2019, Unsupervised learning of object keypoints for perception and control, Advances in neural information processing systems, 32, 10724
- kipf, 2020, Contrastive learning of structured world models, 8th International Conference on Learning Representations ICLR 2020, 2020
- ciccarelli, 2019, Particle-based fluid simulation with nvidia flex
- 10.15607/RSS.2021.XVII.071 · DOI
- van den oord, 2018, Representation learning with contrastive predictive coding, ArXiv Preprint
- qi, 2017, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, ArXiv Preprint
- qi, 2017, Pointnet: Deep learning on point sets for 3d classification and segmentation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652
- sanchez-gonzalez, 2020, Learning to simulate complex physics with graph networks, International Conference on Machine Learning, 8459
- 10.1109/TRO.2007.907486 · DOI
- seita, 2019, Deep imitation learning of sequential fabric smoothing policies, ArXiv Preprint
Cited by
Sorted by citations
Goal-Conditioned Model Simplification for 1-D and 2-D Deformable Object Manipulation
Wang, S., Leonetti, M., & Dogar, M. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3577052 · Cited by 4
“Other approaches involve using graph neural networks to learn deformable dynamics based on key points or mesh representations [35], [43], [44], [45].”
Metadata
- Authors
- Ma, X., Hsu, D., & Lee, W. S.
- Affiliations
- National University of Singapore (SG)
- ORCID
- Venue
- 2022 International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2022-05-23
- Pages
- 8266-8273
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W4285102216
- Cited by
- 39
- References
- 2
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 16 · 2023: 9 · 2022: 3
- FWCI
- 12.98
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Research Foundation (NRF), Singapore (AISG2-RP-2020-016)
- Agency of Science, Technology & Research, Singapore (19225 00054)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- deformable object manipulation, graph neural networks, self-occlusion, contrastive learning, model predictive control, rope manipulation, fabric manipulation, robotic manipulation, sim-to-real transfer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
