CoChat
Sign Up
PreprintCited by 241Reinforcement Learning in RoboticsRobot Manipulation and LearningSoft Robotics and Applications

Sim-to-real reinforcement learning for deformable object manipulation

Open access
Matas, Jan, James, Stephen, Davison, Andrew J.·2018·arXiv
DOI 10.48550/arxiv.1806.07851arXiv 1806.07851
Open this paper in CoChat — free accountOpen PDF

Abstract

We have seen much recent progress in rigid object manipulation, but interaction with deformable objects has notably lagged behind. Due to the large configuration space of deformable objects, solutions using traditional modelling approaches require significant engineering work. Perhaps then, bypassing the need for explicit modelling and instead learning the control in an end-to-end manner serves as a better approach? Despite the growing interest in the use of end-to-end robot learning approaches, only a small amount of work has focused on their applicability to deformable object manipulation. Moreover, due to the large amount of data needed to learn these end-to-end solutions, an emerging trend is to learn control policies in simulation and then transfer them over to the real world. To-date, no work has explored whether it is possible to learn and transfer deformable object policies. We believe that if sim-to-real methods are to be employed further, then it should be possible to learn to interact with a wide variety of objects, and not only rigid objects. In this work, we use a combination of state-of-the-art deep reinforcement learning algorithms to solve the problem of manipulating deformable objects (specifically cloth). We evaluate our approach on three tasks --- folding a towel up to a mark, folding a face towel diagonally, and draping a piece of cloth over a hanger. Our agents are fully trained in simulation with domain randomisation, and then successfully deployed in the real world without having seen any real deformable objects.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.07851

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A different solution was proposed in [43], where progressive nets [74] are employed for sim-to-real information transfer.

    “A different solution was proposed in [43], where progressive nets [74] are employed for sim-to-real information transfer.”

    From Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning (2021) · cites this paper as [43]

    The abstract (p. 1) says this paper trained its agents in simulation with domain randomisation, not progressive nets. Related Work (p. 3) describes a separate study requiring additional real-world training; the References (p. 10) identify that study as one using progressive nets. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Robust velocity control is still an unsolved challenge in this domain, although few attempt such low-level velocity control to conduct dynamic manipulation.

    “Robust velocity control is still an unsolved challenge in this domain [20,29,226], although few attempt such low-level velocity control to conduct dynamic manipulation [20,29,226,227,228].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [20]

    Section 5.1 (p. 6) describes a cloth-manipulation system controlled by end-effector and gripper velocities, and the Introduction (p. 1) says relatively little work had applied end-to-end learning to deformable objects. The paper does not establish that robust velocity control remains unsolved or characterize its tasks as dynamic manipulation, so those parts of the claim are unsupported. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The inability to properly simulate the degree of deformability of objects is a main drawback of the simulation environment, with the real robot being unable to grasp stiffer objects.

    “The authors identify as one of the main drawbacks of the simulation environment the inability to properly simulate the degree of deformability of the objects, with the real robot being unable to grasp stiffer objects.”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [6]

    Section 5.3 (p. 8) says the real cloth was stiffer than simulated cloth and that low simulation fidelity contributed to imprecise movements that crumpled it. It separately identifies failed grasps, particularly of thin face towels, as a common problem caused by a narrow margin for gripper height. It does not say stiffness made the robot unable to grasp objects. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • An exception to the limited success of DRL beyond simulation is robotic tasks involving object manipulation.

    “An exception to this is, however, robotic tasks involving object manipulation [5], [6].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [6]

  • Robotic manipulation tasks that have been possible with DRL range from learning peg-in-hole tasks to deformable object manipulation, including more dexterous manipulation with multi-fingered hands, or learning force control policies.

    “Robotic manipulation tasks that have been possible with DRL ranging from learning peg-in-hole tasks [40] to deformable object manipulation [6], and including more dexterous manipulation with multi-fingered hands [5], or learning force control policies [78].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [6]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [2][2] M. Laskey, C. Powers, R. Joshi, A. Poursohi, and K. Goldberg. Learning Robust Bed Making using Deep Imitation Learning with DART. CoRR , 2017. -
  2. [3][3] Y. Gao, H. J. Chang, and Y. Demiris. Iterative path optimisation for personalised dressing assistance using vision and force information. International Conference on Intelligent Robots and Systems , 2016. -
  3. [6][6] J. Schulman, J. Ho, C. Lee, and P. Abbeel. Generalization in Robotic Manipulation Through The Use of Non-Rigid Registration. International Symposium on Robotics Research , 2013. -
  4. [7][7] M. Veˇ cer´ ık, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Roth¨ orl, T. Lampe, and M. Riedmiller. Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards. CoRR , 2017. -
  5. [8][8] Y. Li, Y. Yue, D. Xu, E. Grinspun, and P. Allen. Folding Deformable Objects using Predictive Simulation and Trajectory Optimization. International Conference on Intelligent Robots and Systems , 2015. -
  6. [9][9] M. Cusumano-Towner, A. Singh, S. Miller, J. F. O'Brien, and P. Abbeel. Bringing clothing into desired configurations with limited perception. International Conference on Robotics and Automation , 2011. -
  7. [10][10] Y. Yamakawa, A. Namiki, and M. Ishikawa. Motion planning for dynamic folding of a cloth with two high-speed robot hands and two high-speed sliders. International Conference on Robotics and Automation , 2011. -
  8. [12][12] F. Osawa, H. Seki, and Y. Kamiy. Unfolding of Massive Laundry and Classification Types by Dual Manipulator. Journal of Advanced Computational Intelligence and Intelligent Informatics , 2007. -
  9. [13][13] C. Bersch, B. Pitzer, and S. Kammel. Bimanual robotic cloth manipulation for laundry folding. International Conference on Intelligent Robots and Systems , 2011. -
  10. [14][14] D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine. Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods. CoRR , 2018. -
  11. [15][15] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. CoRR , 2015. -
  12. [16][16] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. International Conference on Intelligent Robots and Systems , 2017. -
  13. [17][17] S. James, A. J. Davison, and E. Johns. Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task. Conference on Robot Learning , 2017. -
  14. [19][19] A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel. Learning from Multiple Demonstrations using Trajectory-Aware Non-Rigid Registration with Applications to Deformable Object Manipulation. International Conference on Intelligent Robots and Systems , 2015. -
  15. [20][20] B. Balaguer and S. Carpin. Combining imitation and reinforcement learning to fold deformable planar objects. International Conference on Intelligent Robots and Systems , 2011. -
  16. [21][21] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates. International Conference on Robotics and Automation , 2016. -
  17. [23][23] P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng. An Application of Reinforcement Learning to Aerobatic Helicopter Flight. International Conference on Neural Information Processing Systems , 2006. -
  18. [24][24] X. B. Peng, G. Berseth, and Y. Kangkang. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Transactions on Graphics , 2017. -
  19. [25][25] S. Fujimoto, H. van Hoof, and D. Meger. Addressing Function Approximation Error in ActorCritic Methods. CoRR , 2018. -
  20. [26][26] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. Mcgrew, J. Tobin, P. Abbeel, and W. Z. Openai. Hindsight Experience Replay. Neural Information Processing Systems Conference , 2017. -
  21. [27][27] G. Barth-Maro, M. W. Hoffma, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, T. Lillicrap, and London. Distributed Distributional Deterministic Policy Gradients. International Conference on Learning Representations , 2018. -
  22. [28][28] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming Exploration in Reinforcement Learning with Demonstrations. International Conference on Robotics and Automation , 2017. -
  23. [29][29] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized Experience Replay. CoRR , 2015. -
  24. [31][31] S. James and E. Johns. 3D Simulation for Robot Arm Control with Deep Q-Learning. Neural Information Processing Systems Conference Workshop , 2016. -
  25. [32][32] A. A. Rusu, M. Vecerik, T. Roth¨ orl, N. Heess, R. Pascanu, and R. Hadsell. Sim-to-Real Robot Learning from Pixels with Progressive Nets. Conference on Robot Learning , 2017. -
  26. [34][34] K. Perlin. An image synthesizer. ACM SIGGRAPH Computer Graphics , 1985. -
  27. [35][35] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR , 2017. -

Cited by

Sorted by citations

Authors
Matas, J., James, S., & Davison, A. J.
Affiliations
Imperial College London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
734-743
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.07851
arXiv
1806.07851
OpenAlex
W2808844346
Cited by
241
References
3
Citations by year
2025: 12 · 2024: 44 · 2023: 36 · 2022: 50 · 2021: 46 · 2020: 34 · 2019: 17 · 2018: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
Keywords
sim-to-real transfer, deformable object manipulation, fabric manipulation, deep reinforcement learning, domain randomization, cloth folding, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review