CoChat
Sign Up
PreprintCited by 140Reinforcement Learning in RoboticsRobot Manipulation and LearningSoft Robotics and Applications

SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation

Open access
Lin, Xingyu, Wang, Yufei, Olkin, Jake, Held, David·2020·arXiv
DOI 10.48550/arxiv.2011.07215arXiv 2011.07215
Open this paper in CoChat — free accountOpen PDF

Abstract

Manipulating deformable objects has long been a challenge in robotics due to its high dimensional state representation and complex dynamics. Recent success in deep reinforcement learning provides a promising direction for learning to manipulate deformable objects with data driven methods. However, existing reinforcement learning benchmarks only cover tasks with direct state observability and simple low-dimensional dynamics or with relatively simple image-based environments, such as those with rigid objects. In this paper, we present SoftGym, a set of open-source simulated benchmarks for manipulating deformable objects, with a standard OpenAI Gym API and a Python interface for creating new environments. Our benchmark will enable reproducible research in this important area. Further, we evaluate a variety of algorithms on these tasks and highlight challenges for reinforcement learning algorithms, including dealing with a state representation that has a high intrinsic dimensionality and is partially observable. The experiments and analysis indicate the strengths and limitations of existing methods in the context of deformable object manipulation that can help point the way forward for future methods development. Code and videos of the learned policies can be found on our project website.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.07215

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SoftGym and RFUniverse have the closest simulation features and realism to OMNIGIBSON, but lack the task diversity needed to support human-centered general robots.

    “Similarly, SoftGym [45] and RFUniverse [56] have the closest simulation features and hence realism to OMNIGIBSON, but they also lack the task diversity needed to support the development of human-centered general robots.”

    From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [45]

    The Introduction (p. 1) describes SoftGym as a simulation benchmark with 10 environments focused on rope, cloth, and fluid manipulation. The Reality Gap section (p. 8) reports that simulated cloth behaved similarly to real cloth in a demonstration. The paper does not compare SoftGym with OMNIGIBSON or RFUniverse, or conclude that its task diversity is insufficient for human-centered general robots. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Achieving realism for a diverse set of activities remains a tremendous challenge due to the effort required to provide realistic models and simulation features.

    “While significant progress in realism has been made in specific domains [43-45], achieving realism for a diverse set of activities remains a tremendous challenge, due to the effort required to provide realistic models and simulation features.”

    From BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (2024) · cites this paper as [45]

    The Reality Gap section (p. 8) treats fidelity to real-world manipulation as an open question and says direct simulation-to-real-world transfer remains challenging. But it reports that simulated cloth behaved similarly to real cloth; it does not say that achieving realism across diverse activities is difficult because realistic models and simulation features require substantial effort. (LLM, conf=0.70)

    Experimental — accuracy not yet measured

  • Simulation-trained policies are affected because modern simulators cannot accurately emulate the interaction between the gripper and the CDO.

    “Simulation-trained policies are affected since modern simulators cannot accurately emulate the interaction between the gripper and the CDO [29,34,38].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [29]

    Section 6.4 (p. 8) says direct transfer of learned policies from simulation to reality remains challenging, and Section 4.1 (p. 3) describes SoftGym’s picker as a simplification of a robot gripper. But the paper does not say that modern simulators cannot accurately emulate gripper–deformable-object interactions or identify that as the cause of transfer difficulties; its cloth demonstration instead reports similar simulated and real behavior. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Robust velocity control is still an unsolved challenge in this domain, although few attempt such low-level velocity control to conduct dynamic manipulation.

    “Robust velocity control is still an unsolved challenge in this domain [20,29,226], although few attempt such low-level velocity control to conduct dynamic manipulation [20,29,226,227,228].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [29]

    The Introduction (p. 1) describes deformable-object manipulation as a longstanding challenge. The Action Space section (p. 3) says SoftGym abstracts away low-level manipulation, but neither passage establishes that robust velocity control remains unsolved or that few researchers attempt it for dynamic manipulation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Curl and DrQ have been examined in cloth-shaping tasks in simulation.

    “Curl and DrQ have been examined in cloth-shaping tasks in simulatioan [29]”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [29]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. [2][2] S. Miller, M. Fritz, T. Darrell, and P. Abbeel. Parametrized shape models for clothing. In 2011 IEEE International Conference on Robotics and Automation , pages 4861-4868. IEEE, 2011. -
  2. [3][3] S. Miller, J. Van Den Berg, M. Fritz, T. Darrell, K. Goldberg, and P. Abbeel. A geometric approach to robotic laundry folding. The International Journal of Robotics Research , 31(2): 249-267, 2012. -
  3. [4][4] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015. -
  4. [5][5] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019. -
  5. [6][6] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265 , 2019. - · arXiv
  6. [8][8] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3-20, 2020. -
  7. [9][9] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR , 2019. -
  8. [10][10] J. Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. Conference on Robot Learning (CoRL) , 2018. -
  9. [11][11] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in neural information processing systems , pages 5074-5082, 2016. -
  10. [12][12] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1134-1141. IEEE, 2018. -
  11. [13][13] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. - · arXiv
  12. [14][14] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018. - · arXiv
  13. [15][15] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning (CoRL) , pages 767-782, 2018. -
  14. [16][16] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning , pages 13291338, 2016. -
  15. [17][17] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA) , pages 6023-6029. IEEE, 2019. -
  16. [18][18] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics , pages 481-495. Springer, 2013. -
  17. [19][19] T. L. Chen, M. Ciocarlie, S. Cousins, P. M. Grice, K. Hawkins, K. Hsiao, C. C. Kemp, C.-H. King, D. A. Lazewatsky, A. E. Leeper, et al. Robots for humanity: using assistive robotics to empower people with disabilities. IEEE Robotics & Automation Magazine , 20(1):30-39, 2013. -
  18. [20][20] Z. Erickson, H. M. Clever, G. Turk, C. K. Liu, and C. C. Kemp. Deep haptic model predictive control for robot-assisted dressing. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1-8. IEEE, 2018. -
  19. [21][21] B. Willimon, S. Birchfield, and I. Walker. Model for unfolding laundry using interactive perception. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4871-4876. IEEE, 2011. -
  20. [22][22] D. Triantafyllou and N. A. Aspragathos. A vision system for the unfolding of highly non-rigid objects on a table by one manipulator. In International Conference on Intelligent Robotics and Applications , pages 509-519. Springer, 2011. -
  21. [23][23] M. Saha and P. Isto. Manipulation planning for deformable linear objects. IEEE Transactions on Robotics , 23(6):1141-1150, 2007. -
  22. [24][24] S. Rodriguez, X. Tang, J.-M. Lien, and N. M. Amato. An obstacle-based rapidly-exploring random tree. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. , pages 895-900. IEEE, 2006. -
  23. [25][25] N. Essahbi, B. C. Bouzgarrou, and G. Gogu. Soft material modeling for robotic manipulation. In Applied Mechanics and Materials , volume 162, pages 184-193. Trans Tech Publ, 2012. -
  24. [26][26] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar. Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey. The International Journal of Robotics Research , 37(7):688-716, 2018. -
  25. [27][27] F. F. Khalil and P. Payeur. Dexterous robotic manipulation of deformable objects with multisensory feedback-a review. In Robot Manipulators Trends and Development . IntechOpen, 2010. -
  26. [28][28] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568 , 2018. - · arXiv
  27. [29][29] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 2050-2053, 2018. -
  28. [30][30] M. Yan, Y. Zhu, N. Jin, and J. Bohg. Self-supervised learning of state estimation for manipulating deformable linear objects. IEEE Robotics and Automation Letters , pages 1-1, 2020. ISSN 2377-3774. doi:10.1109/LRA.2020.2969931. - · DOI
  29. [31][31] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, et al. Deep imitation learning of sequential fabric smoothing policies. arXiv preprint arXiv:1910.04854 , 2019. - · arXiv
  30. [33][33] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013. -
  31. [34][34] I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepezvari, S. Singh, et al. Behaviour suite for reinforcement learning. International Conference on Leanring Representation (ICLR) , 2019. -
  32. [35][35] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL) , 2019. -
  33. [36][36] S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark &learning environment. IEEE Robotics and Automation Letters , 2020. -
  34. [37][37] Y. Lee, E. S. Hu, Z. Yang, A. Yin, and J. J. Lim. IKEA furniture assembly environment for long-horizon complex manipulation tasks. arXiv preprint arXiv:1911.07246 , 2019. URL https://clvrai.com/furniture . - · arXiv
  35. [40][40] M. M¨ uller, B. Heidelberger, M. Hennix, and J. Ratcliff. Position based dynamics. Journal of Visual Communication and Image Representation , 18(2):109-118, 2007. -
  36. [41][41] M. Macklin, M. M¨ uller, N. Chentanez, and T.-Y. Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics (TOG) , 33(4):1-12, 2014. -
  37. [42][42] M. Macklin and M. M¨ uller. Position based fluids. ACM Transactions on Graphics (TOG) , 32 (4):1-12, 2013. -
  38. [44][44] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems , pages 9191-9200, 2018. -
  39. [45][45] A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks: Learning generalizable representations for visuomotor control. volume 80 of Proceedings of Machine Learning Research , pages 4732-4741, Stockholmsm¨ assan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/srinivas18b.html . -
  40. [46][46] C. Florensa, J. Degrave, N. Heess, J. T. Springenberg, and M. Riedmiller. Self-supervised learning of image embedding for continuous control. workshop at Advances in Neural Information Processing Systems (NIPS) 2018 , 2019. arXiv:1901.00943. - · arXiv
  41. [47][47] X. Lin, H. S. Baweja, and D. Held. Reinforcement learning without ground-truth state. arXiv preprint arXiv:1905.07866 , 2019. - · arXiv
  42. [48][48] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability , 1(2):127-190, 1999. -
  43. [49][49] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. volume 80 of Proceedings of Machine Learning Research , pages 1861-1870, Stockholmsm¨ assan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/haarnoja18b.html . -
  44. [50][50] Y. Li, J. Wu, J.-Y . Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA) , pages 1205-1211. IEEE, 2019. -
  45. [51][51] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119 , 2020. arXiv:2004.04136. - · arXiv
  46. [53][53] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learming with augmented data. arXiv:2004.14990. - · arXiv
  47. [54][54] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning , pages 2555-2565, 2019. -
  48. [56][56] K. Cho, B. Van Merri¨ enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014. - · arXiv
  49. [57][57] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Lin, X., Wang, Y., Olkin, J., & Held, D.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
432-448
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.07215
arXiv
2011.07215
OpenAlex
W3105162718
Cited by
140
References
3
Citations by year
2025: 10 · 2024: 47 · 2023: 33 · 2022: 30 · 2021: 20
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
Keywords
deformable object manipulation, deep reinforcement learning, benchmark environments, simulation environment, robotic manipulation, reproducible research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review