CoChat
Sign Up
Cited by 15Robot Manipulation and LearningReinforcement Learning in RoboticsSoft Robotics and Applications

Bag All You Need: Learning a Generalizable Bagging Strategy for Heterogeneous Objects

Open access
Bahety, Arpit, Jain, Shreeya, Ha, Huy, Hager, Nathalie, Burchfiel, Benjamin, Cousineau, Eric, Feng, Siyuan, Song, Shuran·2023·2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
DOI 10.1109/iros55552.2023.10341841
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a practical robotics solution for the task of heterogeneous bagging, requiring the placement of multiple rigid and deformable objects into a deformable bag. This is a difficult task as it features complex interactions between multiple highly deformable objects under limited observability. To tackle these challenges, we propose a robotic system consisting of two learned policies: a rearrangement policy that learns to place multiple rigid objects and fold deformable objects in order to achieve desirable pre-bagging conditions, and a lifting policy to infer suitable grasp points for bi-manual bag lifting. We evaluate these learned policies on a real-world three-arm robot platform that achieves a 70% heterogeneous bagging success rate with novel objects. To facilitate future research and comparison, we also develop a novel heterogeneous bagging simulation benchmark that will be made publicly available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iros55552.2023.10341841

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Advancements in complex 3D deformable bag manipulation do not incorporate an active movable camera for task achievement.

    “Notably, advancements in complex 3D deformable bag manipulation have been presented [27], [28], [29], [30], [31]. These works have shown significant progress in the domain but do not incorporate an active movable camera for task achievement.”

    From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [28]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. D. Seita, J. Kerr, J. Canny, and K. Goldberg, "Initial results on grasping and lifting physical deformable bags with a bimanual robot," in IROS Workshop on Robotic Manipulation of Deformable Objects in Real-world Applications, vol. 2, 2021, p. 3.
  2. H. Zhao, Q. She, C. Zhu, Y. Yang, and K. Xu, "Online 3d bin packing with constrained deep reinforcement learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 1, 2021, pp. 741-749.
  3. H. Zhao, C. Zhu, X. Xu, H. Huang, and K. Xu, "Learning practically feasible policies for online 3d bin packing," Science China Information Sciences, vol. 65, no. 1, pp. 1-17, 2022.
  4. K. Zakka, A. Zeng, J. Lee, and S. Song, "Form2fit: Learning shape priors for generalizable assembly from disassembly," in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 9404-9410.
  5. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al., "Transporter networks: Rearranging the visual world for robotic manipulation," arXiv preprint arXiv:2010.14406, 2020. · arXiv
  6. S. Martello and D. Vigo, "Exact solution of the two-dimensional finite bin packing problem," Management science, vol. 44, no. 3, pp. 388-399, 1998.
  7. B. S. Baker, E. G. Coffman, Jr, and R. L. Rivest, "Orthogonal packings in two dimensions," SIAM Journal on computing, vol. 9, no. 4, pp. 846-855, 1980.
  8. D. S. Johnson, A. Demers, J. D. Ullman, M. R. Garey, and R. L. Graham, "Worst-case performance bounds for simple one-dimensional packing algorithms," SIAM Journal on computing, vol. 3, no. 4, pp. 299-325, 1974.
  9. F. Wang and K. Hauser, "Robot packing with known items and nondeterministic arrival order," IEEE Transactions on Automation Science and Engineering, vol. 18, no. 4, pp. 1901-1915, 2020.
  10. M. Saha and P. Isto, "Manipulation planning for deformable linear objects," IEEE Transactions on Robotics, vol. 23, no. 6, pp. 1141-1150, 2007.
  11. K. Suzuki, M. Kanamura, Y. Suga, H. Mori, and T. Ogata, "In-air knotting of rope using dual-arm robot based on deep learning," in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 6724-6731.
  12. C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, "Iterative residual policy for goal-conditioned dynamic manipulation of deformable objects," in Proceedings of Robotics: Science and Systems (RSS), 2022.
  13. A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine, "Combining self-supervised learning and imitation for vision-based rope manipulation," in 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 2017, pp. 2146-2153.
  14. J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, "Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey," The International Journal of Robotics Research, vol. 37, no. 7, pp. 688-716, 2018.
  15. C. Bersch, B. Pitzer, and S. Kammel, "Bimanual robotic cloth manipu- lation for laundry folding," in 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2011, pp. 1413-1419.
  16. F. Osawa, H. Seki, and Y. Kamiya, "Unfolding of massive laundry and classification types by dual manipulator," Journal of Advanced Computational Intelligence and Intelligent Informatics, vol. 11, no. 5, pp. 457-463, 2007.
  17. M. Cusumano-Towner, A. Singh, S. Miller, J. F. O'Brien, and P. Abbeel, "Bringing clothing into desired configurations with limited perception," in 2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 3893-3900.
  18. A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali et al., "Learning dense visual correspondences in simulation to smooth and fold real fabrics," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 11 515-11 522.
  19. X. Lin, Y. Wang, Z. Huang, and D. Held, "Learning visible connectivity dynamics for cloth smoothing," in Conference on Robot Learning. PMLR, 2022, pp. 256-266.
  20. F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine, "Visual foresight: Model-based deep reinforcement learning for vision-based robotic control," arXiv preprint arXiv:1812.00568, 2018. · arXiv
  21. J. Matas, S. James, and A. J. Davison, "Sim-to-real reinforcement learning for deformable object manipulation," in Conference on Robot Learning. PMLR, 2018, pp. 734-743.
  22. H. Ha and S. Song, "Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding," in Conference on Robot Learning. PMLR, 2022, pp. 24-33.
  23. D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali et al., "Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor," in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 9651-9658.
  24. A. M. Howard and G. A. Bekey, "Intelligent learning for deformable ob- ject manipulation," Autonomous Robots, vol. 9, no. 1, pp. 51-58, 2000.
  25. Z. Xu, C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, "Dextairity: Deformable manipulation can be a breeze," arXiv preprint arXiv:2203.01197, 2022. · arXiv
  26. X. Lin, Y. Wang, J. Olkin, and D. Held, "Softgym: Benchmarking deep reinforcement learning for deformable object manipulation," CoRR, vol. abs/2011.07215, 2020. [Online]. Available: https://arxiv.org/abs/2011.07215 · arXiv
  27. Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba, "Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids," in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=rJgbSn09Ym
  28. B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, "Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set," IEEE Robotics & Automation Magazine, vol. 22, no. 3, pp. 36-52, 2015.
  29. B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar, "Yale-cmu-berkeley dataset for robotic manipulation research," The International Journal of Robotics Research, vol. 36, no. 3, pp. 261-268, 2017.
  30. H. Bertiche, M. Madadi, and S. Escalera, "Cloth3d: Clothed 3d humans," in European Conference on Computer Vision. Springer, 2020, pp. 344-359.
  31. C. H. Sudre, W. Li, T. K. M. Vercauteren, S. Ourselin, and M. J. Cardoso, "Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations," Deep learning in medical image analysis and multimodal learning for clinical decision support : Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, held in conjunction with MICCAI 2017 Quebec City, QC,..., vol. 2017, pp. 240-248, 2017.
  32. J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser, "Spatial action maps for mobile manipulation," arXiv preprint arXiv:2004.09141, 2020. · arXiv
  33. A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, "Tossingbot: Learning to throw arbitrary objects with residual physics," IEEE Transactions on Robotics, vol. 36, no. 4, pp. 1307-1319, 2020.
  34. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980 · arXiv
  35. L. Y. Chen, B. Shi, D. Seita, R. Cheng, T. Kollar, D. Held, and K. Goldberg, "Autobag: Learning to open plastic bags and insert objects," 2022. [Online]. Available: https://arxiv.org/abs/2210.17217 · arXiv

Cited by

Sorted by citations

Authors
Bahety, A., Jain, S., Ha, H., Hager, N., Burchfiel, B., Cousineau, E., Feng, S., & Song, S.
Affiliations
  • Columbia University (US)
  • Toyota Research Institute (US)
ORCID
Venue
2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Publisher
IEEE
Published
2023-10-01
Pages
960-967
ISSN
2153-0858 (ISSN-L)
DOI
10.1109/iros55552.2023.10341841
OpenAlex
W4389667186
Cited by
15
References
6
Citations by year
2026: 2 · 2025: 5 · 2024: 5 · 2023: 3
FWCI
6.42
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Toyota Research Institute
  • NSF (2143601,2037101,2132519)
  • Google
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
Keywords
deformable objects, rigid bodies, object rearrangement, simulation benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review