Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Open accessThe DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.
Abstract
Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end-effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors even amid challenging cases of tightly packed clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu/
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Another line of research acquires active training data with simulated or real robots for affordance learning [16, 19, 26, 28, 41].”
From Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception (2023) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2010) · International Conference on Machine Learning · Cited by 13,995
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
V-REP: A versatile and scalable robot simulation framework
Rohmer, E., Singh, S. P. N., & Freese, M. (2013) · 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2013.6696520 · Cited by 1,446
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.058 · Cited by 1,268
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
Zeng, A., Song, S., Niessner, M., Fisher, M., Xiao, J., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.29 · Cited by 1,128
Self-citationRobotic grasping and contact: a review
Bicchi, A., & Kumar, V. (2002) · Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065) · doi:10.1109/robot.2000.844081 · Cited by 1,069
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047
Real-time grasp detection using convolutional neural networks
Redmon, J., & Angelova, A. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139361 · Cited by 893
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Deep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599
Mechanics and Planning of Manipulator Pushing Operations
Mason, M. T. (1986) · The International Journal of Robotics Research · doi:10.1177/027836498600500303 · Cited by 563
Automated construction of robotic manipulation programs
Takeo Kanade, A., Kuffner, J., & Diankov, R. (2010) · Cited by 458
Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge
Zeng, A., Yu, K.-T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989165 · Cited by 457
Self-citationZeng, A., Song, S., Yu, K.-T., Donlon, E., Hogan, F. R., Bauza, M., Ma, D., Taylor, O., Liu, M., Romo, E., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461044 · Cited by 441
Self-citationGoldfeder, C., Ciocarlie, M., Dang, H., & Allen, P. K. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152709 · Cited by 309
High precision grasp pose detection in dense clutter
Gualtieri, M., ten Pas, A., Saenko, K., & Platt, R. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759114 · Cited by 303
Planar sliding with dry friction Part 1. Limit surface and moment function
Goyal, S., Ruina, A., & Papadopoulos, J. (1991) · Wear · doi:10.1016/0043-1648(91)90104-3 · Cited by 283
Rodriguez, A., Mason, M. T., & Ferry, S. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912442972 · Cited by 226
Prattichizzo, D., & Trinkle, J. C. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_29 · Cited by 206
Zeng, A., Song, S., Yu, K.-T., Donlon, E., Hogan, F. R., Bauza, M., Ma, D., Taylor, O., Liu, M., Romo, E., et al. (2022) · The International Journal of Robotics Research · doi:10.1177/0278364919868017 · Cited by 206
Self-citationPose error robust grasping from contact wrench space metrics
Weisz, J., & Allen, P. K. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224697 · Cited by 171
More than a million ways to be pushed. A high-fidelity experimental dataset of planar pushing
Yu, K.-T., Bauza, M., Fazeli, N., & Rodriguez, A. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7758091 · Cited by 158
A Planning Framework for Non-Prehensile Manipulation under Clutter and Uncertainty
Doğar, M. R., & Srinivasa, S. S. (2012) · Autonomous Robots · doi:10.1007/s10514-012-9306-z · Cited by 150
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerík, M., Lampe, T., Tassa, Y., Erez, T., & Riedmiller, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03073 · Cited by 105
Feedback Control of the Pusher-Slider System: A Story of Hybrid and Underactuated Contact Dynamics
Hogan, F. R., & Rodríguez, A. (2020) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-43089-4_51 · Cited by 105
Learning to Manipulate Unknown Objects in Clutter by Reinforcement
Boularias, A., Bagnell, J. A., & Stentz, A. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9378 · Cited by 104
A probabilistic data-driven model for planar pushing
Bauzá, M., & Rodríguez, A. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989345 · Cited by 100
Learning to push by grasping: Using multiple tasks for effective learning
Pinto, L., & Gupta, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989249 · Cited by 99
A convex polynomial force-motion model for planar sliding: Identification and application
Zhou, J., Paolini, R., Bagnell, J. A., & Mason, M. T. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487155 · Cited by 94
Autonomous acquisition of pushing actions to support object grasping with a humanoid robot
Omrčen, D., Böge, C., Asfour, T., Ude, A., & Dillmann, R. (2009) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2009.5379566 · Cited by 49
A Vision-Based Learning Method for Pushing Manipulation
Salganicoff, M., Giorgio Metta, B., Oddera, A., & Sandini, G. (1993) · Scholarly Commons (University of Pennsylvania) · Cited by 45
Policy transfer via modularity
Clavera, I. (2017) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 7
Research works printed but not linked: 1
- 0, Project webpage
Cited by
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Mousavian, A., Eppner, C., & Fox, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00299 · Cited by 571
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Learning ambidextrous robot grasping policies
Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., & Goldberg, K. (2019) · Science Robotics · doi:10.1126/scirobotics.aau4984 · Cited by 563
Metadata
- Authors
- Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T.
- Affiliations
- Princeton University (US)
- Google (United States) (US)
- Venue
- 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2018-10
- Pages
- 4238-4245
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2794537597
- Cited by
- 606
- References
- 44
- Citations by year
- 2026: 24 · 2025: 59 · 2024: 96 · 2023: 100 · 2022: 101 · 2021: 97 · 2020: 78 · 2019: 44 · 2018: 6
- FWCI
- 84.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Keywords
- pushing, grasping, robotic manipulation, self-supervised learning, deep reinforcement learning, Q-learning, end-effector orientation, novel object generalization, model-free reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.