Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations
Open accessThe DOI matches its Crossref record; 41 of the 45 references it cites were found in a registry or an index.
Abstract
Intelligent manipulation benefits from the capacity to flexibly control an end-effector with high degrees of freedom (DoF) and dynamically react to the environment. However, due to the challenges of collecting effective training data and learning efficiently, most grasping algorithms today are limited to top-down movements and open-loop execution. In this work, we propose a new low-cost hardware interface for collecting grasping demonstrations by people in diverse environments. This data makes it possible to train a robust end-to-end 6DoF closed-loop grasping model with reinforcement learning that transfers to real robots. A key aspect of our grasping model is that it uses “action-view” based rendering to simulate future states with respect to different possible actions. By evaluating these states using a learned value function (e.g., Q-function), our method is able to better select corresponding actions that maximize total rewards (i.e., grasping success). Our final grasping system is able to achieve reliable 6DoF closed-loop grasping of novel objects across various scene configurations, as well as in dynamic scenes with moving objects.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While these approaches simplify tool design, they still require a system-specific tool or modifications to a robot's original hardware to align the end-effector's physical interactions and the recorded visual observations.”
From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [11]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Compared to other methods of aggregating past observations such as using recurrent neural networks or LSTMs [38], our TSDF fusion explicitly leverages accurate industrial-grade robot motion in order to reduce the burden of learning view point registration or 3D reconstruction inside the network.”
MethodDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“The sign of that value indicates whether the voxel is in free space or occluded space [4, 22, 35].”
BackgroundA volumetric method for building complex models from range images
Curless, B., & Levoy, M. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237269 · Cited by 3,124 · cited 1×
“The sign of that value indicates whether the voxel is in free space or occluded space [35]–[37].”
BackgroundMethod for registration of 3-D shapes:
Besl, P. J., & McKay, N. D. (1992) · SPIE Proceedings · doi:10.1117/12.57955 · Cited by 2,413
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · cited 2×
“vily depend on acquiring high-quality training data. However, most prior self-supervised grasping systems are often constrained to learning in simulation [30,18,19, 27] or structured lab environments [23,36,15,14]. Gupta et al. [9] improves the data collection process by physically moving a robot into different enviro…”
BackgroundMethodDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634 · cited 2×
“With human experts directly annotating the training data [2], [22] or controlling the robot via teleoperation [7], [26], the system can quickly obtain positive examples to speed up the training process.”
BackgroundMahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.058 · Cited by 1,268
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
Zeng, A., Song, S., Niessner, M., Fisher, M., Xiao, J., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.29 · Cited by 1,128
Self-citationSupersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047 · cited 4×
“in simple tabletop settings [23, 37, 36], or detect grasps in 6DoF but with open-loop execution [17, 21, 7].”
BackgroundInfluentialReal-time grasp detection using convolutional neural networks
Redmon, J., & Angelova, A. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139361 · Cited by 893 · cited 1×
“More recent data-driven methods explore the prospects of training object-agnostic grasping policies that detect grasps by exploiting learned visual features, without explicitly using object-specific knowledge [1], [2], [5], [6], [12]–[16].”
BackgroundSUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels
Xiao, J., Owens, A., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.458 · Cited by 803 · cited 1×
“To recover 6DoF grasping trajectories from the RGB-D videos of demonstrations, we use classic frame-to-frame visual tracking [30] to estimate the camera pose and trajectory over time.”
MethodQT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697 · cited 7×
“For example, QT-Opt [13] learns only topdown grasping policies (action space) with images from a fixed static camera (state space).”
BackgroundMethodInfluentialClosing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach
Morrison, D., Leitner, J., & Corke, P. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.021 · Cited by 630
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P. (2017) · arXiv · doi:10.48550/arxiv.1710.04615 · Cited by 618
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461249 · Cited by 609
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593986 · Cited by 606
Self-citation6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Mousavian, A., Eppner, C., & Fox, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00299 · Cited by 571 · cited 2×
“Most methods only infer top-down grasps (4Dof: x, y, z, yaw) in simple tabletop settings [1]–[4], or detect grasps in 6DoF but with open-loop execution [5], [6].”
BackgroundMethodMulti-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge
Zeng, A., Yu, K.-T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989165 · Cited by 457 · cited 1×
“Classic vision-based grasping solutions often explicitly model contact forces with prior knowledge of object geometry, pose, and dynamics [8]–[11].”
BackgroundSelf-citationZeng, A., Song, S., Yu, K.-T., Donlon, E., Hogan, F. R., Bauza, M., Ma, D., Taylor, O., Liu, M., Romo, E., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461044 · Cited by 441
Self-citationMahler, J., Pokorny, F. T., Hou, B., Roderick, M., Laskey, M., Aubry, M., Kohlhoff, K., Kroger, T., Kuffner, J., & Goldberg, K. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487342 · Cited by 359 · cited 1×
“However, most prior self-supervised grasping systems are often constrained to learning in simulation [15], [17], [20], [24] or structured lab environments [1], [3], [21], [22].”
BackgroundGoldfeder, C., Ciocarlie, M., Dang, H., & Allen, P. K. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152709 · Cited by 309 · cited 1×
“Classic vision-based grasping solutions often explicitly model contact forces with prior knowledge of object geometry, pose, and dynamics [25, 31, 6, 38].”
BackgroundHigh precision grasp pose detection in dense clutter
Gualtieri, M., ten Pas, A., Saenko, K., & Platt, R. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759114 · Cited by 303 · cited 2×
“[18] 7 7 simulation [23, 37, 36, 15, 14] 7 7 real [8, 17, 21, 7] 7 3 simulation [30, 19] 3 7 simulation [20, 13] 3 7 real Ours 3 3 real”
BackgroundClear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation
Sajjan, S., Moore, M., Pan, M., Nagaraja, G., Lee, J., Zeng, A., & Song, S. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197518 · Cited by 279 · cited 1×
“However, most prior self-supervised grasping systems are often constrained to learning in simulation [15], [17], [20], [24] or structured lab environments [1], [3], [21], [22].”
BackgroundSelf-citationDeep Q-learning from Demonstrations
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03732 · Cited by 270
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00568 · Cited by 235 · cited 1×
“This action-view representation is inspired by prior work, which use predictive models to improve the sample efficiency of reinforcement learning algorithms [33], [34].”
MethodPrattichizzo, D., & Trinkle, J. C. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_29 · Cited by 206
6-DOF Grasping for Target-driven Object Manipulation in Clutter
Murali, A., Mousavian, A., Eppner, C., Paxton, C., & Fox, D. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197318 · Cited by 202
Pose error robust grasping from contact wrench space metrics
Weisz, J., & Allen, P. K. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224697 · Cited by 171 · cited 1×
“Classic vision-based grasping solutions often explicitly model contact forces with prior knowledge of object geometry, pose, and dynamics [25, 31, 6, 38].”
BackgroundLearning a visuomotor controller for real world robotic grasping using simulated depth images
Viereck, U., ten Pas, A., Saenko, K., & Platt, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04652 · Cited by 144 · cited 3×
“However, most prior self-supervised grasping systems are often constrained to learning in simulation [15], [17], [20], [24] or structured lab environments [1], [3], [21], [22].”
BackgroundForm2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
Zakka, K., Zeng, A., Lee, J., & Song, S. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196733 · Cited by 108 · cited 1×
“It would be interesting to investigate how to make use of the other information captured in this dataset, such as picking order and placing trajectories [34].”
BackgroundSelf-citationLearning to push by grasping: Using multiple tasks for effective learning
Pinto, L., & Gupta, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989249 · Cited by 99 · cited 1×
“More recent data-driven methods explore the prospects of training object-agnostic grasping policies that detect grasps by exploiting learned visual features, without explicitly using object-specific knowledge [26, 23, 24, 8, 18, 37, 17, 21, 7].”
BackgroundPlanning Multi-fingered Grasps as Probabilistic Inference in a Learned Deep Network
Lu, Q., Chenna, K., Sundaralingam, B., & Hermans, T. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_35 · Cited by 92
DensePhysNet: Learning Dense Physical Object Representations Via Multi-Step Dynamic Interactions
Xu, Z., Wu, J., Zeng, A., Tenenbaum, J., & Song, S. (2019) · Robotics: Science and Systems XV · doi:10.15607/rss.2019.xv.046 · Cited by 86
Self-citationGeneralization of human grasping for multi-fingered robot hands
Ben Amor, H., Kroemer, O., Hillenbrand, U., Neumann, G., & Peters, J. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386072 · Cited by 83 · cited 1×
“On the other hand, recording videos of direct interactions between human hands and objects does not require expert knowledge from the subject [27], [28].”
BackgroundLearning to See before Learning to Act: Visual Pre-training for Manipulation
Yen-Chen, L., Zeng, A., Song, S., Isola, P., & Lin, T.-Y. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197331 · Cited by 64 · cited 1×
“Most methods only infer top-down grasps (4Dof: x, y, z, yaw) in simple tabletop settings [1]–[4], or detect grasps in 6DoF but with open-loop execution [5], [6].”
MethodSelf-citationRobot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Gupta, A., Murali, A., Gandhi, D., & Pinto, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07049 · Cited by 56 · cited 1×
“[25] improves the data collection process by physically moving a robot into different environments.”
BackgroundLearning 6-DoF Grasping and Pick-Place Using Attention FocusPreprint
Gualtieri, M., & Platt, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06134 · Cited by 52 · cited 1×
“More recent data-driven methods explore the prospects of training object-agnostic grasping policies that detect grasps by exploiting learned visual features, without explicitly using object-specific knowledge [1], [2], [5], [6], [12]–[16].”
BackgroundMultiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
Sharma, P., Mohan, L., Pinto, L., & Gupta, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07121 · Cited by 31 · cited 1×
“With human experts directly annotating the training data [2], [22] or controlling the robot via teleoperation [7], [26], the system can quickly obtain positive examples to speed up the training process.”
BackgroundCharacterizing Input Methods for Human-to-Robot Demonstrations
Praveena, P., Subramani, G., Mutlu, B., & Gleicher, M. L. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673310 · Cited by 22 · cited 1×
“[29] also developed a similar handheld grabber tool as our setup, where the tool is equipped with force-torque sensor, and its movement is tracked with a Optitrack motion capture system.”
MethodPre-grasp Interaction for Object Acquisition in Difficult Tasks
Chang, L., & Pollard, N. S. (2014) · Springer tracts in advanced robotics · doi:10.1007/978-3-319-03017-3_23 · Cited by 1 · cited 1×
“On the other hand, recording videos of direct interactions between human hands and objects does not require expert knowledge from the subject [27], [28].”
Background
Research works printed but not linked: 2
- hester, 0, Proc Assoc Advance Artif Intell, 3223
- besl, 1992, Sensor Fusion IV
Cited by
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
Wu, P., Shentu, Y., Yi, Z., Lin, X., & Abbeel, P. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801581 · Cited by 409
“The focus for the design of GELLO is to create an interface that is both economically accessible and easy-to-use for users aiming to render high-quality demonstrations in robot learning.”
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Wang, C., Shi, H., Wang, W., Zhang, R., Fei-Fei, L., & Liu, C. K. (2024) · Robotics: Science and Systems · doi:10.15607/rss.2024.xx.043 · Cited by 396
(2021) · doi:10.1109/icra48506.2021.9561877 · Cited by 370
“To address the limitations of planar grasping, there has been a recent interest in tackling the problem of 6-DoF grasping of unknown objects [10, 11, 12, 13, 14].”
BackgroundDexMV: Imitation Learning for Dexterous Manipulation from Human Videos
Qin, Y., Wu, Y.-H., Liu, S., Jiang, H., Yang, R., Fu, Y. L., & Wang, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_33 · Cited by 345
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Simeonov, A., Du, Y., Tagliasacchi, A., Tenenbaum, J. B., Rodriguez, A., Agrawal, P., & Sitzmann, V. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812146 · Cited by 148
Metadata
- Authors
- Song, S., Zeng, A., Lee, J., & Funkhouser, T.
- Affiliations
- Google (United States) (US)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2020-07
- Volume
- 5
- Issue
- 3
- Pages
- 4978-4985
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W3037767967
- Cited by
- 185
- References
- 45
- Citations by year
- 2026: 12 · 2025: 33 · 2024: 35 · 2023: 44 · 2022: 32 · 2021: 18 · 2020: 11
- FWCI
- 12.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Keywords
- end-effector control, human demonstrations, reinforcement learning, Q-function, dynamic scenes, novel object grasping, robotic manipulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.