Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection
Open accessAbstract
We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images independent of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. We describe two large-scale experiments that we conducted on two separate robotic platforms. In the first experiment, about 800,000 grasp attempts were collected over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and gripper wear and tear. In the second experiment, we used a different robotic platform and 8 robots to collect a dataset consisting of over 900,000 grasp attempts. The second robotic platform was used to test transfer between robots, and the degree to which data from a different set of robots can be used to aid learning. Our experimental results demonstrate that our approach achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. Our transfer experiment also illustrates that data from different robots can be combined to learn more reliable and effective grasping.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More reliable and effective hand-eye coordination for robotic grasping can be learned through correcting errors.
“Levine et al. [100] found that more reliable and effective hand-eye coordination for robotic grasping can be learned through correcting errors.”
From A Taxonomy of Social Errors in Human-Robot Interaction (2021) · cites this paper as [100]
None of the related concepts fully addresses all three aspects in a single framework: a deformable object, a moving camera, and a robot interacting with the object.
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot interacting with the object.”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [11]
The work in [11] learns hand-eye coordination for grasping but its application is bin-picking, using a static camera, and no deformable objects are considered.
“However, the application is bin-picking, using a static camera, and no deformable objects are considered.”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [11]
Cites
Learning grasp affordance densities
Detry, R., Kraft, D., Kroemer, O., Bodenhagen, L., Peters, J., Krüger, N., & Piater, J. (2011) · Paladyn Journal of Behavioral Robotics · doi:10.2478/s13230-011-0012-x · Cited by 79 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Learning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643 · see citing works
Learning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861 · see citing works
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763 · see citing works
Overcoming Exploration in Reinforcement Learning with Demonstrations
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463162 · Cited by 712 · see citing works
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · see citing works
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460875 · Cited by 636 · see citing works
Self-citationThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Self-citationJames, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · see citing works
Self-citationSAPIEN: A SimulAted Part-Based Interactive ENvironment
Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01111 · Cited by 406 · see citing works
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387 · see citing works
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“A promising approach for enabling generalization in robotic manipulation is by learning from large and diverse datasets [31, 32, 33].”
BackgroundSelf-citationPreparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · see citing works
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations
Song, S., Zeng, A., Lee, J., & Funkhouser, T. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004787 · Cited by 185 · see citing works
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., & Dresp-Langley, B. (2021) · Robotics · doi:10.3390/robotics10010022 · Cited by 188 · see citing works
A Taxonomy of Social Errors in Human-Robot Interaction
Tian, L., & Oviatt, S. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3439720 · Cited by 133 · see citing works
Adversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206245 · Cited by 116 · see citing works
Octo: An Open-Source Generalist Robot Policy
Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., Luo, J., et al. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.090 · Cited by 129 · see citing works
Self-citationHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129 · see citing works
MineRL: A Large-Scale Dataset of Minecraft Demonstrations
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/339 · Cited by 94 · see citing works
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation
Gordon, D., Kadian, A., Parikh, D., Hoffman, J., & Batra, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00111 · Cited by 57 · see citing works
Sim-to-Real Transfer for Optical Tactile Sensing
Ding, Z., Lepora, N. F., & Johns, E. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197512 · Cited by 53 · see citing works
DayDreamer: World Models for Physical Robot Learning
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., & Abbeel, P. (2022) · arXiv · doi:10.48550/arxiv.2206.14176 · Cited by 76 · see citing works
“One class of methods attempts to scale up experience collection through a fleet of robots (Kalashnikov et al., 2018; 2021; Ebert et al., 2021; Dasari et al., 2019; Levine et al., 2018).”
BackgroundModel-free and learning-free grasping by Local Contact Moment matching
Adjigble, M., Marturi, N., Ortenzi, V., Rajasekaran, V., Corke, P., & Stolkin, R. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594226 · Cited by 44 · see citing works
Benchmarking Protocol for Grasp Planning Algorithms
Bekiroglu, Y., Marturi, N., Roa, M. A., Adjigble, K. J. M., Pardi, T., Grimm, C., Balasubramanian, R., Hang, K., & Stolkin, R. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2956411 · Cited by 41 · see citing works
Language-driven Grasp Detection
Vuong, A. D., Vu, M. N., Huang, B., Nguyen, N., Le, H., Vo, T., & Nguyen, A. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01695 · Cited by 39 · see citing works
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34 · see citing works
Human Hands as Probes for Interactive Object Understanding
Goyal, M., Modi, S., Goyal, R., & Gupta, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00329 · Cited by 31 · see citing works
An assisted telemanipulation approach: combining autonomous grasp planning with haptic cues
Adjigble, M., Marturi, N., Ortenzi, V., & Stolkin, R. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968454 · Cited by 19 · see citing works
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.24164 · Cited by 13 · see citing works
“Early work in this area often utilized self-supervised or autonomous data collection [26, 22, 8], providing a tractable data source for simple tasks such as grasping [18, 37] or pushing [56], but without the complexity of more dexterous behaviors.”
Self-citationMoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611121 · Cited by 8 · see citing works
“…learning on real hardware, most of which fall into one of three categories: learning from human demonstrations [36], [37], [38], [39], learning from large uncurated datasets [40], [41], [20], learning from online interaction via RL [42], [43], or any combination of them [17], [44], [45], [46].”
BackgroundInteractive Perception for Deformable Object Manipulation
Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3431943 · Cited by 7 · see citing works
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot…”
BackgroundThe Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Iacob, A., Jovanović, A., Shen, W. F., Burkhardt, D., Kurmanji, M., Tastan, N., Sani, L., Venanzi, N. A. E., Nie, J., Odonnat, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.26294 · Cited by 1 · see citing works
“…task has no direct benchmark, for example paper writing and proof writing lack one while paper re-viewing and proof grading do not [5–7]; when evaluation is slow or weakly informative [8–10]; and when static benchmarks saturate or become vulnerable to reward hacking as agents improve [11, 12].”
- Authors
- Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D.
- Affiliations
- Google (United States) (US)
- X
- Corresponding author
- Levine, Sergey
- ORCID
- Levine, Sergey
- Venue
- The International Journal of Robotics Research
- Publisher
- SAGE Publications
- Published
- 2017-06-12
- Volume
- 37
- Issue
- 4-5
- Pages
- 421-436
- ISSN
- 0278-3649 (ISSN-L)
- 1741-3176
- OpenAlex
- W2293467699
- Cited by
- 1,971
- References
- 1
- Citations by year
- 2026: 80 · 2025: 119 · 2024: 173 · 2023: 178 · 2022: 237 · 2021: 310 · 2020: 278 · 2019: 289 · 2018: 163 · 2017: 117 · 2016: 25 · 2012: 1
- FWCI
- 145.50
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The International Journal of Robotics Research (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- robotic grasping, eye-hand coordination, monocular vision, convolutional neural networks, novel object grasping, error correction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
