More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model - a deep, multimodal convolutional network - predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at 1) estimating grasp adjustment outcomes, 2) selecting efficient grasp adjustments for quick grasping, and 3) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,464
(2011) · Cited by 2,752
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Coding and use of tactile signals from the fingertips in object manipulation tasks
Johansson, R. S., & Flanagan, J. R. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2621 · Cited by 2,063
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Self-citationDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
Yuan, W., Dong, S., & Adelson, E. (2017) · Sensors · doi:10.3390/s17122762 · Cited by 1,166
Self-citationSupersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047
Cited by
Learning ambidextrous robot grasping policies
Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., & Goldberg, K. (2019) · Science Robotics · doi:10.1126/scirobotics.aau4984 · Cited by 563
Lambeta, M., Chou, P.-W., Tian, S., Yang, B., Maloon, B., Most, V. R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., et al. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2977257 · Cited by 553
Self-citationLee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349
“Some of them extend their previous work on grasp stability estimation [5, 13].”
BackgroundA Review of Tactile Information: Perception and Action Through Touch
Li, Q., Kroemer, O., Su, Z., Veiga, F. F., Kaboli, M., & Ritter, H. J. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.3003230 · Cited by 332
Review of machine learning methods in soft robotics
Kim, D., Kim, S.-H., Kim, T., Kang, B. B., Lee, M., Park, W., Ku, S., Kim, D., Kwon, J., Lee, H., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0246102 · Cited by 256
Deep Learning Approaches to Grasp Synthesis: A Review
Newbury, R., Gu, M., Chumbley, L., Mousavian, A., Eppner, C., Leitner, J., Bohg, J., Morales, A., Asfour, T., Kragic, D., et al. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3280597 · Cited by 255
Review of Deep Learning Methods in Robotic Grasp Detection
Caldera, S., Rassau, A., & Chai, D. (2018) · Multimodal Technologies and Interaction · doi:10.3390/mti2030057 · Cited by 219
“to include the 3D depth information by adding the z coordinates to the representation, resulting in a grasp representation, Gz = (x, y, z, θ) [40,41].”
BackgroundTactile Image Sensors Employing Camera: A Review
Shimonomura, K. (2019) · Sensors · doi:10.3390/s19183933 · Cited by 187
Object detection recognition and robot grasping based on machine learning: a survey
Bai, Q., Li, S., Yang, J., Song, Q., Li, Z., & Zhang, X. (2020) · IEEE Access · doi:10.1109/access.2020.3028740 · Cited by 154
“The research direction is mainly divided into single tactile object perception [199]–[203] and vision-tactile fusion [204]–[207] object recognition and grabbing.”
BackgroundA survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90
“Calandra et al. [15] proposed an end-to-end action condition model that can learn grasping strategies from the original touch-visual data.”
In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
Suzuki, K., Kanamura, M., Suga, Y., Mori, H., & Ogata, T. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635954 · Cited by 31
“In addition, we trained DNNs to learn the relationship between the state of the rope as surmised based on the combined sensory information and the corresponding motion of the robot.”
MethodInfluentialSee to touch: Learning tactile dexterity through visual incentives
Guzey, I., Dai, Y., Evans, B., Chintala, S., & Pinto, L. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611407 · Cited by 26
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Lee, M. A., Tan, M., Zhu, Y., & Bohg, J. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561847 · Cited by 24
“Several works have combined visual and tactile information for object tracking [18, 27, 39], shape completion [36], grasp assessment [4, 7, 11, 32], and scene understanding [5, 6].”
BackgroundGrasp Stability Assessment Through Spatio-Temporal Attention Mechanism and Adaptive Gate Fusion
Li, S., Sun, W., Liang, Q., Sun, J., & Liu, C. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3493117 · Cited by 13
Metadata
- Authors
- Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2018-10
- Volume
- 3
- Issue
- 4
- Pages
- 3300-3307
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2804941773
- Cited by
- 359
- References
- 40
- FWCI
- 20.48
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.