The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Open accessAbstract
For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model - a deep, multimodal convolutional network - predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at 1) estimating grasp adjustment outcomes, 2) selecting efficient grasp adjustments for quick grasping, and 3) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
(2011) · Cited by 2,752
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Coding and use of tactile signals from the fingertips in object manipulation tasks
Johansson, R. S., & Flanagan, J. R. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2621 · Cited by 2,063
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Self-citationDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
Yuan, W., Dong, S., & Adelson, E. (2017) · Sensors · doi:10.3390/s17122762 · Cited by 1,166
Self-citationSupersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047
Data-Driven Grasp Synthesis—A Survey
Bohg, J., Morales, A., Asfour, T., & Kragic, D. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2013.2289018 · Cited by 929
Tactile sensing for dexterous in-hand manipulation in robotics—A review
Yousef, H., Boukallel, M., & Althoefer, K. (2011) · Sensors and Actuators A: Physical · doi:10.1016/j.sna.2011.02.038 · Cited by 784
Robot Grasp Synthesis Algorithms: A Survey
Shimoga, K. B. (1996) · The International Journal of Robotics Research · doi:10.1177/027836499601500302 · Cited by 666
1Data-Driven Grasp Synthesis- A Survey
Bohg, J., Morales, A., Asfour, T., & Kragić, D. (2014) · MPG.PuRe (Max Planck Society) · Cited by 532
Human-Inspired Robotic Grasp Control With Tactile Sensing
Romano, J. M., Hsiao, K., Niemeyer, G., Chitta, S., & Kuchenbecker, K. J. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2162271 · Cited by 413
Mahler, J., Pokorny, F. T., Hou, B., Roderick, M., Laskey, M., Aubry, M., Kohlhoff, K., Kroger, T., Kuffner, J., & Goldberg, K. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487342 · Cited by 359
Measurement of shear and slip with a GelSight tactile sensor
Yuan, W., Li, R., Srinivasan, M. A., & Adelson, E. H. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139016 · Cited by 343
Self-citationRetrographic sensing for the measurement of surface texture and shape
Johnson, M. K., & Adelson, E. H. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206534 · Cited by 307
Self-citationLeveraging big data for grasp planning
Kappler, D., Bohg, J., & Schaal, S. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139793 · Cited by 268
Deep learning a grasp function for grasping under gripper pose uncertainty
Johns, E., Leutenegger, S., & Davison, A. J. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759657 · Cited by 243
Rodriguez, A., Mason, M. T., & Ferry, S. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912442972 · Cited by 226
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?Preprint
Calandra, R., Owens, A., Upadhyaya, M., Yuan, W., Lin, J., Adelson, E. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05512 · Cited by 197
Self-citationAssessing Grasp Stability Based on Learning and Haptic Data
Bekiroglu, Y., Laaksonen, J., Jorgensen, J. A., Kyrki, V., & Kragic, D. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2132870 · Cited by 189
Shape-independent hardness estimation using deep learning and a GelSight tactile sensor
Yuan, W., Zhu, C., Owens, A., Srinivasan, M. A., & Adelson, E. H. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989116 · Cited by 177
Self-citationConnecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials
Yuan, W., Wang, S., Dong, S., & Adelson, E. H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.478 · Cited by 167
Self-citationShape-independent hardness estimation using deep learning and a GelSight tactile sensor
Yuan, W., Zhu, C., Owens, A., Srinivasan, M. A., & Adelson, E. H. (2017) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 154
Self-citationStabilizing novel objects by learning to predict tactile slip
Veiga, F. F., van Hoof, H., Peters, J., & Hermans, T. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354090 · Cited by 130
Stable grasping under pose uncertainty using tactile feedback
Dang, H., & Allen, P. K. (2013) · Autonomous Robots · doi:10.1007/s10514-013-9355-y · Cited by 120
Goldfeder, C., & Allen, P. K. (2011) · Autonomous Robots · doi:10.1007/s10514-011-9228-1 · Cited by 120
Learning to grasp using visual information
Kamon, I., Flash, T., & Edelman, S. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1996.506534 · Cited by 119
Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations
Hogan, F. R., Bauzá, M., Canal, O., Donlon, E., & Rodríguez, A. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593528 · Cited by 108
Learning of grasp adaptation through experience and tactile sensing
Li, M., Bekiroglu, Y., Kragić, D., & Billard, A. (2014) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2014.6943027 · Cited by 107
Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
Chebotar, Y., Hausman, K., Su, Z., Sukhatme, G. S., & Schaal, S. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759309 · Cited by 80
Robotic grasping using visual and tactile sensing
Guo, D., Sun, F., Fang, B., Yang, C., & Xi, N. (2017) · Information Sciences · doi:10.1016/j.ins.2017.07.017 · Cited by 67
Retrographic sensing for the measurement of surface texture and shape
Johnson, M. K., & Adelson, E. H. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206534 · Cited by 61
Self-citationLearning continuous grasp stability for a humanoid robot hand based on tactile sensing
Schill, J., Laaksonen, J., Przybylski, M., Kyrki, V., Asfour, T., & Dillmann, R. (2012) · IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) · doi:10.1109/biorob.2012.6290749 · Cited by 48
Grasp stability assessment through unsupervised feature learning of tactile images
Cockbum, D., Roberge, J.-P., Le, T.-H.-L., Maslyczyk, A., & Duchaine, V. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989257 · Cited by 43
Integrated Tactile Sensing for Gripper Fingers
Bicchi, A., Bergamasco, M., Dario, P., & Fiorillo, A. S. (1988) · Cited by 20
Generalizing Regrasping with Supervised Policy Learning
Chebotar, Y., Hausman, K., Kroemer, O., Sukhatme, G. S., & Schaal, S. (2017) · Springer proceedings in advanced robotics · doi:10.1007/978-3-319-50115-4_54 · Cited by 17
Probabilistic consolidation of grasp experience
Bekiroglu, Y., Damianou, A., Detry, R. J., Stork, J. A., Kragić, D., & Ek, C. H. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487133 · Cited by 11
Cited by
Learning ambidextrous robot grasping policies
Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., & Goldberg, K. (2019) · Science Robotics · doi:10.1126/scirobotics.aau4984 · Cited by 563
Lambeta, M., Chou, P.-W., Tian, S., Yang, B., Maloon, B., Most, V. R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., et al. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2977257 · Cited by 553
Self-citationLee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349
“Some of them extend their previous work on grasp stability estimation [5, 13].”
BackgroundA Review of Tactile Information: Perception and Action Through Touch
Li, Q., Kroemer, O., Su, Z., Veiga, F. F., Kaboli, M., & Ritter, H. J. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.3003230 · Cited by 332
Review of machine learning methods in soft robotics
Kim, D., Kim, S.-H., Kim, T., Kang, B. B., Lee, M., Park, W., Ku, S., Kim, D., Kwon, J., Lee, H., et al. (2021) · PLOS ONE · doi:10.1371/journal.pone.0246102 · Cited by 256
Metadata
- Authors
- Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2018-10
- Volume
- 3
- Issue
- 4
- Pages
- 3300-3307
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2804941773
- Cited by
- 359
- References
- 40
- FWCI
- 20.48
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.