Deep learning for tactile understanding from visual and haptic data
Open accessAbstract
Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by this cognitive pattern, we propose and explore a purely visual haptic prediction model. Purely visual models enable a robot to “feel” without physical interaction. Furthermore, we demonstrate that using both visual and physical interaction signals together yields more accurate haptic classification. Our models take advantage of recent advances in deep neural networks by employing a unified approach to learning features for physical interaction and visual observations. Even though we employ little domain specific knowledge, our model still achieves better results than methods based on hand-designed features.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Neural networks are convenient and effective at combining multi-modal sensory inputs, as well as integrating passive and interactive perception.
“NNs are also convenient and effective at combining multi-modal sensory inputs [55], as well as integrating passive and interactive perception [56,57,58].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [56]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Self-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
(2010) · Cited by 14,134 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Self-citation(2011) · Cited by 2,752 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Coding and use of tactile signals from the fingertips in object manipulation tasks
Johansson, R. S., & Flanagan, J. R. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2621 · Cited by 2,063 · see citing works
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,866 · see citing works
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528 · see citing works
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., & Penn, G. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288864 · Cited by 892 · see citing works
Material recognition in the wild with the Materials in Context Database
Bell, S., Upchurch, P., Snavely, N., & Bala, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298970 · Cited by 516 · see citing works
Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., & Hinton, G. (2014) · arXiv · doi:10.48550/arxiv.1412.7449 · Cited by 464 · see citing works
Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Agrawal, P., Girshick, R., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_22 · Cited by 427 · see citing works
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gülçehre, Ç., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 413 · see citing works
Bayesian Exploration for Intelligent Identification of Textures
Fishel, J. A., & Loeb, G. E. (2012) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2012.00004 · Cited by 358 · see citing works
Functional imaging of human crossmodal identification and object recognition
Amedi, A., von Kriegstein, K., van Atteveldt, N., Beauchamp, M. S., & Naumer, M. J. (2005) · Experimental Brain Research · doi:10.1007/s00221-005-2396-5 · Cited by 351 · see citing works
Biomimetic tactile sensor array
Wettels, N., Santos, V. J., Johansson, R. S., & Loeb, G. E. (2008) · Advanced Robotics · doi:10.1163/156855308x314533 · Cited by 348 · see citing works
Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks
Graves, A., Liwicki, M., Horst Bunke, A., Schmidhuber, J., & Fernández, S. (2007) · Bern Open Repository and Information System (University of Bern) · Cited by 222 · see citing works
Robotic learning of haptic adjectives through physical interaction
Chu, V. C.-M., McMahon, I. T., Riano, L., McDonald, C. G., He, Q. L., Perez-Tejada, J. M., Arrigo, M., Darrell, T., & Kuchenbecker, K. J. (2014) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.09.021 · Cited by 142 · see citing works
Self-citationUsing robotic exploratory procedures to learn the meaning of haptic adjectives
Chu, V. C.-M., McMahon, I. T., Riano, L., et al. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6631000 · Cited by 106 · see citing works
Mukaibo, Y., Shirado, H., Konyo, M., & Maéno, T. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2005.1570499 · Cited by 99 · see citing works
Quantification and evaluation of human tactile sense towards fabrics
Pan, N. (2006) · International Journal of Design & Nature · doi:10.2495/d&n-v1-n1-48-60 · Cited by 97 · see citing works
Visual, haptic and cross-modal recognition of objects and scenes
Woods, A. T., & Newell, F. N. (2004) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2004.03.006 · Cited by 62 · see citing works
A Behavior-Grounded Approach to Forming Object Categories: Separating Containers From Noncontainers
Griffith, S., Sinapov, J., Sukhoy, V., & Stoytchev, A. (2011) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2011.2157504 · Cited by 45 · see citing works
EmoNets: Multimodal deep learning approaches for emotion recognition in videoPreprint
Kahou, S. E., Bouthillier, X., Lamblin, P., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01800 · Cited by 41 · see citing works
Fabric Handle Clusters Based on Fuzzy Clustering Algorithm
Du, Z., Yu, W., Yu, Y. H., et al. (2008) · International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) · doi:10.1109/fskd.2008.571 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Learning the signatures of the human grasp using a scalable tactile glove
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A., & Matusik, W. (2019) · Nature · doi:10.1038/s41586-019-1234-z · Cited by 1,162 · see citing works
A deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027 · see citing works
Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349 · see citing works
Haptic Codecs for the Tactile Internet
Steinbach, E., Strese, M., Eid, M., Liu, X., Bhardwaj, A., Liu, Q., Al-Ja'Afreh, M., Mahmoodi, T., Hassen, R., El Saddik, A., et al. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2867835 · Cited by 168 · see citing works
Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82 · see citing works
Convolutional neural network and long short-term memory models for ice-jam predictions
Madaeni, F., Chokmani, K., Lhissou, R., Homayouni, S., Gauthier, Y., & Tolszczuk-Leclerc, S. (2022) · The Cryosphere · doi:10.5194/tc-16-1447-2022 · Cited by 51 · see citing works
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Lee, M. A., Tan, M., Zhu, Y., & Bohg, J. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561847 · Cited by 24 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“NNs are also convenient and effective at combining multi-modal sensory inputs [55], as well as integrating passive and interactive perception [56–58].”
Background
- Authors
- Gao, Y., Hendricks, L. A., Kuchenbecker, K. J., & Darrell, T.
- Affiliations
- University of California, Berkeley (US)
- University of Pennsylvania (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 536-543
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2963654998
- Cited by
- 264
- References
- 36
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 18 · 2023: 27 · 2022: 34 · 2021: 32 · 2020: 39 · 2019: 38 · 2018: 28 · 2017: 24 · 2016: 6
- FWCI
- 50.78
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1426787, 1427425)
- Public Health Agency of Canada
- Defense Advanced Research Projects Agency
- National Defense Science and Engineering Graduate
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Keywords
- classification of surfaces, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review