Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is nontrivial to manually design a robot controller that combines these modalities, which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to train directly on real robots due to sample complexity. In this article, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 1×
“Specifically, we choose trust-region policy optimization (TRPO), a policy gradient method [71], which we found to outperform other model-free RL algorithms after initial testing [70], [75],”
MethodWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 1×
“e only on one task [20, 31], or require human demonstration and object CAD models [1]. There have been promising works that train manipulation policies in simulation and transfer them to a real robot [3, 10, 50]. However, only few works focused on contact-rich tasks [24] and none relied on haptic feedback in simulatio…”
BackgroundAudio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
Cited by
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414
“Currently, separate feature extractors are used for each modality (Kaiser et al., 2017; Arandjelovic & Zisserman, 2018; Wang et al., 2020b; Chen et al., 2020b; Alayrac et al., 2020; Lee et al., 2020; Xiao et al., 2020) – it is not sensible to concatenate an audio spectrogram or a raw audio sequence with an image and p…”
MethodObject detection recognition and robot grasping based on machine learning: a survey
Bai, Q., Li, S., Yang, J., Song, Q., Li, Z., & Zhang, X. (2020) · IEEE Access · doi:10.1109/access.2020.3028740 · Cited by 154
Roveda, L., Magni, M., Cantoni, M., Piga, D., & Bucca, G. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2020.103711 · Cited by 80
Fusion of tactile and visual information in deep learning models for object recognition
Babadian, R. P., Faez, K., Amiri, M., & Falotico, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.032 · Cited by 67
Bottom-Up Skill Discovery From Unsegmented Demonstrations for Long-Horizon Robot Manipulation
Zhu, Y., Stone, P., & Zhu, Y. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3146589 · Cited by 50
Self-citationDetect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Lee, M. A., Tan, M., Zhu, Y., & Bohg, J. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561847 · Cited by 24
“The decoder architectures for these four selfsupervised objectives are described in [19].”
BackgroundMethodInfluentialSelf-citationMultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85
“We include the large-scale M U J O C O P USH [90] and V ISION &T OUCH [92] datasets which record the manipulation of simulated and real robotic arms equipped with visual (RGB and depth), force , and proprioception sensors.”
MethodSelf-citationA Perspective on Open Challenges in Deformable Object Manipulation
McKennaa, R. P., & Oyekan, J. (2026) · arXiv · doi:10.48550/arxiv.2602.22998 · Cited by 1
“Robots need fast, precise, and multi-modal perception capabilities to understand their environment, which is essential for performing complex manipulation tasks [25].”
Metadata
- Authors
- Lee, M. A., Zhu, Y., Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J.
- Venue
- IEEE Transactions on Robotics
- Published
- 2020-06
- Volume
- 36
- Issue
- 3
- Pages
- 582-596
- ISSN
- 1552-3098 (ISSN-L)
- 1941-0468
- OpenAlex
- W2965591307
- Cited by
- 238
- References
- 18
- FWCI
- 14.53
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.