Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
Open accessAbstract
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. We present results in simulation and on a real robot.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Neural networks are convenient and effective at combining multi-modal sensory inputs, as well as integrating passive and interactive perception.
“NNs are also convenient and effective at combining multi-modal sensory inputs [55], as well as integrating passive and interactive perception [56,57,58].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
(2011) · Cited by 2,752 · see citing works
(1987) · Medical Entomology and Zoology · Cited by 2,512 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Quasi-Static Assembly of Compliantly Supported Rigid Parts
Whitney, D. E. (1982) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3149634 · Cited by 968 · see citing works
FlowNet: Learning Optical Flow with Convolutional NetworksPreprint
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 814 · see citing works
Historical Perspective and State of the Art in Robot Force Control
Whitney, D. E. (1987) · The International Journal of Robotics Research · doi:10.1177/027836498700600101 · Cited by 766 · see citing works
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704 · see citing works
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460875 · Cited by 636 · see citing works
Deep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599 · see citing works
A direct method for trajectory optimization of rigid bodies through contact
Posa, M., Cantu, C., & Tedrake, R. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913506757 · Cited by 597 · see citing works
Inertial Properties in Robotic Manipulation: An Object-Level Framework
Khatib, O. (1995) · The International Journal of Robotics Research · doi:10.1177/027836499501400103 · Cited by 545 · see citing works
Human-Inspired Robotic Grasp Control With Tactile Sensing
Romano, J. M., Hsiao, K., Niemeyer, G., Chitta, S., & Kuchenbecker, K. J. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2162271 · Cited by 412 · see citing works
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 383 · see citing works
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2852779 · Cited by 365 · see citing works
Stochastic Variational Video PredictionPreprint
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11252 · Cited by 351 · see citing works
State representation learning for control: An overview
Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., & Filliat, D. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.006 · Cited by 302 · see citing works
Reinforcement and Imitation Learning for Diverse Visuomotor Skills
Zhu, Y., Wang, Z., Merel, J., Rusu, A. A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R. T., de Freitas, N., et al. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.009 · Cited by 267 · see citing works
Self-citationDeep learning for tactile understanding from visual and haptic data
Gao, Y., Hendricks, L. A., Kuchenbecker, K. J., & Darrell, T. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487176 · Cited by 264 · see citing works
Learning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.00177 · Cited by 248 · see citing works
Interactive Perception: Leveraging Action in Perception and Perception in Action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragić, D., Schaal, S., & Sukhatme, G. (2017) · MPG.PuRe (Max Planck Society) · Cited by 242 · see citing works
Self-citationSu, Z., Hausman, K., Chebotar, Y., Molchanov, A., Loeb, G. E., Sukhatme, G. S., & Schaal, S. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363558 · Cited by 228 · see citing works
Interactive perception: Leveraging action in perception and perception in action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2721939 · Cited by 227 · see citing works
Self-citationNeural Synergy Between Kinetic Vision and Touch
Blake, R., Sobel, K. V., & James, T. W. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00691.x · Cited by 225 · see citing works
Exploitation of environmental constraints in human and robotic grasping
Eppner, C., Deimel, R., Álvarez-Ruiz, J., Maertens, M., & Brock, O. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914559753 · Cited by 200 · see citing works
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?Preprint
Calandra, R., Owens, A., Upadhyaya, M., Yuan, W., Lin, J., Adelson, E. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05512 · Cited by 163 · see citing works
Learning robot in-hand manipulation with tactile features
van Hoof, H., Hermans, T., Neumann, G. M., & Peters, J. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363524 · Cited by 162 · see citing works
Learning force control policies for compliant manipulation
Kalakrishnan, M., Righetti, L., Pastor, P., & Schaal, S. (2011) · 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2011.6095096 · Cited by 161 · see citing works
An Efficient Acyclic Contact Planner for Multiped Robots
Tonneau, S., Prete, A. D., Pettré, J., et al. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2819658 · Cited by 153 · see citing works
Abu‐Dakka, F. J., Nemec, B., Jørgensen, J. A., Savarimuthu, T. R., Krüger, N., & Ude, A. (2015) · Autonomous Robots · doi:10.1007/s10514-015-9435-2 · Cited by 142 · see citing works
Stable reinforcement learning with autoencoders for tactile and visual data
van Hoof, H., Chen, N., Karl, M., van der Smagt, P., & Peters, J. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759578 · Cited by 141 · see citing works
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors
Fu, J., Levine, S., & Abbeel, P. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759592 · Cited by 126 · see citing works
Path integral guided policy search
Chebotar, Y., Kalakrishnan, M., Yahya, A., Li, A., Stefan Schaal, A., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989384 · Cited by 124 · see citing works
Integrating State Representation Learning Into Deep Reinforcement Learning
de Bruin, T., Kober, J., Tuyls, K., & Babuška, R. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2800101 · Cited by 120 · see citing works
Reinforcement and Imitation Learning for Diverse Visuomotor SkillsPreprint
Zhu, Y., Wang, Z., Merel, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09564 · Cited by 107 · see citing works
Self-citationStabilizing novel objects by learning to predict tactile slip
Veiga, F. F., van Hoof, H., Peters, J., & Hermans, T. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354090 · Cited by 106 · see citing works
Deep Multimodal Representation Learning from Temporal Data
Yang, X., Ramesh, P. S., Chitta, R., Madhvanath, S., Bernal, E. A., & Luo, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.538 · Cited by 104 · see citing works
A convex model of humanoid momentum dynamics for multi-contact motion generation
Pontón, B., Herzog, A., Schaal, S., et al. (2016) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2016.7803371 · Cited by 85 · see citing works
Conti, F. (2003) · Cited by 81 · see citing works
Assembly strategies for chamferless parts
Caine, M. E., Lozano‐Pérez, T., & Seering, W. P. (2003) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1989.100031 · Cited by 81 · see citing works
An autonomous manipulation system based on force control and optimization
Righetti, L., Kalakrishnan, M., Pástor, P., Binney, J., Kelly, J. W., Voorhies, R. C., Sukhatme, G. S., & Stefan Schaal, A. (2013) · Autonomous Robots · doi:10.1007/s10514-013-9365-9 · Cited by 77 · see citing works
Probabilistic Articulated Real-Time Tracking for Robot Manipulation
Cifuentes, C. G., Issac, J., Wüthrich, M., Stefan Schaal, A., & Bohg, J. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2645124 · Cited by 72 · see citing works
Self-citationAutomated guidance of peg-in-hole assembly tasks for complex-shaped parts
Song, H.-C., Kim, Y.-L., & Song, J.-B. (2014) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2014.6943202 · Cited by 59 · see citing works
Learning End-to-end Multimodal Sensor Policies for Autonomous NavigationPreprint
Liu, G.-H., Siravuru, A., Prabhakar, S., Veloso, M., & Kantor, G. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10422 · Cited by 58 · see citing works
Data-Driven Online Decision Making for Autonomous Manipulation
Kappler, D., Pástor, P., Kalakrishnan, M., Wüthrich, M., & Stefan Schaal, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.044 · Cited by 57 · see citing works
Learning relational object categories using behavioral exploration and multimodal perception
Sinapov, J., Schenck, C., & Stoytchev, A. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907696 · Cited by 56 · see citing works
Learning tactile characterizations of object- and pose-specific grasps
Bekiroglu, Y., Detry, R. J., & Kragić, D. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094878 · Cited by 43 · see citing works
A probabilistic framework for task-oriented grasp stability assessment
Bekiroglu, Y., Song, D., Wang, L., et al. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630999 · Cited by 39 · see citing works
Learning to represent haptic feedback for partially-observable tasks
Sung, J., Salisbury, J., & Saxena, A. (2017) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2017.7989326 · Cited by 35 · see citing works
Fazeli, N., Zapolsky, S., Drumwright, E., & Rodríguez, A. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_41 · Cited by 33 · see citing works
Crossmodal and multisensory interactions between vision and touch
Lacey, S., & Sathian, K. (2015) · Scholarpedia · doi:10.4249/scholarpedia.7957 · Cited by 17 · see citing works
Integrated Tactile Sensing for Gripper Fingers
Bicchi, A., Bergamasco, M., Dario, P., et al. (1988) · Cited by 15 · see citing works
Crossmodal and Multisensory Interactions Between Vision and Touch
Lacey, S., & Sathian, K. (2015) · Atlantis Press eBooks · doi:10.2991/978-94-6239-133-8_25 · Cited by 13 · see citing works
A Framework for Real-Time Multi-Contact Multi-Body Dynamic Simulation
Conti, F., & Khatib, O. (2016) · Springer tracts in advanced robotics · doi:10.1007/978-3-319-28872-7_16 · Cited by 7 · see citing works
Ganguly, S., & Khatib, O. (2020) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-33950-0_37 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Mul- timodal deep learning", in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 689-696.
- J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh, "Action- conditional video prediction using deep networks in atari games", in Advances in neural information processing systems, 2015, pp. 2863- 2871.
- N. Srivastava and R. R. Salakhutdinov, "Multimodal learning with deep boltzmann machines", in Advances in neural information processing systems, 2012, pp. 2222-2230.
Cited by
Sorted by citations
Revisiting Self-Supervised Visual Representation Learning
Kolesnikov, A., Zhai, X., & Beyer, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00202 · Cited by 675 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Mees, O., Hermann, L., Rosete-Beas, E., & Burgard, W. B. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3180108 · Cited by 189 · see citing works
In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
Suzuki, K., Kanamura, M., Suga, Y., Mori, H., & Ogata, T. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635954 · Cited by 31 · see citing works
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Lee, M. A., Tan, M., Zhu, Y., & Bohg, J. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561847 · Cited by 24 · see citing works
Self-citationMultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“…recognition [48] to the recent explosion of interest in language, vision, and video understanding [48] for applications such as multimedia [102, 116], affective computing [101, 127], robotics [84, 91], finance [70], dialogue [126], human-computer interaction [47, 117], and healthcare [51, 172].”
BackgroundMethodInfluentialSelf-citationKadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
- Authors
- Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J.
- Affiliations
- Stanford University (US)
- ORCID
- Venue
- 2019 International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 8943-8950
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2897170587
- Cited by
- 349
- References
- 59
- Citations by year
- 2026: 28 · 2025: 25 · 2024: 43 · 2023: 38 · 2022: 54 · 2021: 68 · 2020: 66 · 2019: 27
- FWCI
- 83.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toyota Research Institute
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
- Keywords
- self-supervised learning, multimodal representation, contact-rich manipulation, robotic manipulation, sample efficiency, deep reinforcement learning, peg insertion, vision and touch
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
