Sim-to-Real Transfer for Optical Tactile Sensing
Open accessAbstract
Deep learning and reinforcement learning methods have been shown to enable learning of flexible and complex robot controllers. However, the reliance on large amounts of training data often requires data collection to be carried out in simulation, with a number of sim-to-real transfer methods being developed in recent years. In this paper, we study these techniques for tactile sensing using the TacTip optical tactile sensor, which consists of a deformable tip with a camera observing the positions of pins inside this tip. We designed a model for soft body simulation which was implemented using the Unity physics engine, and trained a neural network to predict the locations and angles of edges when in contact with the sensor. Using domain randomisation techniques for sim-to-real transfer, we show how this framework can be used to accurately predict edges with less than 1 mm prediction error in real-world testing, without any real-world data at all.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2× · see citing works
“In this paper, we study sim-to-real transfer for tactile sensing with domain randomisation [11], where simulation parame-”
MethodElastic-Plastic Deformation at Finite Strains
Lee, E. H. (1969) · Journal of Applied Mechanics · doi:10.1115/1.3564580 · Cited by 2,630 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · see citing works
Learning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643 · see citing works
The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies
Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M. E., Rossiter, J. M., & Lepora, N. F. (2018) · Soft Robotics · doi:10.1089/soro.2017.0052 · Cited by 579 · see citing works
Self-citationRychlewski, J. (1984) · Journal of Applied Mathematics and Mechanics · doi:10.1016/0021-8928(84)90137-0 · Cited by 298 · see citing works
Deep learning a grasp function for grasping under gripper pose uncertainty
Johns, E., Leutenegger, S., & Davison, A. J. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759657 · Cited by 233 · see citing works
Self-citationImproved GelSight tactile sensor for measuring geometry and slip
Dong, S., Yuan, W., & Adelson, E. H. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202149 · Cited by 224 · see citing works
Improved GelSight tactile sensor for measuring geometry and slipPreprint
Dong, S., Yuan, W., & Adelson, E. H. (2017) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 216 · see citing works
Optimal control with learned local models: Application to dexterous manipulation
Kumar, V., Todorov, E. V., & Levine, S. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487156 · Cited by 207 · see citing works
Martín-Martín, R., Lee, M. A., Gardner, R., Savarese, S., Bohg, J., & Garg, A. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968201 · Cited by 186 · see citing works
Sensing tactile microvibrations with the BioTac — Comparison with human sensitivity
Fishel, J. A., & Loeb, G. E. (2012) · IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) · doi:10.1109/biorob.2012.6290741 · Cited by 171 · see citing works
Learning robot in-hand manipulation with tactile features
van Hoof, H., Hermans, T., Neumann, G. M., & Peters, J. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363524 · Cited by 162 · see citing works
Stable reinforcement learning with autoencoders for tactile and visual data
van Hoof, H., Chen, N., Karl, M., van der Smagt, P., & Peters, J. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759578 · Cited by 141 · see citing works
Lepora, N. F., Church, A., de Kerckhove, C., et al. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2899192 · Cited by 115 · see citing works
Self-citationTactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations
Hogan, F. R., Bauzá, M., Canal, O., Donlon, E., & Rodríguez, A. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593528 · Cited by 104 · see citing works
Exploratory Tactile Servoing With Active Touch
Lepora, N. F., Aquilina, K., & Cramphorn, L. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2662071 · Cited by 88 · see citing works
Self-citationFingerVision Tactile Sensor Design and Slip Detection Using Convolutional LSTM NetworkPreprint
Zhang, Y., Kan, Z., Tse, Y. A., Yang, Y., & Wang, M. Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02653 · Cited by 80 · see citing works
Ground Truth Force Distribution for Learning-Based Tactile Sensing: A Finite Element Approach
Sferrazza, C., Wahlsten, A., Trueeb, C., & Raffaello D’andrea, B. (2019) · IEEE Access · doi:10.1109/access.2019.2956882 · Cited by 75 · see citing works
TactileGCN: A Graph Convolutional Network for Predicting Grasp Stability with Tactile Sensors
García-García, A., Zapata-Impata, B. S., Orts‐Escolano, S., Gil, P., & García‐Rodríguez, J. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851984 · Cited by 66 · see citing works
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
Lin, J., Calandra, R., & Levine, S. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793885 · Cited by 52 · see citing works
Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement LearningPreprint
Huang, S. H., Zambelli, M., Kay, J., Martins, M. F., Tassa, Y., Pilarski, P. M., & Hadsell, R. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.08542 · Cited by 45 · see citing works
Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing
Sferrazza, C., Bi, T., & D’Andrea, R. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341285 · Cited by 44 · see citing works
Simulation of the SynTouch BioTac Sensor
Ruppel, P., Jonetzko, Y., Görner, M., et al. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-01370-7_30 · Cited by 15 · see citing works
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal MatchingPreprint
Lin, J., Calandra, R., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03591 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Stephen James, Andrew J Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In Conference on Robot Learning (CoRL), 2017.
- Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics ran- domization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8. IEEE, 2018.
- Jeremy A Fishel and Gerald E Loeb. Sensing tactile microvibrations with the biotaccomparison with human sensitivity. In 2012 4th IEEE RAS & EMBS international conference on biomedical robotics and biomechatronics (BioRob), pages 1122-1127. IEEE, 2012.
- Daniel Fernandes Gomes, Achu Wilson, and Shan Luo. Gelsight simulation for sim2real learning. In ICRA ViTac Workshop, 2019.
Cited by
Sorted by citations
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · see citing works
“Some other applications of DRL and sim-to-real transfer in robotics that have emerged over the past years are the control of a plasma jet [32], tactile sensing [43], or multi-agent manipulation [44].”
Background
- Authors
- Ding, Z., Lepora, N. F., & Johns, E.
- Affiliations
- Imperial College London (GB)
- University of Bristol (GB)
- Venue
- 2020 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 1639-1645
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3014297871
- Cited by
- 53
- References
- 27
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 9 · 2023: 8 · 2022: 12 · 2021: 11 · 2020: 4
- FWCI
- 3.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Keywords
- sim-to-real transfer, domain randomization, soft body simulation, edge detection, contact prediction, robot tactile sensing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
