Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation

Yu, Kelin, Han, Yunhai, Wang, Qixian, Saxena, Vaibhav, Xu, Danfei, Zhao, Ye·2023·arXiv
DOI 10.48550/arxiv.2310.16917arXiv 2310.16917
Open this paper in CoChat — free account

Abstract

Tactile sensing is critical to fine-grained, contact-rich manipulation tasks, such as insertion and assembly. Prior research has shown the possibility of learning tactile-guided policy from teleoperated demonstration data. However, to provide the demonstration, human users often rely on visual feedback to control the robot. This creates a gap between the sensing modality used for controlling the robot (visual) and the modality of interest (tactile). To bridge this gap, we introduce "MimicTouch", a novel framework for learning policies directly from demonstrations provided by human users with their hands. The key innovations are i) a human tactile data collection system which collects multi-modal tactile dataset for learning human's tactile-guided control strategy, ii) an imitation learning-based framework for learning human's tactile-guided control strategy through such data, and iii) an online residual RL framework to bridge the embodiment gap between the human hand and the robot gripper. Through comprehensive experiments, we highlight the efficacy of utilizing human's tactile-guided control strategy to resolve contact-rich manipulation tasks. The project website is at https://sites.google.com/view/MimicTouch.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.16917

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, F. Ramos, D. Fox, and Y. Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  2. Y. Narang, K. Storey, I. Akinola, M. Macklin, P. Reist, L. Wawrzyniak, Y. Guo, A. Moravan- szky, G. State, M. Lu, A. Handa, and D. Fox. Factory: Fast contact for robotic assembly. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  3. G. Schoettler, A. Nair, J. Luo, S. Bahl, J. Aparicio Ojea, E. Solowjow, and S. Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs and natural rewards. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5548-5555, 2020. doi:10.1109/IROS45743.2020.9341714. · DOI
  4. G. Schoettler, A. Nair, J. A. Ojea, S. Levine, and E. Solowjow. Meta-reinforcement learning for robotic industrial insertion tasks. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9728-9735, 2020. doi:10.1109/IROS45743.2020.9340848. · DOI
  5. G. Lengyel, G. Žalalytė, A. Pantelides, J. N. Ingram, J. Fiser, M. Lengyel, and D. M. Wolpert. Unimodal statistical learning produces multimodal object-like representations. eLife, 8:e43942, may 2019. ISSN 2050-084X. doi:10.7554/eLife.43942. URL https://doi.org/ 10.7554/eLife.43942. · DOI
  6. H. Li, Y. Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. In CoRL, 2022.
  7. I. Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: Self-supervised pre- training of tactile representations with robotic play. In 7th Annual Conference on Robot Learn- ing, 2023. URL https://openreview.net/forum?id=EXQ0eXtX3OW.
  8. A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 879-893. PMLR, 29-31 Oct 2018. URL https://proceedings. mlr.press/v87/mandlekar18a.html.
  9. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  10. C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
  11. Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRL), 2022.
  12. J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  13. W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for esti- mating geometry and force. Sensors, 17(12), 2017. ISSN 1424-8220. doi:10.3390/s17122762. URL https://www.mdpi.com/1424-8220/17/12/2762. · DOI
  14. E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez. Gelslim: A high- resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 1927-1934, 2018. doi: 10.1109/IROS.2018.8593661. · DOI
  15. D. Ma, E. Donlon, S. Dong, and A. Rodriguez. Dense tactile force estimation using gelslim and inverse fem. In 2019 International Conference on Robotics and Automation (ICRA), pages 5418-5424, 2019. doi:10.1109/ICRA.2019.8794113. · DOI
  16. S. Dong and A. Rodriguez. Tactile-based insertion for dense box-packing. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7953-7960, 2019. doi:10.1109/IROS40897.2019.8968204. · DOI
  17. M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 IEEE International Conference on Robotics and Automation (ICRA), 2019. URL https://arxiv.org/abs/1810.10191. · arXiv
  18. S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez. Tactile-rl for inser- tion: Generalization to objects of unknown geometry. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6437-6443, 2021. doi:10.1109/ICRA48506.2021. 9561646. · DOI
  19. S. Kim and A. Rodriguez. Active extrinsic contact sensing: Application to general peg-in-hole insertion. In ICRA, 2022.
  20. Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  21. S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess. Shapemap 3-d: Efficient shape mapping through dense touch and vision. In 2022 International Conference on Robotics and Automation (ICRA), pages 7073-7080, 2022. doi:10.1109/ICRA46639.2022.9812040. · DOI
  22. E. J. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. arXiv preprint arXiv:2007.03778, 2020. · arXiv
  23. E. Smith, D. Meger, L. Pineda, R. Calandra, J. Malik, A. Romero Soriano, and M. Drozdzal. Active 3d shape reconstruction from vision and touch. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16064-16078.
  24. Curran Asso- ciates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/ 2021/file/8635b5fd6bc675033fb72e8a3ccc10a0-Paper.pdf.
  25. R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, oct 2018. doi:10.1109/lra.2018.2852779. URL https: //doi.org/10.1109%2Flra.2018.2852779. · DOI
  26. L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta. The curious robot: Learning visual representations via physical interactions. In European Conference on Computer Vision, pages 3-18. Springer, 2016.
  27. R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine. The feeling of success: Does touch sensing help predict grasp outcomes? In S. Levine, V. Van- houcke, and K. Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learn- ing, volume 78 of Proceedings of Machine Learning Research, pages 314-323. PMLR, 13-15 Nov 2017. URL https://proceedings.mlr.press/v78/calandra17a.html.
  28. Y. Han, K. Yu, R. Batra, N. Boyd, C. Mehta, T. Zhao, Y. She, S. Hutchinson, and Y. Zhao. Learning generalizable vision-tactile robotic grasping strategy for deformable objects via trans- former. arXiv preprint arXiv:2112.06374, 2023. · arXiv
  29. A. Thankaraj and L. Pinto. That sounds right: Auditory self-supervision for dynamic robot ma- nipulation. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview. net/forum?id=sLhk0keeiseH.
  30. J. Sinapov, M. Wiemer, and A. Stoytchev. Interactive learning of the acoustic properties of household objects. In 2009 IEEE International Conference on Robotics and Automation, pages 2518-2524, 2009. doi:10.1109/ROBOT.2009.5152802. · DOI
  31. J. Christie and N. Kottege. Acoustics based terrain classification for legged robots. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3596-3603, 2016. doi:10.1109/ICRA.2016.7487543. · DOI
  32. C. Matl, Y. Narang, D. Fox, R. Bajcsy, and F. Ramos. Stressd: Sim-to-real from sound for stochastic dynamics. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 935-958. PMLR, 16-18 Nov 2021. URL https://proceedings.mlr.press/v155/ matl21a.html.
  33. D. Gandhi, A. Gupta, and L. Pinto. Swoosh! rattle! thump! -actions that sound. In Proceed- ings of Robotics: Science and Systems (RSS), 2020.
  34. P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
  35. H. Bharadhwaj, A. Gupta, S. Tulsiani, and V. Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011, 2023. · arXiv
  36. H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
  37. S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5962-5969, 2023. doi:10.1109/ICRA48891.2023.10160547. · DOI
  38. C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=hRZ1YjDZmTo.
  39. M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, 2023.
  40. J. Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 29.
  41. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/ paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf.
  42. M. Xie, A. Handa, S. Tyree, D. Fox, H. Ravichandar, N. D. Ratliff, and K. V. Wyk. Neural ge- ometric fabrics: Efficiently learning high-dimensional policies from demonstration. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1355-1367. PMLR, 14-18 Dec 2023.
  43. Y. Han, M. Xie, Y. Zhao, and H. Ravichandar. On the utility of koopman operator theory in learning dexterous manipulation skills. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=pw-OTIYrGa.
  44. Y. Huang, L. Rozo, J. Silvério, and D. G. Caldwell. Non-parametric imitation learning of robot motor skills. In 2019 International Conference on Robotics and Automation (ICRA), pages 5266-5272, 2019. doi:10.1109/ICRA.2019.8794267. · DOI
  45. M. Vaandrager, R. Babuška, L. Bus ¸oniu, and G. A. Lopes. Imitation learning with non- parametric regression. In Proceedings of 2012 IEEE International Conference on Automation, Quality and Testing, Robotics, pages 91-96, 2012. doi:10.1109/AQTR.2012.6237681. · DOI
  46. H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3: 297-330, 2020.
  47. S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez. Auto- matic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit., 47:2280-2292, 2014. URL https://api.semanticscholar.org/CorpusID: 12519167.
  48. M. Du, O. Y. Lee, S. Nair, and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  49. S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  50. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doer- sch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent -a new approach to self-supervised learn- ing. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284.
  51. Curran As- sociates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf.
  52. D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino. Byol for audio: Self- supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021. doi:10.1109/IJCNN52387.2021. 9534474. · DOI
  53. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum en- tropy deep reinforcement learning with a stochastic actor. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/haarnoja18b.html.
  54. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi:10.1109/CVPR.2016.90. · DOI · Paper page

Cited by

See all 1

Metadata

Authors
Yu, K., Han, Y., Wang, Q., Saxena, V., Xu, D., & Zhao, Y.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2310.16917
arXiv
2310.16917
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation. https://app.cochat.ai/papers/10.48550/arxiv.2310.16917
The paper
Yu, K., Han, Y., Wang, Q., Saxena, V., Xu, D., & Zhao, Y. (2023). MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation. arXiv. https://doi.org/10.48550/arxiv.2310.16917

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.