CoChat
Sign Up
PreprintCited by 43Robot Manipulation and LearningReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

MimicPlay: Long-horizon imitation learning by watching human play

Open access
Wang, Chen, Fan, Linxi, Sun, Jiankai, Zhang, Ruohan, Fei-Fei, Li, Xu, Danfei, Zhu, Yuke, Anandkumar, Anima·2023·arXiv
DOI 10.48550/arxiv.2302.12422arXiv 2302.12422
Open this paper in CoChat — free accountOpen PDF

Abstract

Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often requires an unattainable amount of demonstrations. To reduce the high data requirement, we resort to human play data - video sequences of people freely interacting with the environment using their hands. Even with different morphologies, we hypothesize that human play data contain rich and salient information about physical interactions that can readily facilitate robot policy learning. Motivated by this, we introduce a hierarchical learning framework named MimicPlay that learns latent plans from human play data to guide low-level visuomotor control trained on a small number of teleoperated demonstrations. With systematic evaluations of 14 long-horizon manipulation tasks in the real world, we show that MimicPlay outperforms state-of-the-art imitation learning methods in task success rate, generalization ability, and robustness to disturbances. Code and videos are available at https://mimic-play.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.12422

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning from videos of direct human demonstrations limits robot deployment because it relies on third-person visual observations and lacks real-world physical interaction data.

    “However, robot deployment is limited by its reliance on third-person visual observations and the absence of real-world physical interaction data.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [19]

    Section 3.1 (p. 4) says common human-video datasets provide single-view, 2D hand trajectories that suffer from depth ambiguity and occlusion, supporting a limitation of visual observations but not specifically a reliance on third-person views. The claimed absence of real-world physical interaction data is not supported: Section 3.1 (p. 3) describes humans physically interacting with the task environment, and the Introduction (p. 2) says the method also uses robot teleoperation demonstrations. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 43

Research works printed but not linked: 60

  1. [1][1] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988. -
  2. [2][2] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 5628-5635. IEEE, 2018. -
  3. [3][3] A. Mandlekar, D. Xu, R. Mart´ ın-Mart´ ın, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085 , 2020. - · arXiv
  4. [4][4] K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning , pages 4654-4663. PMLR, 2018. -
  5. [5][5] C. Lynch, M. Khansari, T. Xiao, V . Kumar, J. Tompson, S. Levine, and P . Sermanet. Learning latent plans from play. In Conference on robot learning , pages 1113-1132. PMLR, 2020. -
  6. [6][6] Z. J. Cui, Y . Wang, N. Muhammad, L. Pinto, et al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047 , 2022. - · arXiv
  7. [7][7] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task-agnostic offline reinforcement learning. arXiv preprint arXiv:2209.08959 , 2022. - · arXiv
  8. [8][8] S. Calinon, F. D'halluin, E. L. Sauser, D. G. Caldwell, and A. G. Billard. Learning and reproduction of gestures by imitation. IEEE Robotics & Automation Magazine , 17(2):44-54, 2010. -
  9. [9][9] A. Ijspeert, J. Nakanishi, and S. Schaal. Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292) , volume 2, pages 1398-1403 vol.2, 2002. doi:10.1109/ROBOT.2002.1014739. - · DOI
  10. [10][10] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences , 3(6): 233-242, 1999. -
  11. [11][11] J. Kober and J. Peters. Imitation and reinforcement learning. IEEE Robotics & Automation Magazine , 17(2):55-62, 2010. -
  12. [12][12] P. Englert and M. Toussaint. Learning manipulation skills from a single demonstration. The International Journal of Robotics Research , 37(1):137-154, 2018. -
  13. [13][13] C. Finn, T. Y u, T. Zhang, P . Abbeel, and S. Levine. One-shot visual imitation learning via metalearning. In Conference on robot learning , pages 357-368. PMLR, 2017. -
  14. [14][14] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Robot programming by demonstration. In Springer handbook of robotics , pages 1371-1394. Springer, 2008. -
  15. [15][15] B. D. Argall, S. Chernova, M. V eloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems , 57(5):469-483, 2009. -
  16. [16][16] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines , pages 261-280. Springer, 2006. -
  17. [17][17] J. Kober and J. Peters. Learning motor primitives for robotics. In 2009 IEEE International Conference on Robotics and Automation , pages 2112-2118. IEEE, 2009. -
  18. [18][18] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper/2013/file/ e53a0a2978c28872a4505bdb51db06dc-Paper.pdf . -
  19. [19][19] A. Paraschos, C. Daniel, J. Peters, and G. Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots , 42(3):529-551, 2018. -
  20. [20][20] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ ın-Mart´ ın. What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning , 2021. URL https://openreview. net/forum?id=JrsfBJtDFdI . -
  21. [21][21] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters , 5(2):492-499, 2019. -
  22. [22][22] Y. Zhu, A. Joshi, P. Stone, and Y . Zhu. VIOLA: Object-centric imitation learning for visionbased robot manipulation. In 6th Annual Conference on Robot Learning , 2022. URL https: //openreview.net/forum?id=L8hCfhPbFho . -
  23. [23][23] C. Wang, R. Wang, A. Mandlekar, L. Fei-Fei, S. Savarese, and D. Xu. Generalization through hand-eye coordination: An action space for learning spatially-invariant visuomotor control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8913-8920. IEEE, 2021. -
  24. [24][24] P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL) , 2021. -
  25. [25][25] A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V . V anhoucke, S. V ega, Q. Vuong, F. Xia, T. Xiao, P . Xu, S. Xu, T. Y u, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06817 , 2022. - · arXiv
  26. [26][26] brian ichter, A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y . Lu, C. Parada, K. Rao, P . Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P . Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P . Pastor, L. Luu, K.-H. Lee, Y . Kuang, S. Jesmonth, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning , 2022. -
  27. [27][27] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 4414-4420. IEEE, 2020. -
  28. [28][28] D. Xu, S. Nair, Y . Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 3795-3802. IEEE, 2018. -
  29. [29][29] H. Xiong, Q. Li, Y .-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7827-7834. IEEE, 2021. -
  30. [30][30] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. In Conference on Robot Learning , pages 1930-1942. PMLR, 2021. -
  31. [31][31] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning , pages 537-546. PMLR, 2022. -
  32. [32][32] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research , 40(12-14):1419-1434, 2021. -
  33. [34][34] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems , 32, 2019. -
  34. [35][35] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv:1912.04443 , 2019. - · arXiv
  35. [36][36] K. Schmeckpeper, A. Xie, O. Rybkin, S. Tian, K. Daniilidis, S. Levine, and C. Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision , pages 708-725. Springer, 2020. -
  36. [37][37] A. D. Edwards and C. L. Isbell. Perceptual values from observation. arXiv preprint arXiv:1905.07861 , 2019. - · arXiv
  37. [38][38] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning , volume 155 of Proceedings of Machine Learning Research , pages 339-354. PMLR, 16-18 Nov 2021. URL https://proceedings. mlr.press/v155/schmeckpeper21a.html . -
  38. [39][39] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. CoRL , 2022. -
  39. [40][40] S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning , 2022. URL https:// openreview.net/forum?id=tGbpgz6yOrI . -
  40. [41][41] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173 , 2022. - · arXiv
  41. [42][42] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022. -
  42. [43][43] N. Hansen, Z. Yuan, Y . Ze, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre-training for visuo-motor control: Revisiting a learning-from-scratch baseline. arXiv preprint arXiv:2212.05749 , 2022. - · arXiv
  43. [44][44] Y. Liu, A. Gupta, P . Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1118-1125. IEEE, 2018. -
  44. [45][45] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. Conference on Robot Learning (CoRL) , 2022. -
  45. [46][46] M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imitation. In Conference on Robot Learning , pages 979-989. PMLR, 2020. -
  46. [47][47] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:2207.09450 , 2022. - · arXiv
  47. [48][48] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022. -
  48. [49][49] D. Shan, J. Geng, M. Shu, and D. Fouhey. Understanding human hands in contact at internet scale. In CVPR , 2020. -
  49. [51][51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. -
  50. [52][52] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  51. [53][53] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. arXiv preprint arXiv:2206.11251 , 2022. - · arXiv
  52. [55][55] D. Dwibedi, Y . Aytar, J. Tompson, P . Sermanet, and A. Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1801-1810, 2019. -
  53. [56][56] A. Mandlekar, Y . Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning , pages 879-893. PMLR, 2018. -
  54. [57][57] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. -
  55. [58][58] O. Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation , 3(1):43-53, 1987. doi:10.1109/JRA. 1987.1087068. - · DOI
  56. [59][59] A. Graves and A. Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks , pages 37-45, 2012. -
  57. [61][61] Y. Zhu, J. Wong, A. Mandlekar, R. Mart´ ın-Mart´ ın, A. Joshi, S. Nasiriany, and Y . Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 , 2020. - · arXiv
  58. [62][62] E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109. - · DOI · Paper page
  59. [63][63] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Mart´ ın-Mart´ ın, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning , pages 80-93. PMLR, 2023. -
  60. [64][64] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research , 9:2579-2605, 2008. URL http://www.jmlr.org/papers/v9/ vandermaaten08a.html .

Authors
Wang, C., Fan, L., Sun, J., Zhang, R., Fei-Fei, L., Xu, D., Zhu, Y., & Anandkumar, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2302.12422
arXiv
2302.12422
OpenAlex
W4322616306
Cited by
43
References
0
Citations by year
2025: 10 · 2024: 28 · 2023: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
imitation learning, long-horizon manipulation, hierarchical learning, visuomotor control, robotic manipulation, physical interaction, generalization to unseen tasks, robustness to disturbances

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review