Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
Open accessAbstract
Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Self-citationA survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
Apprenticeship learning via inverse reinforcement learning
Abbeel, P., & Ng, A. Y. (2004) · doi:10.1145/1015330.1015430 · Cited by 2,873 · see citing works
Self-citationApplied Optimal Control: Optimization, Estimation, and Control
Bryson, A. E., Ho, Y., & Siouris, G. M. (1979) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1979.4310229 · Cited by 2,607 · see citing works
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340 · see citing works
Is imitation learning the route to humanoid robots?
Schaal, S. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01327-3 · Cited by 1,325 · see citing works
Robot Programming by Demonstration
Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008) · Springer eBooks · doi:10.1007/978-3-540-30301-5_60 · Cited by 1,078 · see citing works
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831 · see citing works
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738 · see citing works
Learning and generalization of motor skills by learning from demonstration
Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152385 · Cited by 702 · see citing works
Autonomous Helicopter Aerobatics through Apprenticeship Learning
Abbeel, P., Coates, A., & Ng, A. (2010) · Int. J. Robotics Res. · doi:10.1177/0278364910371999 · Cited by 682 · see citing works
Self-citationA Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots
Giusti, A., Guzzi, J., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Caro, G. D., et al. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2509024 · Cited by 665 · see citing works
A direct method for trajectory optimization of rigid bodies through contact
Posa, M., Cantu, C., & Tedrake, R. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913506757 · Cited by 597 · see citing works
Guided Cost Learning: Deep Inverse Optimal Control via Policy OptimizationPreprint
Finn, C., Levine, S., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00448 · Cited by 589 · see citing works
Self-citationAutonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer tracts in advanced robotics · doi:10.1007/11552246_35 · Cited by 557 · see citing works
Learning and Reproduction of Gestures by Imitation
Calinon, S., D'halluin, F., Sauser, E. L., Caldwell, D. G., & Billard, A. (2010) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2010.936947 · Cited by 461 · see citing works
Learning monocular reactive UAV control in cluttered natural environments
Ross, S., Melik-Barkhudarov, N., Shankar, K. S., Wendel, A., Dey, D., Bagnell, J. A., & Hebert, M. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630809 · Cited by 410 · see citing works
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · Cited by 379 · see citing works
Self-citationSchaal, S., Peters, J., Nakanishi, J., & Ijspeert, A. J. (2005) · Springer tracts in advanced robotics · doi:10.1007/11008941_60 · Cited by 349 · see citing works
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
Liu, Y., Gupta, A., Abbeel, P., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462901 · Cited by 288 · see citing works
Self-citationNonlinear Inverse Reinforcement Learning with Gaussian Processes
Levine, S., Popović, Z., & Koltun, V. (2011) · Cited by 276 · see citing works
Stochastic policy gradient reinforcement learning on a simple 3D biped
Tedrake, R., Zhang, T., & Seung, H. S. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2004.1389841 · Cited by 273 · see citing works
Self-citationEmpagliflozin for Type 2 Diabetes Mellitus: An Overview of Phase 3 Clinical Trials
Levine, M. J. (2016) · Current Diabetes Reviews · doi:10.2174/1573399812666160613113556 · Cited by 235 · see citing works
Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing
Lipton, J. I., Fay, A. J., & Rus, D. L. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2737046 · Cited by 215 · see citing works
Rahmatizadeh, R., Abolghasemi, P., Bölöni, L. L., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461076 · Cited by 214 · see citing works
Keyframe-based Learning from Demonstration
Akgün, B., Çakmak, M., Jiang, K., & Thomaz, A. L. (2012) · International Journal of Social Robotics · doi:10.1007/s12369-012-0160-0 · Cited by 197 · see citing works
Discovering optimal imitation strategies
Billard, A., Epars, Y., Calinon, S., Stefan Schaal, A., & Cheng, G. (2004) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2004.03.002 · Cited by 176 · see citing works
Third-Person Imitation LearningPreprint
Stadie, B. C., Abbeel, P., & Sutskever, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01703 · Cited by 153 · see citing works
Self-citationLearning from Demonstrations Through the Use of Non-rigid Registration
Schulman, J., Ho, J., Lee, C., & Abbeel, P. (2016) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-28872-7_20 · Cited by 133 · cited 1× · see citing works
“It has been applied successfully to a wide range of domains in robotics, for example to autonomous driving [4], [5], [6], autonomous helicopter flight [7], gesturing [8], and manipulation [9], [10].”
Self-citationRobotic Gastrointestinal Surgery: Early Experience and System Description
Talamini, M. A., Campbell, K. A., & Stanfield, C. (2002) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1089/109264202760267970 · Cited by 104 · see citing works
Imitation learning for locomotion and manipulation
Ratliff, N., Bagnell, J. A., & Srinivasa, S. S. (2007) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2007.4813899 · Cited by 104 · see citing works
MuJoCo HAPTIX: A virtual reality system for hand manipulation
Kumar, V., & Todorov, E. (2015) · 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363441 · Cited by 102 · see citing works
Learning from Demonstrations for Real World Reinforcement LearningPreprint
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., et al. (2017) · arXiv (Cornell University) · Cited by 92 · see citing works
Communicating Robot Arm Motion Intent Through Mixed Reality Head-Mounted Displays
Rosen, E., Whitney, D., Phillips, E. K., Chien, G., Tompkin, J., Konidaris, G., & Tellex, S. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_26 · Cited by 90 · see citing works
Teleoperation with Intelligent and Customizable Interfaces
Dragan, A. D., Srinivasa, S. S., & Lee, K. K. (2013) · Journal of Human-Robot Interaction · doi:10.5898/jhri.2.2.dragan · Cited by 43 · see citing works
First-person tele-operation of a humanoid robot
Fritsche, L., Unverzag, F., Peters, J., & Calandra, R. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363475 · Cited by 41 · see citing works
Learning Grasping Interaction with Geometry-aware 3D Representations.
Yan, X., Khansari, M., Bai, Y., Hsu, J., Pathak, A., Gupta, A., Davidson, J., & Lee, H. (2017) · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674 · see citing works
Diffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613 · see citing works
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., & Song, S. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.026 · Cited by 530 · see citing works
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations
Song, S., Zeng, A., Lee, J., & Funkhouser, T. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004787 · Cited by 185 · see citing works
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119 · see citing works
“Extensive work has been done on Learning from Demonstrations (LfD) where human supervision is usually provided through teleoperation (of a joystick or VR interface) [77, 115, 133] or kinesthetic teaching, where a user physically moves the robot arm [13, 16, 27, 70, 94].”
BackgroundInteractive Language: Talking to Robots in Real Time
Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P. (2022) · arXiv · doi:10.48550/arxiv.2210.06407 · Cited by 113 · see citing works
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Smith, L., Dhawan, N., Zhang, M., Abbeel, P., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.024 · Cited by 102 · see citing works
Self-citationThe Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.010 · Cited by 95 · see citing works
High-Fidelity Grasping in Virtual Reality using a Glove-based System
Liu, H., Zhang, Z., Xie, X., Zhu, Y., Liu, Y., Wang, Y., & Zhu, S.-C. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794230 · Cited by 73 · see citing works
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
Li, C., Xia, F., Martín-Martín, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.03272 · Cited by 58 · see citing works
Mandlekar, A., Booher, J., Spero, M., Tung, A., Gupta, A., Zhu, Y., Garg, A., Savarese, S., & Fei-Fei, L. (2019) · arXiv · doi:10.48550/arxiv.1911.04052 · Cited by 48 · see citing works
AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild
Fang, H., Fang, H.-S., Wang, Y., Ren, J., Chen, J., Zhang, R., Wang, W., & Lu, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610799 · Cited by 38 · see citing works
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020) · arXiv · doi:10.48550/arxiv.2012.06733 · Cited by 35 · see citing works
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., & Fox, D. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.133 · Cited by 29 · see citing works
Training Robots Without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfer
Kim, H., Ohmura, Y., Nagakubo, A., & Kuniyoshi, Y. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3262423 · Cited by 28 · see citing works
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24 · see citing works
“• Action Space : Besides our absolute end effector pose action space, Delta-Action and velocity control spaces is another popular action space choice [4, 15, 63–65].”
BackgroundDexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning
Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L. J., & Zhu, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11127809 · Cited by 24 · see citing works
Language Conditioned Imitation Learning over Unstructured Data
Lynch, C., & Sermanet, P. (2020) · arXiv · doi:10.48550/arxiv.2005.07648 · Cited by 18 · see citing works
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2303.04137 · Cited by 12 · see citing works
Learning Multi-Stage Tasks with One Demonstration via Self-Replay
Di Palo, N., & Johns, E. (2021) · arXiv · doi:10.48550/arxiv.2111.07447 · Cited by 8 · see citing works
“Manipulation Tasks We designed a set of 8 tasks, inspired by the recent imitation learning literature 257 [3, 13, 4, 5, 7, 8], and by everyday tasks to test our method.”
MethodGR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. "., Fang, Y., Fox, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14734 · Cited by 10 · see citing works
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., & Fox, D. (2023) · arXiv · doi:10.48550/arxiv.2310.17596 · Cited by 5 · see citing works
“One popular approach is to have human operators teleoperate robot arms through different control interfaces [1,2], resulting in several demonstrations of robots performing various manipulation tasks, and consequently to use the data to train the robots to perform these tasks on their own.”
BackgroundMethodInfluentialRoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., & Zhu, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.02523 · Cited by 3 · see citing works
“It has been used extensively in prior works [49, 31, 9, 2, 16, 6, 17].”
BackgroundMethod
- Authors
- Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2018 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2018-05
- Pages
- 5628-5635
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2963669336
- Cited by
- 609
- References
- 45
- Citations by year
- 2026: 27 · 2025: 90 · 2024: 85 · 2023: 68 · 2022: 87 · 2021: 87 · 2020: 82 · 2019: 66 · 2018: 16 · 2016: 1
- FWCI
- 25.61
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
- Keywords
- imitation learning, deep reinforcement learning, virtual reality teleoperation, hand tracking, visuomotor skills, robotic manipulation, skill acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
