CoChat
Sign Up
Cited by 110Robot Manipulation and LearningReinforcement Learning in RoboticsMultimodal Machine Learning Applications

Language Conditioned Imitation Learning Over Unstructured Data

Open access
Lynch*, Corey, Sermanet*, Pierre·2021·Robotics: Science and Systems XVII
DOI 10.15607/rss.2021.xvii.047
Open this paper in CoChat — free account

Abstract

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot.Prior work in imitation learning typically requires each task be specified with a task id or goal image-something that is often impractical in open-world environments.On the other hand, previous approaches in instruction following allow agent behavior to be guided by language, but typically assume structure in the observations, actuators, or language that limit their applicability to complex settings like robotics.In this work, we present a method for incorporating free-form natural language conditioning into imitation learning.Our approach learns perception from pixels, natural language understanding, and multitask continuous control end-to-end as a single neural network.Unlike prior work in imitation learning, our method is able to incorporate unlabeled and unstructured demonstration data (i.e.no task or language labels).We show this dramatically improves language conditioned performance, while reducing the cost of language annotation to less than 1% of total data.At test time, a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment, specified only with natural language descriptions of each task (e.g."open the drawer...now pick up the block...now press the green button...") (see video).To scale up the number of instructions an agent can follow, we propose combining text conditioned policies with large pretrained neural language models.We find this allows a policy to be robust to many out-of-distribution synonym instructions, without requiring new demonstrations.See videos of a human typing live text commands to our agent at https://groundinglanguage.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15607/rss.2021.xvii.047

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Octo learns control mostly from self-supervised visual observations and reduces the burden on language annotation.

    “This enables our model to learn control mostly from self-supervised visual observations and reduces the burden on language annotation, similar to prior work on multi-context imitation learning [54, 62, 61, 63].”

    From Octo: An Open-Source Generalist Robot Policy (2024) · cites this paper as [54]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 52 · Cited by 110

Research works printed but not linked: 47

  1. Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics, 4(4):343-355, 2012.
  2. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pages 5048-5058, 2017.
  3. Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
  4. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2018.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
  6. Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems, pages 15298-15309, 2019.
  7. Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learn- ing for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018. · arXiv
  8. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
  9. Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self-supervised learning of image embedding for continuous control. arXiv preprint arXiv:1901.00943, 2019. · arXiv
  10. Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
  11. Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019.
  12. Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-346, 1990.
  13. Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstra- tions using generative adversarial nets. arXiv preprint arXiv:1705.10479, 2017. · arXiv
  14. Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. · arXiv
  15. Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro. Emergent systematic generalization in a situated agent. arXiv preprint arXiv:1910.00571, 2019. · arXiv
  16. Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pages 1094-1099. Citeseer, 1993.
  17. Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018. · arXiv
  18. Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 259-266. IEEE, 2010.
  19. Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926, 2019. · arXiv
  20. Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1903.01973. · arXiv
  21. Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795, 2017. · arXiv
  22. Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self- supervised robotic learning. In Conference on Robot Learning, pages 530-539. PMLR, 2020.
  23. Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018.
  24. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, pages 2863-2871, 2015.
  25. Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero- shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2661-2670. JMLR. org, 2017.
  26. Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pages 15578-15588, 2019.
  27. Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online object representations with contrastive learning. arXiv preprint arXiv:1906.04312, 2019. · arXiv
  28. Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.
  30. Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765. IEEE, 2018.
  31. Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017. · arXiv
  32. Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233-242, 1999.
  33. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320, 2015.
  34. Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self- supervised learning from video. Proceedings of International Conference in Robotics and Automation (ICRA), 2018. URL http://arxiv.org/abs/ 1704.06888.
  35. Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854, 2019. · arXiv
  36. Avi Singh, Eric Jang, Alexander Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levine, Mohi Khansari, and Chelsea Finn. Scalable multi- task imitation learning with autonomous improvement. arXiv preprint arXiv:2003.02636, 2020. · arXiv
  37. Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. arXiv preprint arXiv:2010.12083, 2020. · arXiv
  38. Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
  39. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pages 4496-4506, 2017.
  40. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pages 5026-5033. IEEE, 2012. ISBN 978-1-4673-1737-5. URL http://dblp.uni-trier.de/db/conf/iros/iros2012. html#TodorovET12.
  41. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision- language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6629-6638, 2019.
  42. Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
  43. Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maxi- mum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015. · arXiv
  44. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307, 2019. · arXiv
  45. Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in a 2d world. arXiv preprint arXiv:1802.01433, 2018. · arXiv
  46. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8. IEEE, 2018.
  47. Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint arXiv:1910.08210, 2019. · arXiv

Authors
Lynch*, C., & Sermanet*, P.
Venue
Robotics: Science and Systems XVII
Publisher
Robotics: Science and Systems Foundation
Published
2021-07-12
DOI
10.15607/rss.2021.xvii.047
OpenAlex
W3176484337
Cited by
110
References
52
Citations by year
2026: 18 · 2025: 31 · 2024: 20 · 2023: 20 · 2022: 14 · 2021: 6 · 2020: 1
FWCI
22.38
Type
conference paper
Language
en
Open access
Locations
Robotics: Science and Systems (publishedVersion)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
instruction following, robotic manipulation, visuomotor policies, natural language grounding, pre-trained language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review