CoChat
Sign Up
PreprintCited by 32Robot Manipulation and LearningReinforcement Learning in RoboticsAI-based Problem Solving and Planning

Instruction-driven history-aware policies for robotic manipulations

Open access
Guhur, Pierre-Louis, Chen, Shizhe, Garcia, Ricardo, Tapaswi, Makarand, Laptev, Ivan, Schmid, Cordelia·2022·arXiv
DOI 10.48550/arxiv.2209.04899arXiv 2209.04899
Open this paper in CoChat — free accountOpen PDF

Abstract

In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.04899

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 73

  1. [2][2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019. -
  2. [3][3] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020. -
  3. [4][4] J. Lu, D. Batra, D. Parikh, and S. Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS , 2019. -
  4. [5][5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763. PMLR, 2021. -
  5. [9][9] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and where pathways for robotic manipulation. In CoRL , pages 894-906. PMLR, 2022. -
  6. [10][10] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BCZ: Zero-shot task generalization with robotic imitation learning. In CoRL , pages 991-1002. PMLR, 2022. -
  7. [11][11] Y. Yang, D. Cer, A. Ahmad, M. Guo, J. Law, N. Constant, G. H. Abrego, S. Yuan, C. Tar, Y.-H. Sung, et al. Multilingual universal sentence encoder for semantic retrieval. In ACL , pages 87-94, 2020. -
  8. [13][13] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In CoRL , pages 726-747. PMLR, 2021. -
  9. [14][14] S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine Q-Attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR , 2022. -
  10. [15][15] S. Liu, S. James, A. J. Davison, and E. Johns. Auto-Lambda: Disentangling dynamic task relationships. TMLR , 2022. -
  11. [16][16] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research , 40(12-14):1419-1434, 2021. -
  12. [17][17] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In CoRL , pages 1303-1315. PMLR, 2022. -
  13. [18][18] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI , 2018. -
  14. [19][19] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. RLBench: The robot learning benchmark & learning environment. RA-L , 5(2):3019-3026, 2020. -
  15. [20][20] O. Ronneberger, P. Fischer, and T. Brox. UNet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computerassisted intervention , pages 234-241. Springer, 2015. -
  16. [21][21] J. Hill. Real time control of a robot with a mobile camera. In 9th Int. Symp. on Industrial Robots, 1979 , pages 233-246, 1979. -
  17. [22][22] F. Chaumette and S. Hutchinson. Visual servo control. i. basic approaches. IEEE Robotics & Automation Magazine , 13(4):82-90, 2006. -
  18. [23][23] R. Strudel, A. Pashevich, I. Kalevatykh, I. Laptev, and C. Schmid. Learning to combine primitive skills: A step towards versatile robotic manipulation. In ICRA , pages 4637-4643. IEEE, 2020. -
  19. [24][24] A. X. Lee, S. Levine, and P. Abbeel. Learning visual servoing with deep features and fitted q-iteration. In ICLR , 2017. -
  20. [25][25] Q. Bateux, E. Marchand, J. Leitner, F. Chaumette, and P. Corke. Training deep neural networks for visual servoing. In ICRA , pages 1-8. IEEE, 2018. -
  21. [26][26] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In CoRL , pages 158-168. PMLR, 2022. -
  22. [27][27] A. I. K´ aroly, P. Galambos, J. Kuti, and I. J. Rudas. Deep learning in robotics: Survey on model structures and training strategies. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 51(1):266-279, 2020. -
  23. [28][28] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic. Cosypose: Consistent multi-view multi-object 6d pose estimation. In ECCV , 2020. -
  24. [29][29] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019. - · arXiv
  25. [30][30] R. Caruana. Learning many related tasks at the same time with backpropagation. NeurIPS , 7, 1994. -
  26. [31][31] H. B. Ammar, E. Eaton, P. Ruvolo, and M. Taylor. Online multi-task learning for policy gradient methods. In ICML , pages 1206-1214. PMLR, 2014. -
  27. [32][32] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In ICRA , pages 2169-2176. IEEE, 2017. -
  28. [33][33] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning , pages 9767-9779. PMLR, 2021. -
  29. [34][34] T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning , pages 9120-9132. PMLR, 2020. -
  30. [35][35] S. Thrun and J. O'Sullivan. Discovering structure in multiple learning tasks: The TC algorithm. In ICML , volume 96, pages 489-497, 1996. -
  31. [36][36] B. Chen, A. Sax, G. Lewis, I. Armeni, S. Savarese, A. Zamir, J. Malik, and L. Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. CoRL , 2020. -
  32. [37][37] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In ICRA , pages 2786-2793. IEEE, 2017. -
  33. [39][39] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL , pages 1094-1100. PMLR, 2020. -
  34. [40][40] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In CoRL , pages 357-368. PMLR, 2017. -
  35. [41][41] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. MT-Opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 , 2021. - · arXiv
  36. [43][43] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018. -
  37. [44][44] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, and P. Abbeel. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018. - · arXiv
  38. [45][45] J. Peters, S. Vijayakumar, and S. Schaal. Reinforcement learning for humanoid robotics. In Proceedings of the third IEEE-RAS international conference on humanoid robots , pages 1-20, 2003. -
  39. [46][46] A. Pashevich, D. Hafner, J. Davidson, R. Sukthankar, and C. Schmid. Modulated policy hierarchies. NeurIPS Deep RL Workshop , 2018. -
  40. [47][47] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages 4950-4957, 2018. -
  41. [48][48] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016. -
  42. [49][49] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In CoRL , pages 1113-1132. PMLR, 2020. -
  43. [50][50] V. Petr´ ık, M. Tapaswi, I. Laptev, and J. Sivic. Learning object manipulation skills via approximate state estimation from real videos. In CoRL , 2020. -
  44. [51][51] Y. Hristov, D. Angelov, M. Burke, A. Lascarides, and S. Ramamoorthy. Disentangled relational representations for explaining and learning from demonstration. In CoRL , 2019. -
  45. [52][52] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016. - · arXiv
  46. [53][53] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi. ManipulaTHOR: A framework for visual object manipulation. In CVPR , pages 44974506, 2021. -
  47. [55][55] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11097-11107, 2020. -
  48. [56][56] O. Ahmed, F. Tr¨ auble, A. Goyal, A. Neitz, Y. Bengio, B. Sch¨ olkopf, M. W¨ uthrich, and S. Bauer. CausalWorld: A robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXiv:2010.04296 , 2020. - · arXiv
  49. [57][57] D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. ACL , 2017. -
  50. [58][58] E. Stengel-Eskin, A. Hundt, Z. He, A. Murali, N. Gopalan, M. Gombolay, and G. Hager. Guiding multi-step rearrangement tasks with natural language instructions. In Conference on Robot Learning , pages 1486-1501. PMLR, 2021. -
  51. [59][59] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunso. Grounded language learning in a simulated 3D world. NeurIPS Workshop , 2017. -
  52. [60][60] D. I. A. Team, J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, F. Fischer, P. Georgiev, A. Goldin, T. Harley, et al. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint arXiv:2112.03763 , 2021. - · arXiv
  53. [63][63] W. Goodwin, S. Vaze, I. Havoutis, and I. Posner. Semantically grounded object matching for robust robotic scene rearrangement. ICRA , 2021. -
  54. [64][64] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Twenty-Fifth AAAI Conference on Artificial Intelligence , 2011. -
  55. [65][65] W. Liu, C. Paxton, T. Hermans, and D. Fox. StructFormer: Learning spatial structure for language-guided semantic rearrangement of novel objects. ICRA , 2021. -
  56. [66][66] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 , 2022. - · arXiv
  57. [67][67] D. Garg, S. Vaidyanath, K. Kim, J. Song, and S. Ermon. LISA: Learning interpretable skill abstractions from language. arXiv preprint arXiv:2203.00054 , 2022. - · arXiv
  58. [68][68] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14829-14838, 2022. -
  59. [69][69] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. -
  60. [70][70] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2020. -
  61. [71][71] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP , 2019. -
  62. [72][72] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS , 2021. -
  63. [73][73] A. L. Putterman, K. Lu, I. Mordatch, and P. Abbeel. Pretraining for language conditioned imitation with transformers. In NeurIPS , 2021. -
  64. [74][74] S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in Neural Information Processing Systems , 34, 2021. -
  65. [75][75] A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15942-15952, 2021. -
  66. [76][76] P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid. Airbert: In-domain pretraining for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1634-1643, 2021. -
  67. [77][77] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. - · arXiv
  68. [78][78] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. - · arXiv
  69. [79][79] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. NeurIPS , 2017. -
  70. [80][80] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011. -
  71. [81][81] S. James and P. Abbeel. Coarse-to-fine Q-Attention with learned path ranking. arXiv preprint arXiv:2204.01571 , 2022. - · arXiv
  72. [82][82] A. Silva, N. Moorman, W. Silva, Z. Zaidi, N. Gopalan, and M. Gombolay. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters , 7(2):1635-1642, 2021. -
  73. [83][83] E. Olson. Apriltag: A robust and flexible visual fiducial system. In ICRA , 2011.

Cited by

Sorted by citations

Authors
Guhur, P.-L., Chen, S., Garcia, R., Tapaswi, M., Laptev, I., & Schmid, C.
Affiliations
  • École Normale Supérieure (BI)
  • International Institute of Information Technology, Hyderabad (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.04899
arXiv
2209.04899
OpenAlex
W4295688231
Cited by
32
References
0
Citations by year
2025: 8 · 2024: 20 · 2023: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Agence Nationale de la Recherche (ANR-21-FAI1-0002)
  • Grand Équipement National De Calcul Intensif
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
robotic manipulation, natural language instructions, transformer models, long-term memory, task generalization, RLBench, human-robot interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review