CoChat
Sign Up
PreprintCited by 287AI-based Problem Solving and PlanningRobot Manipulation and LearningReinforcement Learning in Robotics

Inner Monologue: Embodied Reasoning through Planning with Language Models

Open access
Huang, Wenlong, Xia, Fei, Xiao, Ted, Chan, Harris, Liang, Jacky, Florence, Pete, Zeng, Andy, Tompson, Jonathan +9 more·2022·arXiv
DOI 10.48550/arxiv.2207.05608arXiv 2207.05608
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.05608

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modeling inner speech in an artificial agent has been shown to enhance learning from external information.

    “Modeling IS in an artificial agent has also been shown to enhance learning from external information.”

    From Inner speech as language process and cognitive tool (2023) · cites this paper as [80]

    Section 3.2 (p. 3) says the agent incorporates external feedback into its inner-monologue prompts, improving reasoning and replanning. The paper reports improved task completion, not enhanced learning from external information; the Introduction (p. 1) says the system uses no additional training. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • An LLM has been embedded into the control system for a robot, which has an internal dialog with itself while carrying out tasks like fetching a glass of water.

    “Even more ambitious, an LLM has been embedded into the control system for a robot, which has an internal dialog with itself while carrying out tasks like fetching a glass of water (Huang et al., 2022).”

    From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Huang et al., 2022

    Section 3.2 (p. 3) describes an LLM planner receiving feedback as a robot acts, and the Conclusion (p. 9) says this planning is fused with robotic control policies. The kitchen-task prompts (pp. 23–24) include bringing drinks and moving a water bottle, but do not show the robot fetching a glass of water specifically. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Inner Monologue provides three types of feedback to the agent after it takes actions: task success, passive scene descriptions, and active scene descriptions.

    “Inner Monologue [61] provides three types of feedback to the agent after it takes actions: (1) whether the task is successfully Fig. 3 Comparison between the strategies of single-path and multi-path reasoning.LMZSP is the model proposed in [54] completed, (2) passive scene descriptions, and (3) active scene descriptions.”

    From A survey on large language model based autonomous agents (2024) · cites this paper as [61]

  • Researchers leverage the common sense of large language models for task planning, robot control, and image understanding.

    “For instance, researchers leverage the common sense of large language models for task planning [257], robot control [82], and image understanding [122].”

    From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [82]

  • LLMs can be used to combine multiple modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing, goal interpretation with subgoal decomposition, action sequencing with subgoal decomposition, and action sequencing with transition modeling.

    “LLMs can also be used to combine multiple of the above modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing [13–32], goal interpretation with subgoal decomposition [2, 27, 33], action sequencing with subgoal decomposition [27, 34, 18, 35], action sequencing with transition modeling [8, 28, 32, 36, 37, 13, 38].”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [21]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 287

Research works printed but not linked: 88

  1. [1][1] L. P. Kaelbling and T. Lozano-P´ erez. Integrated task and motion planning in belief space. The International Journal of Robotics Research , 32(9-10):1194-1227, 2013. -
  2. [2][2] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems , 13(1):41-77, 2003. -
  3. [3][3] F. Petroni, T. Rockt¨ aschel, P . Lewis, A. Bakhtin, Y . Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 , 2019. - · arXiv
  4. [4][4] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438, 2020. -
  5. [5][5] J. Davison, J. Feldman, and A. M. Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages 1173-1178, 2019. -
  6. [6][6] A. Talmor, Y . Elazar, Y . Goldberg, and J. Berant. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics , 8:743-758, 2020. -
  7. [7][7] A. Roberts, C. Raffel, and N. Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910 , 2020. - · arXiv
  8. [8][8] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P . Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. - · arXiv
  9. [9][9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  10. [10][10] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022. - · arXiv
  11. [11][11] T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022. - · arXiv
  12. [12][12] A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 , 2022. - · arXiv
  13. [13][13] M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021. - · arXiv
  14. [14][14] L. S. Vygotsky. Thought and language . MIT press, 2012. -
  15. [15][15] P. Carruthers. Thinking in language?: evolution and a modularist possibility. Cambridge University Press, 1998. -
  16. [17][17] L. S. Vygotsky. Play and its role in the mental development of the child. Soviet psychology , 5(3): 6-18, 1967. -
  17. [18][18] C. Colas, T. Karch, C. Moulin-Frier, and P .-Y . Oudeyer. Vygotskian autotelic artificial intelligence: Language and culture internalization for human-like ai. arXiv preprint arXiv:2206.01134 , 2022. - · arXiv
  18. [19][19] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022. - · arXiv
  19. [20][20] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning . PMLR, 2022. -
  20. [21][21] M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine, Y . Lu, L. Luu, C. Parada, P . Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P . Sermanet, N. Sievers, C. Tan, A. Toshev, V . V anhoucke, F. Xia, T. Xiao, P . Xu, S. Xu, and M. Yan. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691 , 2022. - · arXiv
  21. [22][22] L. P. Kaelbling and T. Lozano-P´ erez. Hierarchical planning in the now. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence , 2010. -
  22. [23][23] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P . Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE international conference on robotics and automation (ICRA) , 2014. -
  23. [26][26] D. Nau, Y . Cao, A. Lotem, and H. Munoz-Avila. Shop: Simple hierarchical ordered planner. In Proceedings of the 16th international joint conference on Artificial intelligence , 1999. -
  24. [27][27] S. M. LaValle. Planning algorithms . Cambridge university press, 2006. -
  25. [28][28] M. Toussaint. Logic-geometric programming: An optimization-based approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015. -
  26. [30][30] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems , 2019. -
  27. [31][31] D. Xu, S. Nair, Y . Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , 2018. -
  28. [32][32] D. Xu, R. Mart´ ın-Mart´ ın, D.-A. Huang, Y . Zhu, S. Savarese, and L. F. Fei-Fei. Regression planning networks. Advances in Neural Information Processing Systems , 32, 2019. -
  29. [33][33] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, L. P . Kaelbling, and J. Tenenbaum. Inventing relational state and action abstractions for effective and efficient bilevel planning. arXiv preprint arXiv:2203.09634 , 2022. - · arXiv
  30. [34][34] D. Shah, P. Xu, Y. Lu, T. Xiao, A. Toshev, S. Levine, and B. Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. ICLR , 2022. URL https://openreview.net/pdf?id=vgqS1vkkCbE . -
  31. [35][35] A. Srinivas, A. Jabri, P . Abbeel, S. Levine, and C. Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning , pages 4732-4741. PMLR, 2018. -
  32. [36][36] T. Kurutach, A. Tamar, G. Yang, S. J. Russell, and P . Abbeel. Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems , 31, 2018. -
  33. [37][37] A. Akakzia, C. Colas, P.-Y . Oudeyer, M. Chetouani, and O. Sigaud. Grounding language to autonomously-acquired skills via goal generation. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=chPj I5KMHG . -
  34. [38][38] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. arXiv preprint arXiv:2006.04843 , 2020. - · arXiv
  35. [39][39] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 259-266. IEEE, 2010. -
  36. [40][40] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 1507-1514, 2011. -
  37. [41][41] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics , pages 481-495. Springer, 2013. -
  38. [43][43] T. Kollar, S. Tellex, D. Roy, and N. Roy. Grounding verbs of motion in natural language commands to robots. In Experimental robotics , pages 31-47. Springer, 2014. -
  39. [44][44] V. Blukis, Y . Terme, E. Niklasson, R. A. Knepper, and Y . Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.09664 , 2019. - · arXiv
  40. [45][45] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. ArXiv , abs/1909.05829, 2020. - · arXiv
  41. [46][46] F. Xia, C. Li, R. Mart´ ın-Mart´ ın, O. Litany, A. Toshev, and S. Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , 2021. -
  42. [47][47] C. Li, F. Xia, R. Martin-Martin, and S. Savarese. Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators. In Conference on Robot Learning , 2020. -
  43. [48][48] Y. Jiang, S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurIPS , 2019. -
  44. [49][49] D. Hafner, K.-H. Lee, I. Fischer, and P . Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.04114 , 2022. - · arXiv
  45. [50][50] S. Mirchandani, S. Karamcheti, and D. Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems , 34:29529-29540, 2021. -
  46. [51][51] P. A. Jansen. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint arXiv:2009.14259 , 2020. - · arXiv
  47. [52][52] P. Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517 , 2021. - · arXiv
  48. [53][53] S. Li, X. Puig, Y . Du, C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771 , 2022. - · arXiv
  49. [54][54] M. Chen, J. Tworek, H. Jun, Q. Y uan, H. P . d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. - · arXiv
  50. [55][55] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. - · arXiv
  51. [56][56] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 , 2019. - · arXiv
  52. [57][57] J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Y u, B. Lester, N. Du, A. M. Dai, and Q. V . Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021. - · arXiv
  53. [58][58] C. Paxton, Y . Bisk, J. Thomason, A. Byravan, and D. Foxl. Prospection: Interpretable plans from language by predicting the future. In 2019 International Conference on Robotics and Automation (ICRA) , pages 6942-6948. IEEE, 2019. -
  54. [59][59] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems , 33:13139-13150, 2020. -
  55. [60][60] V. Blukis, R. A. Knepper, and Y . Artzi. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:2011.07384 , 2020. - · arXiv
  56. [61][61] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems , 2021. URL https://arxiv.org/abs/2005.07648 . - · arXiv
  57. [62][62] Y. Chen, R. Xu, Y . Lin, and P . A. V ela. A joint network for grasp detection conditioned on natural language commands. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 4576-4582. IEEE, 2021. -
  58. [63][63] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning. arXiv preprint arXiv:2204.06252 , 2022. - · arXiv
  59. [64][64] C. Yan, F. Carnevale, P . Georgiev, A. Santoro, A. Guy, A. Muldal, C.-C. Hung, J. Abramson, T. Lillicrap, and G. Wayne. Intra-agent speech permits zero-shot task acquisition. arXiv preprint arXiv:2206.03139 , 2022. - · arXiv
  60. [65][65] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021. -
  61. [66][66] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
  62. [67][67] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019. -
  63. [68][68] Z. Wang, J. Y u, A. W. Y u, Z. Dai, Y . Tsvetkov, and Y . Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 , 2021. - · arXiv
  64. [69][69] A. Suglia, Q. Gao, J. Thomason, G. Thattai, and G. Sukhatme. Embodied bert: A transformer model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927 , 2021. - · arXiv
  65. [70][70] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020. -
  66. [71][71] A. Jain, M. Guo, K. Srinivasan, T. Chen, S. Kudugunta, C. Jia, Y . Yang, and J. Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint arXiv:2109.05125 , 2021. - · arXiv
  67. [72][72] J. Sun, D.-A. Huang, B. Lu, Y .-H. Liu, B. Zhou, and A. Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters , 7(2):4924-4930, 2022. -
  68. [73][73] F. Sener and A. Yao. Zero-shot anticipation for instructional activities. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 862-871, 2019. -
  69. [74][74] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14829-14838, 2022. -
  70. [75][75] A. Zeng, P . Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL) , 2020. -
  71. [76][76] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022. -
  72. [77][77] X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021. - · arXiv
  73. [78][78] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research , 34(4-5):705-724, 2015. -
  74. [79][79] F.-J. Chu, R. Xu, and P . A. V ela. Real-world multiobject, multigrasp detection. IEEE Robotics and Automation Letters , 3(4):3355-3362, 2018. -
  75. [80][80] D. Kalashnikov, J. Varley, Y . Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 , 2021. - · arXiv
  76. [81][81] T. Migimatsu and J. Bohg. Grounding predicates through actions. arXiv preprint arXiv:2109.14718 , 2021. - · arXiv
  77. [82][82] Y. Cui, S. Niekum, A. Gupta, V . Kumar, and A. Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference , pages 893-905. PMLR, 2022. -
  78. [83][83] M. Liang and X. Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3367-3375, 2015. -
  79. [84][84] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015. -
  80. [85][85] Z. Zou, Z. Shi, Y. Guo, and J. Ye. Object detection in 20 years: A survey. arXiv preprint arXiv:1905.05055 , 2019. - · arXiv
  81. [86][86] A. Bochkovskiy, C.-Y . Wang, and H.-Y . M. Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 , 2020. - · arXiv
  82. [87][87] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015. -
  83. [88][88] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 13041-13049, 2020. -
  84. [89][89] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations , 2020. URL https://arxiv.org/pdf/1908.09791.pdf . - · arXiv
  85. [90][90] J.-B. Alayrac, J. Donahue, P . Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. - · arXiv
  86. [91][91] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P . Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. - · arXiv
  87. [92][92] A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021. -
  88. [93][93] T. Xiao, E. Jang, D. Kalashnikov, S. Levine, J. Ibarz, K. Hausman, and A. Herzog. Thinking while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv:2004.06089 , 2020. · arXiv

Authors
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S., Hausman, K., & Ichter, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2207.05608
arXiv
2207.05608
OpenAlex
W4285428875
Cited by
287
References
0
Citations by year
2026: 7 · 2025: 63 · 2024: 137 · 2023: 77 · 2022: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
large language models, embodied reasoning, planning, inner speech, robot control, natural language feedback, closed-loop feedback, mobile manipulation, kitchen environment, human-robot interaction, scene description

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review