CoChat
Sign Up
PreprintCited by 12Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningArtificial Intelligence in Games

Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling

Open access
Nottingham, Kolby, Ammanabrolu, Prithviraj, Suhr, Alane, Choi, Yejin, Hajishirzi, Hannaneh, Singh, Sameer, Fox, Roy·2023·arXiv
DOI 10.48550/arxiv.2301.12050arXiv 2301.12050
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world. However, if initialized with knowledge of high-level subgoals and transitions between subgoals, RL agents could utilize this Abstract World Model (AWM) for planning and exploration. We propose using few-shot large language models (LLMs) to hypothesize an AWM, that will be verified through world experience, to improve sample efficiency of RL agents. Our DECKARD agent applies LLM-guided exploration to item crafting in Minecraft in two phases: (1) the Dream phase where the agent uses an LLM to decompose a task into a sequence of subgoals, the hypothesized AWM; and (2) the Wake phase where the agent learns a modular policy for each subgoal and verifies or corrects the hypothesized AWM. Our method of hypothesizing an AWM with LLMs and then verifying the AWM based on agent experience not only increases sample efficiency over contemporary methods by an order of magnitude but is also robust to and corrects errors in the LLM, successfully blending noisy internet-scale information from LLMs with knowledge grounded in environment dynamics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.12050

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • It would not be an apple-to-apple comparison to directly compare with prior methods that take Minecraft screen pixels as input and output low-level controls, because VOYAGER relies on the high-level Mineflayer API to control the agent.

    “Note that we do not directly compare with prior methods that take Minecraft screen pixels as input and output low-level controls [53-55]. It would not be an apple-to-apple comparison, because we rely on the high-level Mineflayer [52] API to control the agent.”

    From Voyager: An Open-Ended Embodied Agent with Large Language Models (2023) · cites this paper as [53]

    Section 5.3 (page unavailable) says DECKARD’s subgoal policies use pixel observations and camera and keyboard actions, supporting its identification as a pixel-input Minecraft agent. But it also says DECKARD uses high-level, single-action crafting rather than exclusively low-level controls. Section 2.4 notes that differences between Minecraft agents make comparisons difficult; the Mineflayer rationale is the citing authors’ own explanation, not a finding attributed to this source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • In Minecraft, acquiring and collecting distinct items is a common measure of individual progression.

    “In Minecraft, a common measure of individual progression is the acquisition and collection of distinct items (48; 35; 17; 2; 9; 16).”

    From Project Sid: Many-agent simulations toward AI civilization (2024) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 12

Research works printed but not linked: 45

  1. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Reincarnating reinforcement learn- ing: Reusing prior computation to accelerate progress. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=t3X5yMI_4G2.
  2. Ammanabrolu, P. and Riedl, M. Learning knowledge graph- based world models of textual environments. Advances in Neural Information Processing Systems, 34:3720-3731, 2021.
  3. Ammanabrolu, P., Tien, E., Hausknecht, M., and Riedl, M. O. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07409, 2020. · arXiv
  4. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and van den Hengel, A. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  5. Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching un- labeled online videos. In Advances in Neural Information Processing Systems, 2022.
  6. Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8718-8735, 2020.
  7. Blukis, V., Paxton, C., Fox, D., Garg, A., and Artzi, Y. A persistent spatial semantic representation for high-level natural language instruction execution. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Con- ference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 706-717. PMLR, 08- 11 Nov 2022. URL https://proceedings.mlr. press/v164/blukis22a.html.
  8. Branavan, S., Silver, D., and Barzilay, R. Learning to win by reading manuals in a monte-carlo framework. In Pro- ceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 268-277, 2011.
  9. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
  10. Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. BabyAI: First steps towards grounded language learning with a hu- man in the loop. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=rJeXCo0cYX.
  11. Dambekodi, S., Frazier, S., Ammanabrolu, P., and Riedl, M. Playing text-based games with common sense. In Pro- ceedings of the NeurIPS Workshop on Wordplay: When Language Meets Games, 2020.
  12. Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  13. Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  14. Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4089-4098, 2018.
  15. Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., and Salakhutdinov, R. Minerl: A large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial In- telligence, IJCAI'19, pp. 2442-2448. AAAI Press, 2019. ISBN 9780999241141.
  16. Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Master- ing diverse domains through world models, 2023. URL https://arxiv.org/abs/2301.04104. · arXiv
  17. Hanjie, A. W., Zhong, V. Y., and Narasimhan, K. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference on Machine Learning, pp. 4051-4062. PMLR, 2021.
  18. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
  19. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents. In Interna- tional Conference on Machine Learning, pp. 9118-9147.
  20. PMLR, 2022a.
  21. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022b.
  22. Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A. T., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers, N., Tan, C., Xu, S., Reyes, D., Rettinghouse, J., Quiambao, J., Pastor, P., Luu, L., Lee, K.-H., Kuang, Y., Jesmonth, S., Jeffrey, K., Ruano, R. J., Hsu, J., Gopalakrishnan, K., David, B., Zeng, A., and Fu, C. K. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022. URL https://openreview.net/ forum?id=bdHkMjBJG_w.
  23. Ku, A., Anderson, P., Patel, R., Ie, E., and Baldridge, J. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4392-4412, 2020.
  24. Kuo, Y.-L., Katz, B., and Barbu, A. Compositional networks enable systematic generalization for grounded language understanding. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pp. 216-226, 2021.
  25. Liang, A., Singh, I., Pertsch, K., and Thomason, J. Transformer adapters for robot learning. In CoRL 2022 Workshop on Pre-training Robot Learning, 2022a. URL https://openreview.net/forum? id=H--wvRYBmF.
  26. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Flo- rence, P., Zeng, A., et al. Code as policies: Language model programs for embodied control. In Workshop on Language and Robotics at CoRL 2022, 2022b.
  27. Liu, R., Wei, J., Gu, S. S., Wu, T.-Y., Vosoughi, S., Cui, C., Zhou, D., and Dai, A. M. Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.05359, 2022. · arXiv
  28. Mu, J., Zhong, V., Raileanu, R., Jiang, M., Goodman, N., Rocktäschel, T., and Grefenstette, E. Improving in- trinsic exploration with language abstractions. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=ALIYCycCsTy.
  29. Nottingham, K., Liang, L., Shin, D., Fowlkes, C. C., Fox, R., and Singh, S. Modular framework for visuomotor language grounding. In Embodied AI Workshop @ CVPR, 2021.
  30. Nottingham, K., Pyla, A., Singh, S., and Fox, R. Learning to query internet text for informing reinforcement learning agents. In Reinforcement Learning and Decision Making Conference, 2022.
  31. Patil, V., Hofmarcher, M., Dinu, M.-C., Dorfer, M., Blies, P. M., Brandstetter, J., Arjona-Medina, J. A., and Hochre- iter, S. Align-rudder: Learning from few demonstrations by reward redistribution. In International Conference on Machine Learning, 2020.
  32. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. Language models as knowl- edge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, 2019.
  33. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  34. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  35. Simpkins, C. and Isbell, C. Composable modu- lar reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4975-4982, Jul. 2019. doi: 10.1609/aaai.v33i01. 33014975. URL https://ojs.aaai.org/index. php/AAAI/article/view/4428. · DOI
  36. Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt: Generating situated robot task plans using large language models. In Second Workshop on Language and Reinforcement Learning, 2022.
  37. Skrynnik, A., Staroverov, A., Aitygulov, E., Aksenov, K., Davydov, V., and Panov, A. I. Forgetful experi- ence replay in hierarchical reinforcement learning from expert demonstrations. Know.-Based Syst., 218(C), apr 2021. ISSN 0950-7051. doi: 10.1016/j.knosys. 2021.106844. URL https://doi.org/10.1016/ j.knosys.2021.106844. · DOI
  38. Song, C. H., Kil, J., Pan, T.-Y., Sadler, B. M., Chao, W.-L., and Su, Y. One step at a time: Long-horizon vision-and- language navigation with milestones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 15482-15491, June 2022a.
  39. Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. Llm-planner: Few-shot grounded plan- ning for embodied agents with large language models. arXiv preprint arXiv:2212.04088, 2022b. · arXiv
  40. Suglia, A., Gao, Q., Thomason, J., Thattai, G., and Sukhatme, G. Embodied bert: A transformer model for embodied, language-guided visual task completion. CoRR, abs/2108.04927, 2021. URL https://arxiv. org/abs/2108.04927. · arXiv
  41. Tam, A. C., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C., Strouse, D., Wang, J. X., Banino, A., and Hill, F. Semantic exploration from language ab- stractions and pretrained representations. arXiv preprint arXiv:2204.05080, 2022. · arXiv
  42. Valmeekam, K., Olmo, A., Sreedharan, S., and Kambham- pati, S. Large language models still can't plan (a bench- mark for llms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Mak- ing Workshop, 2022.
  43. Yu, H., Zhang, H., and Xu, W. Interactive grounded lan- guage acquisition and generalization in a 2d world. In International Conference on Learning Representations, 2018.
  44. Zellers, R., Holtzman, A., Peters, M. E., Mottaghi, R., Kem- bhavi, A., Farhadi, A., and Choi, Y. Piglet: Language grounding through neuro-symbolic interaction in a 3d world. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 2040-2050, 2021.
  45. Zhong, V., Rocktäschel, T., and Grefenstette, E. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=SJgob6NKvH.

Authors
Nottingham, K., Ammanabrolu, P., Suhr, A., Choi, Y., Hajishirzi, H., Singh, S., & Fox, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.12050
arXiv
2301.12050
OpenAlex
W4318751399
Cited by
12
References
0
Citations by year
2025: 3 · 2024: 6 · 2023: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Hasso Plattner Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
embodied agents, subgoal decomposition, sample efficiency, Minecraft, reinforcement learning from human feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review