Language Models Meet World Models: Embodied Experiences Enhance Language Models
Open accessThe DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.
Abstract
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities. The limitation arises from the fact that LMs are trained only on written text and miss essential embodied knowledge and skills. In this paper, we propose a new paradigm of enhancing LMs by finetuning them with world models, to gain diverse embodied knowledge while retaining their general language capabilities. Our approach deploys an embodied agent in a world model, particularly a simulator of the physical world (VirtualHome), and acquires a diverse set of embodied experiences through both goal-oriented planning and random exploration. These experiences are then used to finetune LMs to teach diverse abilities of reasoning and acting in the physical world, e.g., planning and completing goals, object permanence and tracking, etc. Moreover, it is desirable to preserve the generality of LMs during finetuning, which facilitates generalizing the embodied knowledge across tasks rather than being tied to specific simulations. We thus further introduce the classical (EWC) for selective weight updates, combined with low-rank adapters (LoRA) for training efficiency. Extensive experiments show our approach substantially improves base LMs on 18 downstream tasks by 64.28% on average. In particular, the small LMs (1.3B, 6B, and 13B) enhanced by our approach match or even outperform much larger LMs (e.g., ChatGPT).
Overview
CoChat Summary
TL;DR
The paper proposes E2WM, a paradigm that finetunes language models on embodied experiences collected from a world model (VirtualHome) to improve physical reasoning and planning while preserving general language ability. Using goal-oriented planning and random exploration plus EWC-LoRA, it reports large gains on 18 downstream tasks and competitive performance with ChatGPT for small LMs.
Key findings
- E2WM improves base LMs on 18 downstream tasks by 64.28% on average, with small LMs (1.3B, 6B, 13B) matching or outperforming much larger LMs such as ChatGPT.
- Specific reported gains include plan generation from 34.31 to 51.23 Rouge-L and counting from 30.41% to 67.01% accuracy; GPT-J object path tracking reaches 98.67 LCS versus 33.86 for base GPT-J and 59.53 for ChatGPT.
- Language modeling ability is largely preserved: Pile test subset perplexity changes from 3.443 to 3.537 for GPT-J, and EWC-LoRA achieves the best trade-off between downstream performance and perplexity compared with EWC, LoRA, and KL regularization.
- On bAbI, the approach outperforms base LMs on all selected tasks except the Two Supporting Fact task, and GPT-J surpasses ChatGPT on Three Supporting Fact and Lists/Sets.
Methods
- The approach collects embodied experiences from VirtualHome using goal-oriented planning with Monte Carlo Tree Search and random exploration with one or multiple agents.
- Experiences are compiled into supervised finetuning tasks: plan generation, activity recognition, counting, and object path tracking, trained with cross-entropy loss and task weights.
- Models finetuned include GPT-Neo-1.3B, GPT-J-6B, OPT-13B, and LLaMA-13B, using EWC-LoRA for parameter-efficient updates and regularization; evaluation covers constructed downstream tasks, bAbI, Pile perplexity, and SuperGLUE, with metrics including Rouge-L, accuracy, and LCS.
Limitations
- The paper states the work is limited to a single household environment as the world model, and future work should integrate embodied experiences from different world models and generalize across domains.
- The paper states broader-impact risks: the trained language model may produce unintended output from harmful or adversarial input, and the authors advise against deployment outside controlled research environments until risks are mitigated.
- Evident from the text: evaluation relies heavily on constructed tasks and VirtualHome-derived data, and the human evaluation is limited to GPT-J plan generation with 3 annotators on 150 sampled tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 6 Not checked · 70 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the performance of plan generation drops significantly when plan generation is removed from the training tasks.”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.4 Ablation Studies
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, it outperforms ChatGPT on the Three Supporting Fact task, where the model is required to use three supporting facts from the context to answer a question like”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.2 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, our proposed EWC-LoRA achieves the best trade-off between the perplexity and the downstream performance, which outperforms baselines significantly while almost not increasing the perplexity on the pretraining dataset.”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.3 Comparison of Different Regularization Methods
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, after finetuned with VirtualHome experiences, GPT-J surpasses the much stronger ChatGPT on the most challenging tasks.”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.2 Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides, We can find that combing LoRA with KL will greatly increase perplexity while not achieving better downstream performance.”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.3 Comparison of Different Regularization Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
42 more claims
Create a free account to see all 76 claims and every source verdict.
- It may state two separate claims as one. 17
- The quote only partly states this claim. 3
- Probably not a substantial claim. 11
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > 4.2 Results
“To verify the generality on other NLP tasks, we also include the results on SuperGLUE [40] in Appendix A.8.”
§ Language Models Meet World Models: Embodied Experiences Enhance Language Models > References
“40. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970
Research works printed but not linked: 4
- Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366- 3385, 2021.
- Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199-31212, 2022.
- Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
- Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Cited by
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Vision--Language--Action Models for Embodied AI
Ma, Y., Song, Z., Zhuang, Y., Hao, J., & King, I. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3650584 · Cited by 474
SWE-smith: Scaling Data for Software Engineering Agents
Yang, J., Lieret, K., Jimenez, C., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025) · neural information processing systems · doi:10.52202/085713-3239 · Cited by 430
Exploring Large Language Model‐Based Intelligent Agents: Definitions, Methods, and Prospects
Cheng, Y., Zhang, C., Zhang, Z., Meng, X., Hong, S., Li, W., Wang, Z., Wang, Z., Yin, F., Zhao, J., et al. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70111 · Cited by 339
Self-citationSpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
Metadata
- Authors
- Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., & Hu, Z.
- ORCID
- Wang, Zirui
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 75392-75412
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133231263
- Cited by
- 34
- References
- 27
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 14 · 2023: 7
- FWCI
- 17.53
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- language models, world models, embodied experience, embodied AI, language model training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Language Models Meet World Models: Embodied Experiences Enhance Language Models. https://app.cochat.ai/papers/10.52202/075280-3295
- The paper
- Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., & Hu, Z. (2023). Language Models Meet World Models: Embodied Experiences Enhance Language Models. Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-3295
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.