PLM-based World Models for Text-based Games
Open accessAbstract
World models have improved the ability of reinforcement learning agents to operate in a sample efficient manner, by being trained to predict plausible changes in the underlying environment.As the core tasks of world models are future prediction and commonsense understanding, our claim is that pre-trained language models (PLMs) already provide a strong base upon which to build world models.Worldformer is a recently proposed world model for text-based game environments, based only partially on PLM and transformers.Our distinction is to fully leverage PLMs as actionable world models in text-based game environments, by reformulating generation as constrained decoding which decomposes actions into verb templates and objects.We show that our model improves future valid action prediction and graph change prediction. 1 Additionally, we show that our model better reflects commonsense than standard PLM.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Past works on LM user simulation have not used such simulators to benchmark the reliability of agents, instead focusing on showcasing the ability of LMs to enable realistic simulations.
“However, all these past works have not used such simulators to benchmark the reliability of agents, instead focusing on showcasing the ability of LMs to enable realistic simulations.”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [13]
The Results and Related Works sections discuss evaluating PLM-based world models and agents in text-based games, not LM user simulators. Searches for user simulation, realistic simulations, and benchmarking agent reliability found no passage addressing what past LM user-simulation work did or aimed to show. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“…of our work is that while we aim to adapt pre-trained language models as world models for text-based game environments, due to limited computational resources, we are not able to test our models on larger scales, and take greater advantage of language model scaling laws (Kaplan et al., 2020).”
BackgroundRecurrent World Models Facilitate Policy EvolutionPreprint
Ha, D. R., & Schmidhuber, J. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1809.01999 · Cited by 1,592 · see citing works
Contextual String Embeddings for Sequence Labeling
Akbik, A., Blythe, D. A. J., & Vollgraf, R. (2018) · International Conference on Computational Linguistics · Cited by 1,086 · see citing works
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1470 · Cited by 904 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
Model-Based Reinforcement Learning for Atari
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1903.00374 · Cited by 467 · see citing works
Multi-Stage Document Ranking with BERTPreprint
Nogueira, R. F., Yang, W., Cho, K., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14424 · Cited by 358 · see citing works
Language Understanding for Text-based Games using Deep Reinforcement Learning
Narasimhan, K., Kulkarni, T. D., & Barzilay, R. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1001 · Cited by 304 · see citing works
Mastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247 · see citing works
TextWorld: A Learning Environment for Text-Based Games
Côté, M.-A., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J. A. Y., Hausknecht, M., Asri, L. E., Adada, M., et al. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-24337-1_3 · Cited by 210 · see citing works
Learning to Speak and Act in a Fantasy Text Adventure Game
Urbanek, J., Fan, A., Karamcheti, S., Jain, S., Humeau, S., Dinan, E., Rocktäschel, T., Kiela, D., Szlam, A. D., & Weston, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1062 · Cited by 163 · see citing works
Constrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139 · cited 1× · see citing works
“While we are not generating in a formal syntax as in semantic parsing, we are still constrained by a controlled sublanguage (Shin et al., 2021) , which is closer to natural language but follows a grammar defined by the engine’s parser.”
BackgroundInteractive Fiction Games: A Colossal Adventure
Hausknecht, M., Ammanabrolu, P., Côté, M.-A., & Yuan, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6297 · Cited by 122 · see citing works
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., & Fergus, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17276 · Cited by 108 · see citing works
Keep CALM and Explore: Language Models for Action Generation in Text-based Games
Yao, S., Rao, R., Hausknecht, M., & Narasimhan, K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.704 · Cited by 75 · see citing works
Graph Constrained Reinforcement Learning for Natural Language Action SpacesPreprint
Ammanabrolu, P., & Hausknecht, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08837 · Cited by 63 · see citing works
Learning Dynamic Belief Graphs to Generalize on Text-Based GamesPreprint
Adhikari, A., Yuan, X., Côté, M.-A., Zelinka, M., Rondeau, M.-A., Laroche, R., Poupart, P., Tang, J., Trischler, A., & Hamilton, W. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09127 · Cited by 58 · see citing works
(Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Da, J., Sakaguchi, K., Bosselut, A., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16792 · Cited by 53 · see citing works
Transfer in Deep Reinforcement Learning Using Knowledge Graphs
Ammanabrolu, P., & Riedl, M. O. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5301 · Cited by 33 · see citing works
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering
Bosselut, A., Bras, R. J. L., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16625 · Cited by 25 · see citing works
Playing Text-Based Games with Common SensePreprint
Dambekodi, S. N., Frazier, S., Ammanabrolu, P., & Riedl, M. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.02757 · Cited by 21 · see citing works
Learning Knowledge Graph-based World Models of Textual EnvironmentsPreprint
Ammanabrolu, P., & Riedl, M. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09608 · Cited by 16 · see citing works
Murugesan, K., Atzeni, M., Kapanipathi, P., Talamadupula, K., Sachan, M., & Campbell, M. S. (2021) · doi:10.18653/v1/2021.acl-short.91 · Cited by 7 · see citing works
Fire Burns, Sword Cuts: Commonsense Inductive Bias for Exploration in Text-based Games
Ryu, D., Shareghi, E., Fang, M., Xu, Y., Pan, S., & Haf, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.56 · Cited by 6 · see citing works
Flagged first, then by citations
1 of the 29 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Prithviraj Ammanabrolu and Mark O. Riedl. 2021b. Modeling worlds in text. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
- Mattia Atzeni, Shehzaad Dhuliawala, Keerthiram Mu- rugesan, and Mrinmaya Sachan. 2021. Case-based reasoning for better generalization in text-adventure games. CoRR, abs/2110.08470. · arXiv
- Ishika Singh, Gargi Singh, and Ashutosh Modi. 2022. Pre-trained language models as prior knowledge for playing text-based games. In 21st International Con- ference on Autonomous Agents and Multiagent Sys- tems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages 1729-1731. International Founda- tion for Autonomous Agents and Multiagent Systems (IFAAMAS). · DOI
Cited by
Sorted by citations
Neural network developments: A detailed survey from static to dynamic models
Verma, P. R., Singh, N. P., Pantola, D., & Cheng, X. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109710 · Cited by 25 · see citing works
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9 · see citing works
“This includes papers on simulating non-player characters (NPCs) in text adventure games [13] multiple agents in human-like societies [15] or specific collaborative tasks [21], and enabling human-in-the-loop interaction for tasks like online shopping [6] or web search [28].”
BackgroundWang, X., Li, L., & Zeng, D. D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.309 · Cited by 1 · see citing works
Multi-Task Reinforcement Learning with Cost-based HTN Planning
Hu, Y., & Zhuo, H. H. (2024) · International Conference on Computer Engineering and Application (ICCEA) · doi:10.1109/iccea62105.2024.10603549 · Cited by 2 · see citing works
Remember what you did so you know what to do next
Ciosici, M. R., Hedges, A., Kankanampati, Y., Martin, J. D., Freedman, M., & Weischedel, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.104
- Authors
- Kim, M., Jung, Y., Lee, D., & Hwang, S.-W.
- Affiliations
- Seoul National University (KR)
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 1324-1341
- OpenAlex
- W4385573730
- Cited by
- 4
- References
- 29
- Citations by year
- 2024: 2 · 2023: 2
- FWCI
- 0.55
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Seoul National University (2021-0-01343)
- Ministry of Science and ICT, South Korea (NO.2021-0-01343, 2022-0-00077, 2021-0-01343)
- Institute for Information and Communications Technology Promotion (2021–0-01343, 2022-)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- world models, text-based games, pre-trained language models, action prediction, constrained decoding, future prediction, sample efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
