CoChat
Sign Up
PreprintCited by 15Reinforcement Learning in RoboticsArtificial Intelligence in GamesMultimodal Machine Learning Applications

Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning

Open access
Hanjie, Austin W., Zhong, Victor, Narasimhan, Karthik·2021·arXiv
DOI 10.48550/arxiv.2101.07393arXiv 2101.07393
Open this paper in CoChat — free accountOpen PDF

Abstract

We investigate the use of natural language to drive the generalization of control policies and introduce the new multi-task environment Messenger with free-form text manuals describing the environment dynamics. Unlike previous work, Messenger does not assume prior knowledge connecting text and state observations $-$ the control policy must simultaneously ground the game manual to entity symbols and dynamics in the environment. We develop a new model, EMMA (Entity Mapper with Multi-modal Attention) which uses an entity-conditioned attention module that allows for selective focus over relevant descriptions in the manual for each entity in the environment. EMMA is end-to-end differentiable and learns a latent grounding of entities and dynamics from text to observations using only environment rewards. EMMA achieves successful zero-shot generalization to unseen games with new dynamics, obtaining a 40% higher win rate compared to multiple baselines. However, win rate on the hardest stage of Messenger remains low (10%), demonstrating the need for additional work in this direction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2101.07393

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The original MESSENGER manuals are challenging to understand even for human readers.

    “The manual, by design, is challenging to understand even for human readers.”

    From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hanjie et al., 2021

    Section B.1 (p. 15) says MESSENGER’s entities were deliberately chosen to make grounding harder, and the Introduction (p. 2) notes extraneous descriptions and multiple synonyms in the manuals. But neither passage says the manuals are challenging for human readers; the human-playthrough description in Appendix B (p. 14) does not establish that either. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • SmartPlay features a text game (Messenger) with procedural game rule generation to test the generalization of the LLM agents at language understanding and planning.

    “SmartPlay features a text game (Messenger) with procedural game rule generation (Hanjie et al., 2021) to test the generalization of the LLM agents at language understanding and planning.”

    From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hanjie et al., 2021

  • To ensure compatibility with prior works, reward aligns with the score/reward definition in games originally designed for RL.

    “To ensure compatibility with prior works, reward aligns with the score/reward definition in games originally designed for RL (i.e., Bandits, Rock Paper Scissors, Messenger, Crafter (Hanjie et al., 2021; Hafner, 2021)).”

    From SmartPlay: A Benchmark for LLMs as Intelligent Agents (2023) · cites this paper as Hanjie et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 34 · Cited by 15

Research works printed but not linked: 31

  1. Andreas, J. and Klein, D. Alignment-based compositional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pp. 1165-1174, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1138. URL https://www.aclw eb.org/anthology/D15-1138 . · DOI
  2. Andreas, J., Klein, D., and Levine, S. Modular multitask reinforcement learning with policy sketches. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pp. 166-175. PMLR, 2017. URL http://proceedings.mlr. press/v70/andreas17a.html .
  3. Bahdanau, D., Hill, F., Leike, J., Hughes, E., Kohli, P., and Grefenstette, E. Learning to follow language instructions with adversarial reward induction. arXiv preprint arXiv:1806.01946 , 2018. · arXiv
  4. Blukis, V., Terme, Y., Niklasson, E., Knepper, R. A., and Artzi, Y. Learning to map natural language instructions to physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.09664 , 2019. · arXiv
  5. Branavan, S., Silver, D., and Barzilay, R. Learning to win by reading manuals in a Monte-Carlo framework. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 268-277, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL https: //www.aclweb.org/anthology/P11-1028 .
  6. Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., and Mercer, R. L. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics , 19(2):263-311, 1993. URL https://www. aclweb.org/anthology/J93-2003 .
  7. Chen, D. L. and Mooney, R. J. Learning to interpret natural language navigation instructions from observations. In Burgard, W. and Roth, D. (eds.), Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011 . AAAI Press, 2011. URL http://www.aaai.org/ocs/index.php/AAA I/AAAI11/paper/view/3701 .
  8. Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations , 2019. URL https://openre view.net/forum?id=rJeXCo0cYX .
  9. Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/for um?id=HkgSEnA5KQ .
  10. Daniele, A. F., Bansal, M., and Walter, M. R. Navigational instruction generation as inverse reinforcement learning with neural machine translation. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI , pp. 109-118. IEEE, 2017.
  11. Fried, D., Andreas, J., and Klein, D. Unified pragmatic models for generating and following instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 1951-1963, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1177. URL https://www.aclw eb.org/anthology/N18-1177 . · DOI · Paper page
  12. Golland, D., Liang, P., and Klein, D. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing , pp. 410-419, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https://www.aclweb.org/anthology /D10-1040 .
  13. Hemachandra, S., Walter, M. R., Tellex, S., and Teller, S. Learning spatial-semantic representations from natural language descriptions and scene classifications. In Robotics and Automation (ICRA), 2014 IEEE International Conference on , pp. 2623-2630. IEEE, 2014.
  14. Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R., Soyer, H., Szepesvari, D., Czarnecki, W. M., Jaderberg, M., Teplyashin, D., et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551 , 2017. · arXiv
  15. April 26-30, 2020 . OpenReview.net, 2020a. URL https: //openreview.net/forum?id=SklGryBtwr . https://openreview.net/forum?id=SklGryBtwr
  16. Hill, F., Mokra, S., Wong, N., and Harley, T. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382 , 2020b. · arXiv
  17. Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., and Clark, S. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01719 , 2020c. · arXiv
  18. Hu, R., Fried, D., Rohrbach, A., Klein, D., Darrell, T., and Saenko, K. Are you looking? grounding to multiple modalities in vision-and-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 6551-6557, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1655. URL https: //www.aclweb.org/anthology/P19-1655 . · DOI
  19. Janner, M., Narasimhan, K., and Barzilay, R. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics , 6:4961, 2018. doi: 10.1162/tacl a 00004. URL https: //www.aclweb.org/anthology/Q18-1004 . · DOI
  20. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http: //arxiv.org/abs/1412.6980 .
  21. K¨ uttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., and Rockt¨ aschel, T. The nethack learning environment. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurip s.cc/paper/2020/hash/569ff987c643b4b edf504efda8f786c2-Abstract.html .
  22. Luketina, J., Nardelli, N., Farquhar, G., Foerster, J. N., Andreas, J., Grefenstette, E., Whiteson, S., and Rockt¨ aschel, T. A survey of reinforcement learning informed by natural language. In Kraus, S. (ed.), Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019 , pp. 6309-6317. ijcai.org, 2019. doi: 10.24963/ijcai.2019/880. URL https://doi.org/ 10.24963/ijcai.2019/880 . · DOI
  23. Misra, D., Langford, J., and Artzi, Y. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pp. 1004-1015, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-11 06. URL https://www.aclweb.org/antholo gy/D17-1106 . · DOI
  24. Narasimhan, K., Barzilay, R., and Jaakkola, T. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research , 63:849-874, 2018.
  25. Oh, J., Singh, S. P., Lee, H., and Kohli, P. Zero-shot task generalization with multi-task deep reinforcement learning. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pp. 2661-2670. PMLR, 2017. URL http://procee dings.mlr.press/v70/oh17a.html .
  26. Reckman, H., Orkin, J., and Roy, D. Learning meanings of words and constructions, grounded in a virtual game. Semantic Approaches in Natural Language Processing , pp. 67, 2010.
  27. Tellex, S., Gopalan, N., Kress-Gazit, H., and Matuszek, C. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems , 3:25-55, 2020.
  28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp. 5998-6008, 2017. URL https://proceedings.neurips.cc/paper /2017/hash/3f5ee243547dee91fbd053c1c 4a845aa-Abstract.html .
  29. Wang, X., Huang, Q., C ¸ elikyilmaz, A., Gao, J., Shen, D., Wang, Y., Wang, W. Y., and Zhang, L. Reinforced crossmodal matching and self-supervised imitation learning for vision-language navigation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pp. 6629-6638. Computer Vision Foundation / IEEE, 2019. doi: 10.110 9/CVPR.2019.00679. URL http://openaccess .thecvf.com/content CVPR 2019/html/Wan g Reinforced Cross-Modal Matching and S elf-Supervised Imitation Learning for Vision-Language Navigation CVPR 2019 p aper.html .
  30. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface's transformers: State-of-theart natural language processing. ArXiv , abs/1910.03771, 2019. · arXiv
  31. Zhong, V., Rockt¨ aschel, T., and Grefenstette, E. RTFM: generalising to new environment dynamics via reading. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https: //openreview.net/forum?id=SJgob6NKvH .

Authors
Hanjie, A. W., Zhong, V., & Narasimhan, K.
Affiliations
  • Princeton University (US)
  • University of Washington (US)
ORCID
Narasimhan, Karthik
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
4051-4062
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2101.07393
arXiv
2101.07393
OpenAlex
W3122267274
Cited by
15
References
34
Citations by year
2025: 1 · 2024: 2 · 2023: 5 · 2022: 3 · 2021: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
natural language control, policy generalization, zero-shot generalization, multimodal attention, environmental dynamics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review