CoChat
Sign Up
Cited by 16Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsSpatial Cognition and Navigation

FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation

Open access
Dou, Zi-Yi, Peng, Nanyun·2022·Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
DOI 10.18653/v1/2022.naacl-main.322
Open this paper in CoChat — free accountOpen PDF

Abstract

The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model.However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower.In this paper, we present FOAM, a FOllower-Aware speaker Model that is constantly updated given the follower feedback, so that the generated instructions can be more suitable to the current learning state of the follower.Specifically, we optimize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data.Experimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings.Analyses also reveal that our generated instructions are of higher quality than the baselines.1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.naacl-main.322

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The process toward large-scale training involves key sub-problems such as building navigation graphs, recovering corrupted rendered images, and generating navigational instructions, which significantly influence the quality of collected data.

    “the process towards such large-scale training involves solving a series of key sub-problems such as how to build navigation graphs [10, 30, 37], how to recover corrupted rendered images [8, 43], and how to generate navigational instructions [23, 25, 78, 83], which significantly influence the quality of collected data”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. 2018a. On evaluation of em- bodied navigation agents. arXiv preprint.
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition (CVPR).
  3. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matter- port3d: Learning from RGB-D data in indoor envi- ronments. In Proceedings of the International Con- ference on 3D Vision (3DV).
  4. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the Confer- ence on Computer Vision and Pattern Recognition (CVPR).
  5. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
  6. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML).
  7. Daniel Fried, Jacob Andreas, and Dan Klein. 2018a. Unified pragmatic models for generating and follow- ing instructions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
  8. Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pre- training. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition (CVPR).
  9. Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. 2020. Language and visual en- tity relationship graph for agent navigation. In Pro- ceedings of the Conference on Neural Information Processing Systems (NeurIPS).
  10. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Mul- tilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP).
  11. Shuhei Kurita and Kyunghyun Cho. 2021. Generative language-grounded policy in vision-and-language navigation with bayes' rule. In Proceedings of the In- ternational Conference on Learning Representations (ICLR).
  12. Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  13. Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. In Proceed- ings of the International Conference on Learning Representations (ICLR).
  14. Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. Self-monitoring navigation agent via auxil- iary progress estimation. In Proceedings of the In- ternational Conference on Learning Representations (ICLR).
  15. Gabriel Magalhães, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. In Workshop on Visually Grounded Interaction and Language (ViGIL).
  16. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Im- proving vision-and-language navigation with image- text pairs from the web. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV).
  17. Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language genera- tion systems. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL) Demonstrations.
  18. Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan-Chen, Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven em- bodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
  20. Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. 2021a. Meta pseudo labels. In Proceedings of the Conference on Computer Vision and Pattern Recog- nition (CVPR).
  21. Hieu Pham, Xinyi Wang, Yiming Yang, and Graham Neubig. 2021b. Meta back-translation. In Proceed- ings of the International Conference on Learning Representations (ICLR).
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML).
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the An- nual Meeting of the Association for Computational Linguistics (ACL).
  24. Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022. How much can clip benefit vision-and-language tasks? In Proceedings of the In- ternational Conference on Learning Representations (ICLR).
  25. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
  26. Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, and Jianbing Shen. 2021. Structured scene memory for vision-language navigation. In Proceed- ings of the Conference on Computer Vision and Pat- tern Recognition (CVPR).
  27. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the Conference on Computer Vision and Pattern Recog- nition (CVPR).
  28. Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language nav- igation. In Proceedings of the European Conference on Computer Vision (ECCV).
  29. Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anas- tasopoulos, Jaime Carbonell, and Graham Neubig. 2020. Optimizing data usage via differentiable re- wards. In Proceedings of the International Confer- ence on Machine Learning (ICML).
  30. Ronald J Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning. · DOI
  31. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2020. BERTScore: Evaluat- ing text generation with bert. In Proceedings of the International Conference on Learning Representa- tions (ICLR).
  32. Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexander Ku, Jason Baldridge, and Eugene Ie. 2021. On the evaluation of vision-and-language navigation instructions. In Proceedings of the Conference of the European Chapter of the Association for Computa- tional Linguistics (EACL).
  33. Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
  34. Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020b. Baby- walk: Going farther in vision-and-language naviga- tion by taking baby steps. In Proceedings of the An- nual Meeting of the Association for Computational Linguistics (ACL).

Cited by

Sorted by citations

Authors
Dou, Z.-Y., & Peng, N.
Affiliations
University of California, Los Angeles (US)
Corresponding author
Dou, Zi-Yi
ORCID
Peng, Nanyun
Venue
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Publisher
Association for Computational Linguistics
Published
2022
Pages
4332-4340
DOI
10.18653/v1/2022.naacl-main.322
OpenAlex
W4281817909
Cited by
16
References
4
Citations by year
2026: 1 · 2025: 3 · 2024: 5 · 2023: 7
FWCI
1.33
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Institutes of Health
  • Defense Advanced Research Projects Agency (N66001, N66001-19)
  • National Heart, Lung, and Blood Institute (R01HL152270)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
vision-and-language navigation, instruction generation, bi-level optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review