CoChat
Sign Up
Cited by 60Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsSpatial Cognition and Navigation

Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments

Open access
Raychaudhuri, Sonia, Wani, Saim, Patel, Shivansh, Jain, Unnat, Chang, Angel·2021·Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2021.emnlp-main.328
Open this paper in CoChat — free accountOpen PDF

Abstract

In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions.A challenge in this task is how to handle 'off the path' scenarios where an agent veers from a reference path.Prior work supervises the agent with actions based on the shortest path from the agent's location to the goal, but such goal-oriented supervision is often not in alignment with the instruction.Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow.In this work, we propose a simple and effective language-aligned supervision scheme, and a new metric that measures the number of sub-instructions the agent has completed during navigation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.emnlp-main.328

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent large datasets contain hundreds of interactive scenes and vast human demonstrations, leading to significant improvement in agent performance.

    “recent large datasets which contain hundreds of interactive scenes have been created [20, 66, 84], as well as a vast amount of human demonstrations have been collected [45, 67] for learning visual navigation, leading to significant improvement in agent's performance.”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [67]

    The Experiments section reports 4,475 trajectories and about 150,000 augmented trajectories used to train a navigation agent, supporting the claim that substantial navigation data were available. It does not identify those trajectories as human demonstrations or report hundreds of interactive scenes. The Conclusion attributes improved instruction following to language-aligned supervision rather than to dataset size. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, and Manolis Savva. 2018a. On evalua- tion of embodied navigation agents. arXiv preprint arXiv:1807.06757. · arXiv
  2. Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, and Stefan Lee. 2019. Chasing ghosts: Instruction following as bayesian state tracking. In Advances in neural information processing systems.
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
  4. Angel Chang, Angela Dai, Thomas Funkhouser, Ma- ciej Halber, Matthias Niebner, Manolis Savva, Shu- ran Song, Andy Zeng, and Yinda Zhang. 2017. Mat- terport3D: Learning from RGB-D data in indoor en- vironments. In Proc. of International Conference on 3D Vision (3DV).
  5. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Nat- ural language navigation and spatial reasoning in vi- sual street environments. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
  6. Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. 2021. Topolog- ical Planning with Transformers for Vision-and- Language Navigation. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
  7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
  8. Yicong Hong, Cristian Rodriguez-Opazo, Qi Wu, and Stephen Gould. 2020. Sub-instruction aware vision- and-language navigation. In Proc. of the conference on empirical methods in natural language processing (EMNLP).
  9. Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. 2019. Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation. In Proc. of the Conference of the Association for Computational Linguistics (ACL).
  10. Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446. · arXiv
  11. Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and- language navigation. In Proc. of the Conference of the Association for Computational Linguistics (ACL).
  12. Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. 2021. Waypoint Models for Instruction-guided Navigation in Con- tinuous Environments. In Proc. of International Conference on Computer Vision (ICCV).
  13. Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the Nav- Graph: Vision-and-Language Navigation in Con- tinuous Environments. In Proc. of European Conference on Computer Vision (ECCV).
  14. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proc. of the conference on empirical methods in natural language processing (EMNLP).
  15. Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. 2020. Retouchdown: Adding Touchdown to StreetLearn as a shareable resource for language grounding tasks in street view. In Proc. of the Third International Workshop on Spatial Language Understanding.
  16. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems.
  17. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word rep- resentation. In Proc. of the conference on empirical methods in natural language processing (EMNLP).
  18. Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. 2020. Object-and- action aware model for visual language naviga- tion. In Proc. of European Conference on Computer Vision (ECCV).
  19. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proc. of the International Conference on Artificial Intelligence and Statistics (AISTATS).
  20. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, and Jitendra Malik. 2019. Habitat: A platform for embodied AI research. In Proc. of International Conference on Computer Vision (ICCV).
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, volume 30, pages 5998-6008.
  22. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jian- feng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Re- inforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
  23. Erik Wijmans, Abhishek Kadian, Ari Morcos, Ste- fan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. 2020. DD-PPO: Learning near- perfect pointgoal navigators from 2.5 billion frames. In Proc. of International Conference on Learning Representations (ICLR).
  24. Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020. Baby- Walk: Going farther in vision-and-language naviga- tion by taking baby steps. In Proc. of the Conference of the Association for Computational Linguistics (ACL).

Cited by

Sorted by citations

Authors
Raychaudhuri, S., Wani, S., Patel, S., Jain, U., & Chang, A.
Affiliations
  • Simon Fraser University (CA)
  • Indian Institute of Technology Kanpur (IN)
  • University of Illinois Urbana-Champaign (US)
ORCID
Raychaudhuri, Sonia
Venue
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2021
Pages
4018-4028
DOI
10.18653/v1/2021.emnlp-main.328
OpenAlex
W3203338521
Cited by
60
References
5
Citations by year
2026: 24 · 2025: 9 · 2024: 12 · 2023: 8 · 2022: 7
FWCI
2.17
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Empirical Methods in Natural Language Processing (EMNLP)
Funders
  • Canadian Institute for Advanced Research
  • Compute Canada
  • Western Canada Research Grid
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
vision-and-language navigation, natural language instructions, embodied agents, continuous environments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review