Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
Open accessAbstract
Advances in learning and representations have reinvigorated work that connects language to other modalities.A particularly exciting direction is Vision-and-Language Navigation (VLN), in which agents interpret natural language instructions and visual scenes to move through environments and reach goals.Despite recent progress, current research leaves unclear how much of a role language understanding plays in this task, especially because dominant evaluation metrics have focused on goal completion rather than the sequence of actions corresponding to the instructions.Here, we highlight shortcomings of current metrics for the Room-to-Room dataset (Anderson et al., 2018b) and propose a new metric, Coverage weighted by Length Score (CLS).We also show that the existing paths in the dataset are not ideal for evaluating instruction following because they are direct-to-goal shortest paths.We join existing short paths to form more challenging extended paths to create a new data set, Room-for-Room (R4R).Using R4R and CLS, we show that agents that receive rewards for instruction fidelity outperform agents that focus on goal completion.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
R4R extends the R2R dataset by concatenating two adjacent trajectories in R2R, resulting in longer navigation trajectories not biased by the shortest path prior.
“R4R extends the R2R dataset by concatenating two adjacent trajectories in R2R, resulting in longer navigation trajectories not biased by the shortest path prior.”
From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Speaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291 · see citing works
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
Ma, C.-Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1901.03035 · Cited by 200 · see citing works
Flagged first, then by citations
Research works printed but not linked: 32
- Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mot- taghi, Manolis Savva, et al. 2018a. On evalua- tion of embodied navigation agents. arXiv preprint arXiv:1807.06757 . · arXiv
- Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Michael Bain and Claude Sammut. 1999. A frame- work for behavioural cloning. In Machine Intelli- gence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995]. Oxford University, Oxford, UK, UK, pages 103-129.
- Jason Baldridge, Tania Bedrax-Weiss, Daphne Luong, Srini Narayanan, Bo Pang, Fernando Pereira, Radu Soricut, Michael Tseng, and Yuan Zhang. 2018. Points, paths, and playscapes: Large-scale spatial language understanding tasks set in the real world. In Proceedings of the First International Workshop on Spatial Language Understanding. Association for Computational Linguistics, New Orleans, pages 46-52.
- Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. 2018. Learning interpretable spatial opera- tions in a rich 3d blocks world. In Proceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI-18). New Orleans, USA.
- Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. 2018. Mapping navigation instructions to continuous control actions with position visita- tion prediction. In Proceedings of the Conference on Robot Learning.
- Howard Chen, Alane Suhr, Dipendra Misra, and Yoav Artzi. 2019. Touchdown: Natural language naviga- tion and spatial reasoning in visual street environ- ments. In Conference on Computer Vision and Pat- tern Recognition.
- Volkan Cirik, Yuan Zhang, and Jason Baldridge. 2018. Following formulaic map instructions in a street sim- ulation environment. NIPS Visually Grounded Inter- action and Language Workshop .
- Shreyansh Daftry, J. Andrew Bagnell, and Martial Hebert. 2016. Learning transferable policies for monocular reactive MAV control. In International Symposium on Experimental Robotics. Springer, pages 3-11.
- Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. CoRR abs/1807.03367. · arXiv
- Edsger W Dijkstra. 1959. A note on two problems in connexion with graphs. Numerische mathematik 1(1):269-271.
- J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venu- gopalan, S. Guadarrama, K. Saenko, and T. Dar- rell. 2017. Long-term recurrent convolutional net- works for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intel- ligence 39(4):677-691.
- H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollr, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig. 2015. From cap- tions to visual concepts and back. In 2015 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR). pages 1473-1482.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9(8):1735- 1780.
- Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and Eugene Ie. 2019. Multi-modal discriminative model for vision-and-language nav- igation. In Proceedings of the Combined Work- shop on Spatial Language Understanding and Grounded Communication for Robotics (SpLU- RoboNLP-2019). Association for Computational Linguistics, Minneapolis.
- G.J.M. Kruijff, Ivana Kruijff-Korbayova, Shanker Ke- shavdas, Benoit Larochelle, Miroslav Janicek, Fran- cis Colas, Ming Liu, Franois Pomerleau, Roland Siegwart, Mark Neerincx, Rosemarijn Looije, Nanja Smets, Tina Mioch, Jurriaan Diggelen, Fiora Pirri, Mario Gianni, Federico Ferri, Matteo Menna, Rainer Worst, and Vaclav Hlavac. 2014. Designing, de- veloping, and deploying systems to support human- robot teams in disaster response. Advanced Robotics 28. I. Kruijff-Korbayov, L. Freda, M. Gianni, V. Ntouskos, V. Hlav, V. Kubelka, E. Zimmermann, H. Surmann, K. Dulic, W. Rottner, and E. Gissi. 2016. De- ployment of ground and aerial robots in earthquake- struck amatrice in italy (brief report). In 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). pages 278-279.
- Matt Macmahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, action in route instructions. In Proceed- ings of the National Conference on Artificial Intelli- gence (AAAI). pages 1475-1482.
- Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguistics, Brussels, Belgium, pages 2667- 2678.
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP). Association for Computational Lin- guistics, pages 1532-1543.
- M. Schuster and K.K. Paliwal. 1997. Bidirec- tional recurrent neural networks. Trans. Sig. Proc. 45(11):2673-2681.
- Pararth Shah, Marek Fiser, Aleksandra Faust, Chase Kew, and Dilek Hakkani-Tur. 2018. FollowNet: Robot navigation by following natural language di- rections with deep reinforcement learning. In Third Machine Learning in Planning and Control of Robot Motion Workshop at ICRA.
- D. Skočaj, A. Vrečko, M. Mahnič, M. Janíček, G.- J. M. Kruijff, M. Hanheide, N. Hawes, J. L. Wy- att, T. Keller, K. Zhou, M. Zillich, and M. Kristan. 2016. An integrated system for interactive contin- uous learning of categorical knowledge. Journal of Experimental & Theoretical Artificial Intelligence 28:823-848.
- Jesse Thomason, Daniel Gordon, and Yonatan Bisk. 2019. Shifting the baseline: Single modality perfor- mance on visual navigation & QA. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
- Jesse Thomason, Jivko Sinapov, Raymond Mooney, and Peter Stone. 2018. Guiding exploratory behav- iors for multi-modal grounding of linguistic descrip- tions. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18).
- Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. 2018. Video caption- ing via hierarchical reinforcement learning. 2018 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition pages 4213-4222.
- Xin Wang, Qiuyuan Huang, Asli C ¸elikyilmaz, Jian- feng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Re- inforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR).
- Edward C. Williams, Nakul Gopalan, Mina Rhee, and Stefanie Tellex. 2018. Learning to parse natural lan- guage to grounded reward functions with weak su- pervision. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018. pages 1-7.
- Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforce- ment learning. Machine Learning 8(3):229-256.
- Claudia Yan, Dipendra Misra, Andrew Bennett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. 2018. CHALET: Cornell House Agent Learning Environ- ment. CoRR abs/1801.07357. · arXiv
- Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2016. Stacked attention networks for image question answering. 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR) pages 21-29.
- Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016. Video paragraph captioning using hi- erarchical recurrent neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pages 4584-4593.
Cited by
Sorted by citations
Structured Scene Memory for Vision-Language Navigation
Wang, H., Wang, W., Liang, W., Xiong, C., & Shen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00835 · Cited by 123 · see citing works
Envedit: Environment Editing for Vision-and-Language Navigation
Li, J., Tan, H., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01497 · Cited by 87 · see citing works
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80 · see citing works
Scaling Data Generation in Vision-and-Language Navigation
Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01103 · Cited by 64 · see citing works
“…in both the HM3D [66] and the Gibson [84] datasets, build navigation graphs for the environments based on the Habitat simulator [70], sample new trajectories and generate corresponding instructions [74], and train agents [16, 18] for solving downstream navigation tasks [10, 35, 44, 60, 75].”
BackgroundMethodReinforced Structured State-Evolution for Vision-Language Navigation
Chen, J., Gao, C., Meng, E., Zhang, Q., & Liu, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01501 · Cited by 45 · see citing works
Embodied navigation with multi-modal information: A survey from tasks to methodology
Wu, Y., Zhang, P., Gu, M., Zheng, J., & Bai, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102532 · Cited by 34 · see citing works
Hwang, M., Jeong, J., Kim, M., Oh, Y., & Oh, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00646 · Cited by 26 · see citing works
Wang, S., Zhou, D., Xie, L., Xu, C., Yan, Y., & Yin, E. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107320 · Cited by 11 · see citing works
- Authors
- Jain, V., Magalhaes, G., Ku, A., Vaswani, A., Ie, E., & Baldridge, J.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1862-1872
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2947178363
- Cited by
- 165
- References
- 4
- Citations by year
- 2026: 18 · 2025: 19 · 2024: 25 · 2023: 27 · 2022: 24 · 2021: 24 · 2020: 22 · 2019: 6
- FWCI
- 6.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Keywords
- vision-and-language navigation, evaluation metrics, shortest path, language comprehension, instruction following, goal achievement, reward function design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
