CoChat
Sign Up
Cited by 39Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsSpatial Cognition and Navigation

A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning

Open access
Kamath, Aishwarya, Anderson, Peter, Wang, Su, Koh, Jing Yu, Ku, Alexander, Waters, Austin, Yang, Yinfei, Baldridge, Jason +1 more·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52729.2023.01041
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 ° panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky [63], a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN [27]. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale training on near-human quality synthetic instructions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01041

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Applying CLIP ViT-H/14 and EnvEdit to our approach leads to 80% SR.

    “we also experiment with applying a more powerful visual encoder, CLIP ViT-H/14 [64], and the image augmentation method EnvEdit [47] to our approach, leading to a remarkable 80% SR”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [37]

    The Approach section describes CLIP as an image encoder with a ViT backbone, supporting its identification as a visual encoder. It does not identify the ViT-H/14 variant; the paper says its own agent uses MURAL-large features instead. The citing authors’ 80% success rate is presented as the result of their own experiment, not as a result attributed to this source. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

  • The paper employs 800 training scenes from HM3D and 491 training and validation scenes from Gibson, resulting in more than 150k m2 navigable area, around 7.5 times larger than the training scenes of downstream MP3D environments (20k m2, 61 scans).

    “we employ 800 training scenes from HM3D, and 491 training and validation scenes from Gibson (same as MARVAL [37]), resulting in more than 150k m 2 navigable area, which is around ˆ 7.5 times larger than the training scenes of downstream MP3D environments (20k m 2 , 61 scans).”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [37]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • MARVAL trains a model to predict navigable directions, which could make errors and overcomplicate the problem.

    “MARVAL [37] trains a model to predict navigable directions, which could make errors and overcomplicate the problem.”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [37]

  • Scaling up the diversity of environments and the quantity of demonstration for training VLN agents is promising for improving generalization to unseen scenes.

    “Recent research shows that scaling up the diversity of environments and the quantity of demonstration for training VLN agents are promising in improving generalization to unseen scenes [17, 37].”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [37]

  • The process toward large-scale training involves key sub-problems such as building navigation graphs, recovering corrupted rendered images, and generating navigational instructions, which significantly influence the quality of collected data.

    “the process towards such large-scale training involves solving a series of key sub-problems such as how to build navigation graphs [10, 30, 37], how to recover corrupted rendered images [8, 43], and how to generate navigational instructions [23, 25, 78, 83], which significantly influence the quality of collected data”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On Evaluation of Embodied Navigation Agents. ArXiv preprint, 2018. 7
  2. Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Sim- to-real transfer for vision-and-language navigation. In CoRL, 2021. 2
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: In- terpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3674-3683. IEEE Com- puter Society, 2018. 1, 2, 3, 5, 6, 8
  4. Jacob Andreas and Dan Klein. Alignment-based composi- tional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1165-1174, Lisbon, Portugal, 2015. Association for Computational Linguistics. 2
  5. Yoav Artzi and Luke Zettlemoyer. Weakly supervised learn- ing of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguis- tics, 1:49-62, 2013. 2
  6. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob An- dreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Ange- liki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nico- las Pinto, and Joseph Turian. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 8718- 8735, Online, 2020. Association for Computational Linguis- tics. 2
  7. David L. Chen and Raymond J. Mooney. Learning to inter- pret natural language navigation instructions from observa- tions. In Wolfram Burgard and Dan Roth, editors, Proceed- ings of the Twenty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press, 2011. 2
  8. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. TOUCHDOWN: natural language naviga- tion and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12538-12547. Computer Vision Foundation / IEEE, 2019. 1, 2
  9. Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. In NeurIPS, 2021. 2, 3, 4, 5, 8
  10. Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navi- gation. In CVPR, 2022. 3, 5
  11. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Brad- bury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-image model. ArXiv preprint, 2022. 2
  12. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 3
  13. Meera Hahn, Jacob Krantz, Dhruv Batra, Devi Parikh, James Rehg, Stefan Lee, and Peter Anderson. Where are you? lo- calization from embodied dialog. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 806-822, Online, 2020. Associa- tion for Computational Linguistics. 2
  14. Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision- and-language navigation via pre-training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13134-13143. IEEE, 2020. 3, 4, 8
  15. Yicong Hong, Cristian Rodriguez Opazo, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity rela- tionship graph for agent navigation. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 6-12, 2020, virtual, 2020. 1
  16. Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. Effective and general evaluation for in- struction conditioned navigation using dynamic time warp- ing. NeurIPS Visually Grounded Interaction and Language Workshop, 2019. 7
  17. Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. MURAL: Multimodal, multitask representations across languages. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 3449-3463, Punta Cana, Dominican Republic, 2021. Association for Computa- tional Linguistics. 3
  18. Jindong Jiang, Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. ArXiv preprint, 2018. 6
  19. Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr -mod- ulated detection for end-to-end multi-modal understanding. ICCV, 2021. 1
  20. Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Simple and effective synthesis of indoor 3d scenes. In AAAI, 2023. 1, 2, 3, 6, 7
  21. Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In ICCV, 2021. 3, 6
  22. Joseph B Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society, 1956. 6
  23. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision- and-language navigation with dense spatiotemporal ground- ing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392-4412, Online, 2020. Association for Computational Linguistics. 1, 2, 5, 6, 8
  24. Taku Kudo and John Richardson. SentencePiece: A sim- ple and language independent subword tokenizer and detok- enizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pages 66-71, Brussels, Belgium, 2018. Association for Computational Linguistics. 4
  25. Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021. 1
  26. Jialu Li, Hao Tan, and Mohit Bansal. CLEAR: Improving vision-language navigation with cross-lingual, environment- agnostic representations. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 633-649, Seattle, United States, 2022. Association for Computational Linguistics. 3, 8
  27. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 1
  28. Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 1494-1499, Hong Kong, China, 2019. Association for Computational Linguistics. 2
  29. Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 1494-1499, Hong Kong, China, 2019. Association for Computational Linguistics. 4
  30. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter An- derson, Devi Parikh, and Dhruv Batra. Improving vision- and-language navigation with image-text pairs from the web. In ECCV, 2020. 1, 2, 3, 4
  31. Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. Retouchdown: Adding touchdown to streetlearn as a shareable resource for language grounding tasks in street view. ArXiv preprint, 2020. 2
  32. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. Lis- ten, attend, and walk: Neural mapping of navigational in- structions to action sequences. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12- 17, 2016, Phoenix, Arizona, USA, pages 2772-2778. AAAI Press, 2016. 2
  33. Dipendra Misra, John Langford, and Yoav Artzi. Map- ping instructions and visual observations to actions with re- inforcement learning. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 1004-1015, Copenhagen, Denmark, 2017. Asso- ciation for Computational Linguistics. 2
  34. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kil- ian Q. Weinberger, editors, Proceedings of the 33nd Interna- tional Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. JMLR.org, 2016. 2, 5
  35. Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal, Stefan Lee, and Dhruv Batra. Soat: A scene-and object-aware trans- former for vision-and-language navigation. NeurIPS, 2021. 1
  36. Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. REVERIE: remote embodied visual referring ex- pression in real indoor environments. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9979-9988. IEEE, 2020. 2
  37. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pages 8748-8763. PMLR, 2021. 3
  38. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, 2021. 1
  39. Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Ab- hishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In CVPR, 2022. 2
  40. Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Process- ing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91-99, 2015. 3
  41. Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010. 5
  42. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A re- duction of imitation learning and structured prediction to no- regret online learning. In AISTATS, 2011. 2
  43. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015. 3
  44. Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International Conference on Computer Vi- sion, ICCV 2019, Seoul, Korea (South), October 27 -Novem- ber 2, 2019, pages 9338-9346. IEEE, 2019. 6
  45. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neu- ral machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, 2016. Association for Computational Linguistics. 3
  46. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, Melbourne, Australia, 2018. Association for Computational Linguistics. 2
  47. Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit vision-and-language tasks? ArXiv, 2021. 3, 8
  48. Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Ka- malika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 2019. 3
  49. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL, 2019. 2
  50. Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birod- kar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Peter Anderson. Less is more: Generating grounded navigation instructions from landmarks. In CVPR, 2022. 1, 2, 3, 5, 6, 7
  51. Xin Wang, Qiuyuan Huang, Asli C ¸elikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self- supervised imitation learning for vision-language naviga- tion. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16- 20, 2019, pages 6629-6638. Computer Vision Foundation / IEEE, 2019. 5
  52. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple visual language model pretraining with weak supervision. ArXiv, 2021. 1
  53. Fei Xia, Amir Roshan Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 9068- 9079. IEEE Computer Society, 2018. 2, 5, 6
  54. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to- text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pages 483-498, Online, 2021. Association for Computational Lin- guistics. 3, 7
  55. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv preprint, 2022. 1
  56. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision-language models. CVPR, 2021. 1
  57. Yubo Zhang, Hao Tan, and Mohit Bansal. Diagnosing the en- vironment bias in vision-and-language navigation. In Chris- tian Bessiere, editor, Proceedings of the Twenty-Ninth Inter- national Joint Conference on Artificial Intelligence, IJCAI 2020, pages 890-897. ijcai.org, 2020. 6
  58. Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexan- der Ku, Jason Baldridge, and Eugene Ie. On the evaluation of vision-and-language navigation instructions. In Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1302-1316, Online, 2021. Association for Computa- tional Linguistics. 3
  59. Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, and William Yang Wang. Diagnosing vision-and-language navi- gation: What really matters. In Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5981-5993, Seattle, United States, 2022. Association for Computational Linguistics. 1

Cited by

Sorted by citations

Authors
Kamath, A., Anderson, P., Wang, S., Koh, J. Y., Ku, A., Waters, A., Yang, Y., Baldridge, J., & Parekh, Z.
Affiliations
  • New York University (US)
  • Google (United States) (US)
  • Carnegie Mellon University (US)
  • Apple (United States) (US)
ORCID
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
10813-10823
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52729.2023.01041
OpenAlex
W4386076252
Cited by
39
References
12
Citations by year
2026: 8 · 2025: 9 · 2024: 16 · 2023: 6
FWCI
3.97
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
vision-and-language navigation, imitation learning, language grounding, spatial language comprehension, 360-degree panorama, robot navigation, unknown environments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review