CoChat
Sign Up
Cited by 82Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsRobot Manipulation and Learning

HOP: History-and-Order Aware Pretraining for Vision-and-Language Navigation

Open access
Qiao, Yanyuan, Qi, Yuankai, Hong, Yicong, Yu, Zheng, Wang, Peng, Wu, Qi·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.01498
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretraining has been adopted in a few of recent works for Vision-and-Language Navigation (VLN). However, pre-vious pre-training methods for VLN either lack the ability to predict future actions or ignore the trajectory contexts, which are essential for a greedy navigation process. In this work, to promote the learning of spatio-temporal visual-textual correspondence as well as the agent's capability of decision making, we propose a novel history-and-order aware pre-training paradigm (HOP) with VLN-specific objectives that exploit the past observations and support future action prediction. Specifically, in addition to the commonly used Masked Language Modeling (MLM) and Trajectory-Instruction Matching (TIM), we design two proxy tasks to model temporal order information: Trajectory Order Modeling (TOM) and Group Order Modeling (GOM). Moreover, our navigation action prediction is also enhanced by intro-ducing the task of Action Prediction with History (APH), which takes into account the history visual perceptions. Extensive experimental results on four downstream VLN tasks (R2R, REVERIE, NDH, RxR) demonstrate the effectiveness of our proposed method compared against several state-of-the-art agents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01498

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing vision-and-language navigation datasets face a common data scarcity issue.

    “To tackle the common data scarcity issue in existing vision-and-language navigation datasets”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [60]

    Section 3.3 (Pre-training Datasets; page unavailable) describes augmenting R2R with synthesized instructions, and Section 4.4 (Ablation Study; page unavailable) reports a benefit from additional pre-training data. These passages support a need for more data in the paper’s approach, but do not establish that data scarcity is a common issue across existing VLN datasets. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

  • The paradigm facilitates different models to achieve new state-of-the-art navigation results on CVDN, REVERIE, and R2R in continuous environments.

    “our paradigm also facilitates different models to achieve new state-of-the-art navigation results on CVDN, REVERIE, and R2R in continuous environments.”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [60]

    The Introduction and Section 4.1 evaluate HOP on R2R, REVERIE, and NDH; the appendix identifies CVDN as the dataset used for NDH. The REVERIE results report improvements over previous methods. But the paper evaluates its own fine-tuned model, not different models enabled by the citing authors’ paradigm, and its Introduction describes navigation on a predefined connectivity graph rather than in continuous environments. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • An agent trained with augmented instructions generated by a simple LSTM-based model can achieve good performance on multiple navigation tasks.

    “an agent trained with augmented instructions generated by a simple LSTM-based model [74] can achieve good performance on multiple navigation tasks [10, 60, 75].”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. Dong An, Yuankai Qi, Yan Huang, Qi Wu, Liang Wang, and Tieniu Tan. Neighbor-view enhanced model for vision and language navigation. In ACM MM, 2021. 6
  2. Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navi- gation agents. CoRR, abs/1807.06757, 2018. 6 · arXiv
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: In- terpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674-3683, 2018. 1, 2, 5, 11
  4. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. TOUCHDOWN: natural language naviga- tion and spatial reasoning in visual street environments. In CVPR, pages 12538-12547, 2019. 1
  5. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, pages 104-120, 2020. 2
  6. Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision- and-language navigation via pre-training. In CVPR, pages 13134-13143, 2020. 1, 2, 4, 5, 6, 11, 12
  7. Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhães, Jason Baldridge, and Eugene Ie. Trans- ferable representation learning in vision-and-language navi- gation. In ICCV, pages 7403-7412, 2019. 1, 3
  8. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Vié- gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. TACL, 5:339-351, 2017. 3
  9. Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Sid- dhartha S. Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In CVPR, pages 6741-6749, 2019. 6, 7
  10. Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, ed- itors, ECCV, volume 12373 of Lecture Notes in Computer Science, pages 104-120. Springer, 2020. 1
  11. Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision- and-language navigation with dense spatiotemporal ground- ing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP, pages 4392-4412, 2020. 1, 2, 5, 7, 11
  12. Jialu Li, Hao Tan, and Mohit Bansal. Improving cross- modal alignment in vision language navigation via syntactic information. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, ed- itors, NAACL, pages 1041-1050, 2021. 7, 12
  13. Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Çelikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In EMNLP, pages 1494-1499, 2019. 6
  14. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137, 2020. 2
  15. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. OpenReview.net, 2019. 5
  16. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, pages 13-23, 2019.
  17. Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR, pages 6732-6740, 2019. 6
  18. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter An- derson, Devi Parikh, and Dhruv Batra. Improving vision- and-language navigation with image-text pairs from the web. In ECCV, pages 259-274, 2020. 1, 2, 5, 11, 12
  19. Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assis- tance via imitation learning with indirect intervention. In CVPR, pages 12527-12537. Computer Vision Foundation / IEEE, 2019. 1
  20. Khanh Nguyen and Hal Daumé III. Help, anna! visual nav- igation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, EMNLP, pages 684-695. Association for Computational Linguistics, 2019. 1
  21. Yuankai Qi, Zizheng Pan, Yicong Hong, Ming-Hsuan Yang, Anton van den Hengel, and Qi Wu. The road to know- where: An object-and-room informed sequential bert for indoor vision-language navigation. In ICCV, pages 1655- 1664, 2021. 2, 6, 7
  22. Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. Object-and-action aware model for vi- sual language navigation. In ECCV, pages 303-317, 2020.
  23. Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. REVERIE: remote embodied visual referring ex- pression in real indoor environments. In CVPR, pages 9979- 9988, 2020. 1, 2, 5, 7, 11
  24. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition chal- lenge. IJCV, 115(3):211-252, 2015. 3
  25. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL, pages 2556-2565, 2018. 12
  26. Hao Tan and Mohit Bansal. LXMERT: learning cross- modality encoder representations from transformers. In EMNLP, pages 5099-5110, 2019. 2, 3
  27. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL, pages 394-406, 2019. 1, 2, 5, 6, 11
  28. Xin Wang, Qiuyuan Huang, Asli Çelikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self- supervised imitation learning for vision-language navigation. In CVPR, pages 6629-6638, 2019. 1, 2, 6, 7
  29. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expres- sions. In ECCV, pages 69-85, 2016. 1, 2
  30. Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. SOON: scenario oriented object navigation with graph-based exploration. In CVPR, pages 12689-12699, 2021. 1
  31. Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang. Vision-dialog nav- igation by exploring cross-modal memory. In CVPR, pages 10727-10736, 2020.

Cited by

Sorted by citations

Authors
Qiao, Y., Qi, Y., Hong, Y., Yu, Z., Wang, P., & Wu, Q.
Affiliations
  • The University of Adelaide (AU)
  • Australian National University (AU)
  • The University of Adelaide
  • Northwestern Polytechnical University (CN)
ORCID
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
15397-15406
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.01498
OpenAlex
W4312938887
Cited by
82
References
10
Citations by year
2026: 10 · 2025: 21 · 2024: 21 · 2023: 27 · 2022: 3
FWCI
6.37
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
vision-and-language navigation, pre-training, action prediction, masked language modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review