Neighbor-view Enhanced Model for Vision and Language Navigation
Open accessAbstract
Vision and Language Navigation (VLN) requires an agent to navigate to a target location by following natural language instructions. Most of existing works represent a navigation candidate by the feature of the corresponding single view where the candidate lies in. However, an instruction may mention landmarks out of the single view as references, which might lead to failures of textual-visual matching of existing methods. In this work, we propose a multi-module Neighbor-View Enhanced Model (NvEM) to adaptively incorporate visual contexts from neighbor views for better textual-visual matching. Specifically, our NvEM utilizes a subject module and a reference module to collect contexts from neighbor views. The subject module fuses neighbor views at a global level, and the reference module fuses neighbor objects at a local level. Subjects and references are adaptively determined via attention mechanisms. Our model also includes an action module to utilize the strong orientation guidance (e.g., "turn left'') in instructions. Each module predicts navigation action separately and their weighted sum is used for predicting the final action. Extensive experimental results demonstrate the effectiveness of the proposed method on the R2R and R4R benchmarks against several state-of-the-art navigators, and NvEM even beats some pre-training ones. Our code is available at https://github.com/MarSaKi/NvEM.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chat-Scene and Scene-LLM optimize structured embeddings and object-based feature representations, excelling in object-related queries and scene reasoning.
“Chat-Scene [286] and Scene-LLM [285] optimize structured embeddings and object-based feature representations, excelling in object-related queries and scene reasoning tasks.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [286]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
An overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962 · see citing works
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · see citing works
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503 · see citing works
Matterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & van den Hengel, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01000 · Cited by 331 · see citing works
Self-citationLearning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Tan, H., Yu, L., & Bansal, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1268 · Cited by 293 · see citing works
Speaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291 · see citing works
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
Krantz, J., Wijmans, E., Majumdar, A., Batra, D., & Lee, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_7 · Cited by 266 · see citing works
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training
Hao, W., Li, C., Li, X., Carin, L., & Gao, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01315 · Cited by 264 · see citing works
Vision-language navigation with self-supervised auxiliary reasoning tasks
Zhu, F., Zhu, Y., Chang, X., & Liang, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01003 · Cited by 223 · see citing works
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
Ma, C.-Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1901.03035 · Cited by 200 · see citing works
Wang, X. E., Xiong, W., Wang, H., & Wang, W. Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_3 · Cited by 187 · see citing works
Nguyen, K., & Daumé III, H. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1063 · Cited by 115 · see citing works
Object-and-action aware model for visual language navigation
Qi, Y., Pan, Z., Zhang, S., van den Hengel, A., & Wu, Q. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58607-2_18 · Cited by 100 · see citing works
Self-citationRobust Navigation with Language Pretraining and Stochastic Sampling
Li, X., Li, C., Xia, Q., Bisk, Y., Çelikyılmaz, A., Gao, J., Smith, N. A., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1159 · Cited by 97 · see citing works
MAttNet: Modular Attention Network for Referring Expression ComprehensionPreprint
Yu, L., Lin, Z., Shen, X., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.08186 · Cited by 89 · see citing works
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
Hu, R., Fried, D., Rohrbach, A., Klein, D., Darrell, T., & Saenko, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1655 · Cited by 76 · see citing works
Thomason, J., Murray, M. H., Çakmak, M., & Zettlemoyer, L. (2019) · Cited by 68 · see citing works
Environment-Agnostic Multitask Learning for Natural Language Grounded Navigation
Wang, X. E., Jain, V., Ie, E., Wang, W. Y., Kozareva, Z., & Ravi, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58586-0_25 · Cited by 54 · see citing works
Soft Expert Reward Learning for Vision-and-Language Navigation
Wang, H., Wu, Q., & Shen, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_8 · Cited by 41 · see citing works
Self-citationA Recurrent Vision-and-Language BERT for NavigationPreprint
Hong, Y., Wu, Q., Qi, Y., Rodríguez-Opazo, C., & Gould, S. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13922 · Cited by 38 · see citing works
Self-citationLocal-Global Video-Text Interactions for Temporal GroundingPreprint
Mun, J., Cho, M., & Han, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07514 · Cited by 27 · see citing works
Wang, X. E., Xiong, W., Wang, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.07729 · Cited by 22 · see citing works
Embodied Vision-and-Language Navigation with Dynamic Convolutional FiltersPreprint
Landi, F., Baraldi, L., Corsini, M., & Cucchiara, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02985 · Cited by 20 · see citing works
The Regretful Agent: Heuristic-Aided Navigation through Progress EstimationPreprint
Ma, C., Wu, Z., AlRegib, G., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01602 · Cited by 14 · see citing works
Robust Navigation with Language Pretraining and Stochastic SamplingPreprint
Li, X., Li, C., Xia, Q., Bisk, Y., Çelikyılmaz, A., Gao, J., Smith, N. A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02244 · Cited by 12 · see citing works
Nguyen, K., & Daumé, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01871 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments
An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., & Wang, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386695 · Cited by 103 · see citing works
Self-citationQi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80 · see citing works
Self-citationScaling Data Generation in Vision-and-Language Navigation
Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01103 · Cited by 64 · see citing works
“To address the problem, early research mainly focuses on developing task-specific models and training methods to better exploit visual-textual correspondence for decision making [2, 7, 21, 38, 48, 52, 58, 59, 77].”
Self-citationTarget-Driven Structured Transformer Planner for Vision-Language Navigation
Zhao, Y., Chen, J., Gao, C., Wang, W., Yang, L., Ren, H., Xia, H., & Liu, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548281 · Cited by 54 · see citing works
Reinforced Structured State-Evolution for Vision-Language Navigation
Chen, J., Gao, C., Meng, E., Zhang, Q., & Liu, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01501 · Cited by 45 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
Self-citationEmbodied navigation with multi-modal information: A survey from tasks to methodology
Wu, Y., Zhang, P., Gu, M., Zheng, J., & Bai, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102532 · Cited by 34 · see citing works
- Authors
- An, D., Qi, Y., Huang, Y., Wu, Q., Wang, L., & Tan, T.
- Affiliations
- Chinese Academy of Sciences (CN)
- The University of Adelaide (AU)
- Venue
- Proceedings of the 29th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2021-10-17
- Pages
- 5101-5109
- OpenAlex
- W3206064582
- Cited by
- 74
- References
- 34
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 21 · 2023: 19 · 2022: 9 · 2021: 3
- FWCI
- 4.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Shandong Provincial Key Research and Development Program (2019JZZY010119)
- Key Research Program of Frontier Sciences CAS Grant (ZDBS-LY-JSC032)
- National Key Research and Development Program of China Grant (2018AAA0100400)
- National Natural Science Foundation of China (61525306, 61633021, 61721004, 61806194, U1803261, and 61976132)
- Beijing Nova Program (Z201100006820079)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- vision-and-language navigation, natural language instructions, multi-view fusion, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
