Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
Open accessAbstract
Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On ScanQA, LLaVA-3D and 3D-LLM show strong generalization and excel in question-answering accuracy and spatial reasoning, mainly due to optimized multi-view feature extraction and 3D spatial embedding.
“Among them, LLaVA-3D [280] and 3D-LLM [282] demonstrate strong generalization capabilities, excelling in question-answering accuracy and spatial reasoning, primarily due to their optimized multi-view feature extraction and 3D spatial embedding mechanisms.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [280]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
Self-citationMatterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858 · see citing works
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · see citing works
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Self-citationSemantic Scene Completion from a Single Depth Image
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.28 · Cited by 1,244 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · see citing works
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187 · see citing works
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · see citing works
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763 · see citing works
(2016) · neural information processing systems · Cited by 735 · see citing works
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · see citing works
On evaluation of embodied navigation agents
(2018) · doi:10.48550/arxiv.1807.06757 · Cited by 622 · see citing works
AI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578 · see citing works
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.496 · Cited by 568 · see citing works
Natural Language Object Retrieval
Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.493 · Cited by 534 · see citing works
Mind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298856 · Cited by 496 · see citing works
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningPreprint
Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04717 · Cited by 486 · see citing works
Segmentation from Natural Language Expressions
Hu, R., Rohrbach, M., & Darrell, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_7 · Cited by 467 · see citing works
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445 · see citing works
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
An analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428 · see citing works
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00008 · Cited by 410 · see citing works
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210183 · Cited by 409 · see citing works
Self-citationNeural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000074 · Cited by 384 · see citing works
Self-citationCount-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377 · see citing works
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Zhang, D., Dai, X.-Y., Wang, X. E., Wang, Y.-F., & Davis, L. (2018) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00134 · Cited by 358 · see citing works
Self-citationSpeaker-Follower Models for Vision-and-Language Navigation
Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1806.02724 · Cited by 291 · see citing works
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00279 · Cited by 288 · see citing works
Video Captioning via Hierarchical Reinforcement Learning
Wang, X. E., Chen, W., Wu, J., Wang, Y.-F., & Wang, W. Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00443 · Cited by 269 · see citing works
Self-citationMINOS: Multimodal Indoor Simulator for Navigation in Complex EnvironmentsPreprint
Savva, M., Chang, A. X., Dosovitskiy, A., Funkhouser, T., & Koltun, V. (2017) · arXiv.org · doi:10.48550/arxiv.1712.03931 · Cited by 262 · see citing works
Visual Representations for Semantic Target Driven Navigation
Mousavian, A., Toshev, A., Fišer, M., Košecká, J., Wahid, A., & Davidson, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793493 · Cited by 227 · see citing works
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
Ma, C.-Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1901.03035 · Cited by 200 · see citing works
Wang, X. E., Xiong, W., Wang, H., & Wang, W. Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_3 · Cited by 187 · see citing works
Self-citationSelf-Imitation LearningPreprint
Oh, J., Guo, Y., Singh, S. P., & Lee, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05635 · Cited by 184 · see citing works
The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation
Ma, C., Wu, Z., AlRegib, G., Xiong, C., & Kira, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00689 · Cited by 166 · see citing works
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Wang, X. E., Chen, W., Wang, Y.-F., & Wang, W. Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1083 · Cited by 161 · see citing works
Self-citationTactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation
Ke, L., Li, X., Bisk, Y., Holtzman, A., Gan, Z., Liu, J., Gao, J., Choi, Y., & Srinivasa, S. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00690 · Cited by 148 · see citing works
Self-citationNguyen, K., Dey, D., Brockett, C., & Dolan, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01281 · Cited by 101 · see citing works
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2125 · Cited by 77 · see citing works
Self-citationLearning models for following natural language directions in unknown environments
Hemachandra, S. M., Duvallet, F., Howard, T. M., Roy, N., Stentz, A., & Walter, M. R. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139984 · Cited by 75 · see citing works
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
Thomason, J., Gordon, D., & Bisk, Y. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1197 · Cited by 61 · see citing works
Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking.Preprint
Lipton, Z. C., Gao, J., Li, L. C., Li, X., Ahmed, F., & Deng, L. (2016) · arXiv (Cornell University) · Cited by 38 · see citing works
Self-citationTurbo Learning for Captionbot and DrawingbotPreprint
Huang, Q., Zhang, P., Wu, D. O., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08170 · Cited by 26 · see citing works
Self-citationShifting the Baseline: Single Modality Performance on Visual Navigation & QAPreprint
Thomason, J., Gordon, D., & Bisk, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00613 · Cited by 23 · see citing works
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05448 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01075 · Cited by 540 · see citing works
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & van den Hengel, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01000 · Cited by 331 · see citing works
Self-citationLearning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Tan, H., Yu, L., & Bansal, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1268 · Cited by 293 · see citing works
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
Krantz, J., Wijmans, E., Majumdar, A., Batra, D., & Lee, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_7 · Cited by 266 · see citing works
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training
Hao, W., Li, C., Li, X., Carin, L., & Gao, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01315 · Cited by 264 · see citing works
Self-citationVLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264 · see citing works
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00323 · Cited by 220 · see citing works
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_16 · Cited by 201 · see citing works
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139 · see citing works
Structured Scene Memory for Vision-Language Navigation
Wang, H., Wang, W., Liang, W., Xiong, C., & Shen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00835 · Cited by 123 · see citing works
Self-citationInteractive Language: Talking to Robots in Real Time
Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., & Florence, P. (2022) · arXiv · doi:10.48550/arxiv.2210.06407 · Cited by 113 · see citing works
ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments
An, D., Wang, H., Wang, W., Wang, Z., Huang, Y., He, K., & Wang, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386695 · Cited by 103 · see citing works
Self-citationWaypoint Models for Instruction-guided Navigation in Continuous Environments
Krantz, J., Gokaslan, A., Batra, D., Lee, S., & Maksymets, O. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01488 · Cited by 94 · see citing works
Transferable Representation Learning in Vision-and-Language Navigation
Huang, H., Jain, V., Mehta, H., Ku, A., Magalhaes, G., Baldridge, J., & Ie, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00750 · Cited by 82 · see citing works
Envedit: Environment Editing for Vision-and-Language Navigation
Li, J., Tan, H., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01497 · Cited by 87 · see citing works
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80 · see citing works
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · see citing works
GridMM: Grid Memory Map for Vision-and-Language Navigation
Wang, Z., Li, X., Yang, J., Liu, Y., & Jiang, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01432 · Cited by 93 · see citing works
Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80 · see citing works
Active visual information gathering for vision-language navigation
Wang, H., Wang, W., Shu, T., Liang, W., & Shen, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58542-6_19 · Cited by 74 · see citing works
Self-citationNeighbor-view Enhanced Model for Vision and Language Navigation
An, D., Qi, Y., Huang, Y., Wu, Q., Wang, L., & Tan, T. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475282 · Cited by 74 · see citing works
Scaling Data Generation in Vision-and-Language Navigation
Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01103 · Cited by 64 · see citing works
“Meanwhile, an agent capable of understanding human natural language and navigating in photorealistic environments is a complex and modularized system [3, 14, 18, 32, 74, 77, 88, 91], and it is important to study how to effectively utilize the large-scale data to benefit the training of navigational agents adequately.”
BackgroundSelf-citationTarget-Driven Structured Transformer Planner for Vision-Language Navigation
Zhao, Y., Chen, J., Gao, C., Wang, W., Yang, L., Ren, H., Xia, H., & Liu, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548281 · Cited by 54 · see citing works
Self-citationReinforced Structured State-Evolution for Vision-Language Navigation
Chen, J., Gao, C., Meng, E., Zhang, Q., & Liu, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01501 · Cited by 45 · see citing works
ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding
Guo, Z., Tang, Y., Zhang, R., Wang, D., Wang, Z., Zhao, B., & Li, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01410 · Cited by 44 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“To address the heterogeneity challenge, researchers have proposed three main types of approaches:(1) Unified Feature Space Learning, which projects data from all modalities into the same feature space by constructing a common representation space, e.g., CLIP [190] employs Visual-Linguistic Contrast…”
Embodied navigation with multi-modal information: A survey from tasks to methodology
Wu, Y., Zhang, P., Gu, M., Zheng, J., & Bai, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102532 · Cited by 34 · see citing works
Hwang, M., Jeong, J., Kim, M., Oh, Y., & Oh, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00646 · Cited by 26 · see citing works
Language Conditioned Imitation Learning over Unstructured Data
Lynch, C., & Sermanet, P. (2020) · arXiv · doi:10.48550/arxiv.2005.07648 · Cited by 18 · see citing works
Self-supervised Regularization for Text Classification
Zhou, M., Li, Z., & Xie, P. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00389 · Cited by 14 · see citing works
Wang, S., Zhou, D., Xie, L., Xu, C., Yan, Y., & Yin, E. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107320 · Cited by 11 · see citing works
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Liu, X., Paul, S., Chatterjee, M., & Cherian, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28167 · Cited by 6 · see citing works
“To tackle the task, Wang et al. (Wang et al. 2019) uses cross-modal attention to focus on the relevant parts of both vision and language modalities, while others (Fried et al. 2018; Tan et al. 2019), used augmented instruction-trajectory pairs to improve the VLN performance.”
- Authors
- Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L.
- Affiliations
- University of California, Santa Barbara (US)
- Microsoft (United States) (US)
- Microsoft Research AI
- Duke University (US)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 6622-6631
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2964935470
- Cited by
- 522
- References
- 60
- Citations by year
- 2026: 28 · 2025: 57 · 2024: 71 · 2023: 76 · 2022: 62 · 2021: 109 · 2020: 86 · 2019: 33
- FWCI
- 23.65
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- vision-and-language navigation, cross-modal grounding, reinforcement learning, natural language instructions, embodied agents, generalization to unseen environments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
